Commit 4e4884e9 authored by Emmanuel Raviart's avatar Emmanuel Raviart
Browse files

Upgrade generated files. Prettify.

parent 260e686f
......@@ -49,51 +49,71 @@ npm run configure
## Usage
### Fetching Nesstar Servers
### Fetching DDI Files
#### Fetching DDI Files from OAI-PMH Servers
```bash
# ADISP
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.progedo-adisp.fr/ ../public_data/adisp-ddi/
# ADISP (OAI-PMH): Contains every french DDIs
npx babel-node --extensions ".ts" -- src/scripts/retrieve_oai-pmh_ddis.ts --url http://www.progedo-adisp.fr/oai/oai2.php ../public_data/adisp-oai-pmh-ddi/
```
#### Fetching DDI Files from Dataverse Servers
```bash
# data.sciencespo
npx babel-node --extensions ".ts" -- src/scripts/retrieve_dataverse_ddis.ts --url https://data.sciencespo.fr/ --verbose ../public_data/sciencespo-dataverse-ddi/
```
#### Fetching DDI Files from Nesstar Servers
```bash
# ADISP (public Nesstar) : Contains some French & English DDIs
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.progedo-adisp.fr/ ../public_data/adisp-nesstar-ddi/
# CDSP Sciences Po (obsolete & closed)
# npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.sciences-po.fr/ ../public_data/cdsp-ddi/
# npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.sciences-po.fr/ ../public_data/cdsp-nesstar-ddi/
# INED
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.ined.fr/ ../public_data/ined-ddi/
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.ined.fr/ ../public_data/ined-nesstar-ddi/
# INED - Generations and Gender Survey
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://ggpsurvey.ined.fr/ ../public_data/ined-gpgsurvey-ddi/
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://ggpsurvey.ined.fr/ ../public_data/ined-gpgsurvey-nesstar-ddi/
# UK Data Service
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.ukdataservice.ac.uk/ ../public_data/ukdataservice-ddi/
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nesstar.ukdataservice.ac.uk/ ../public_data/ukdataservice-nesstar-ddi/
# Norwegian Centre for Research Data
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nsddata.nsd.uib.no ../public_data/nsddata-ddi/
npx babel-node --extensions ".ts" src/scripts/retrieve_nesstar_ddis.ts --url http://nsddata.nsd.uib.no ../public_data/nsddata-nesstar-ddi/
```
### Fetching Dataverse Servers
### Repairing DDI Files
```bash
# data.sciencespo
npx babel-node --extensions ".ts" -- src/scripts/retrieve_dataverse_ddis.ts --url https://data.sciencespo.fr/ --verbose ../public_data/sciences-po-ddi/
# ADISP
npx babel-node --extensions ".ts" src/scripts/repair_adisp_ddis.ts --source=../public_data/adisp-oai-pmh-ddi/ ../public_data/adisp-oai-pmh-ddi-repaired
npx babel-node --extensions ".ts" src/scripts/repair_adisp_ddis.ts --source=../public_data/adisp-nesstar-ddi/ ../public_data/adisp-nesstar-ddi-repaired
```
### Repairing DDI files
### Indexing DDI files
#### Indexing Progedo DDI Files
```bash
# ADISP
npx babel-node --extensions ".ts" src/scripts/repair_adisp_ddis.ts --source=../public_data/adisp-ddi/ ../public_data/adisp-ddi-repaired
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=adisp ../public_data/adisp-oai-pmh-ddi-repaired/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=cdsp ../public_data/sciences-po-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=ined ../public_data/ined-nesstar-ddi/
```
### Indexing DDI files
#### Indexing Other (non Progedo-related) DDI Files
```bash
npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=fr --path=adisp ../public_data/adisp-ddi-repaired/
# npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=fr --path=cdsp ../public_data/cdsp-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=fr --path=cdsp ../public_data/sciences-po-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=fr --path=ined ../public_data/ined-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=fr --path=ined/gpgsurvey ../public_data/ined-gpgsurvey-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=en --path=ukdataservice ../public_data/ukdataservice-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_code_books.ts --language=no --path=nsddata ../public_data/nsddata-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=adisp-nesstar ../public_data/adisp-nesstar-ddi-repaired/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=cdsp ../public_data/sciencespo-dataverse-ddi/
# Obsolete
# npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=cdsp-obsolete ../public_data/cdsp-nesstar-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=fr --path=ined/gpgsurvey ../public_data/ined-gpgsurvey-nesstar-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=en --path=ukdataservice ../public_data/ukdataservice-nesstar-ddi/
npx babel-node --extensions ".ts" -- src/scripts/index_codebooks.ts --language=no --path=nsddata ../public_data/nsddata-nesstar-ddi/
```
### Extracting words from CodeBooks from autocompletion
### Extracting Words from CodeBooks for Autocompletion
```bash
npx babel-node --extensions ".ts" -- src/scripts/index_words.ts
......@@ -101,20 +121,28 @@ npx babel-node --extensions ".ts" -- src/scripts/index_words.ts
## Development
### Extracting TypeScript Raw Types from DDI files
### Extracting TypeScript Raw Types from DDI Files
#### Extracting TypeScript Raw Types from Progedo DDI Files
```bash
npx babel-node --extensions ".ts" --max-old-space-size=10240 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-oai-pmh-ddi-repaired/ ../public_data/sciencespo-dataverse-ddi/ ../public_data/ined-nesstar-ddi/
# Prettify generated TypeScript file:
npm run prettier
```
In `data-catalogue` directory:
#### Extracting TypeScript Raw Types for Other Tests
```bash
npx babel-node --extensions ".ts" --max-old-space-size=10240 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-ddi/ ../public_data/cdsp-ddi/ ../public_data/ined-ddi/ ../public_data/sciences-po-ddi/
npx babel-node --extensions ".ts" --max-old-space-size=10240 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-ddi/ ../public_data/cdsp-ddi/ ../public_data/ined-ddi/ --version=1.2.2
npx babel-node --extensions ".ts" -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-ddi/ ../public_data/cdsp-ddi/ ../public_data/ined-ddi/ --version=1.3
npx babel-node --extensions ".ts" -- src/scripts/raw_types_from_ddi_files.ts ../public_data/sciences-po-ddi/ --version=2.5
npx babel-node --extensions ".ts" --max-old-space-size=8192 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/xml-ddi-adisp/ --target=src/raw_types/enquetes_adisp_fournies.ts
npx babel-node --extensions ".ts" --max-old-space-size=8192 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-ddi/ --target=src/raw_types/enquetes_adisp.ts
npx babel-node --extensions ".ts" -- src/scripts/raw_types_from_ddi_files.ts ../public_data/cdsp-ddi/ --target=src/raw_types/enquetes_cdsp.ts
npx babel-node --extensions ".ts" -- src/scripts/raw_types_from_ddi_files.ts ../public_data/ined-ddi/ --target=src/raw_types/enquetes_ined.ts
npx babel-node --extensions ".ts" --max-old-space-size=10240 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-oai-pmh-ddi-repaired/ ../public_data/sciencespo-dataverse-ddi/ ../public_data/ined-nesstar-ddi/ --version=1.2.2
npx babel-node --extensions ".ts" --max-old-space-size=10240 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-oai-pmh-ddi-repaired/ ../public_data/sciencespo-dataverse-ddi/ ../public_data/ined-nesstar-ddi/ --version=1.3
npx babel-node --extensions ".ts" --max-old-space-size=10240 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-oai-pmh-ddi-repaired/ ../public_data/sciencespo-dataverse-ddi/ ../public_data/ined-nesstar-ddi/ --version=2.5
npx babel-node --extensions ".ts" --max-old-space-size=8192 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-manual-ddi/ --target=src/raw_types/codebooks_adisp_manual.ts
npx babel-node --extensions ".ts" --max-old-space-size=8192 -- src/scripts/raw_types_from_ddi_files.ts ../public_data/adisp-nesstar-ddi/ --target=src/raw_types/codebooks_adisp_nesstar.ts
npx babel-node --extensions ".ts" -- src/scripts/raw_types_from_ddi_files.ts ../public_data/sciencespo-dataverse-ddi/ --target=src/raw_types/codebooks_sciencespo_dataverse.ts
npx babel-node --extensions ".ts" -- src/scripts/raw_types_from_ddi_files.ts ../public_data/ined-nesstar-ddi/ --target=src/raw_types/codebooks_ined_nesstar.ts
# Prettify generated TypeScript files:
npm run prettier
......
import type { Organization } from "./data"
import type { CodeBook, SerStmt, Var } from "./raw_types/code_books"
import type { CodeBook, SerStmt, Var } from "./raw_types/codebooks"
type CodeBookPartialOrganization =
| string
......
......@@ -136,7 +136,8 @@
class="portal mx-2"
on:blur={(event) => setLanguage(event.target.value)}
on:change={(event) => setLanguage(event.target.value)}
value={$language}>
value={$language}
>
{#each proposedLanguagesAndLabels as [language, _label, flag]}
<option value={language}>{flag} {language}</option>
{/each}
......@@ -151,7 +152,8 @@
<button
class="portal mx-2"
value={user.username}
on:click={() => (userMenuOpen = !userMenuOpen)}>
on:click={() => (userMenuOpen = !userMenuOpen)}
>
<span class="mr-1">{user.username}</span>
</button>
{#if userMenuOpen}
......
import type { CodeBook } from "./raw_types/code_books"
import type { CodeBook } from "./raw_types/codebooks"
export enum CodeBookVersion {
Version_1_2_2 = "1.2.2",
......
......@@ -8,7 +8,7 @@ import {
Variable,
VariableSearchResult,
} from "./data"
import type { CodeBook } from "./raw_types/code_books"
import type { CodeBook } from "./raw_types/codebooks"
function codeBookFromPath(
json: JsonData,
......
......@@ -7,7 +7,7 @@ import path from "path"
import { Follow, JsonData, Series, Study, Variable } from "./data"
import config from "./config"
import { db } from "./database"
import type { CodeBook } from "./raw_types/code_books"
import type { CodeBook } from "./raw_types/codebooks"
import type { Request } from "./requests_responses"
const { publicDataDir } = config
......
......@@ -25,7 +25,7 @@ import {
StudyTermRelation,
} from "./data"
import { walkDir } from "./file_systems"
import type { CodeBook } from "./raw_types/code_books"
import type { CodeBook } from "./raw_types/codebooks"
import { slugify } from "./strings"
const publicDataDir = path.resolve(config.publicDataDir)
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
......@@ -5,6 +5,18 @@
const dev = process.env.NODE_ENV === "development"
</script>
<svelte:head>
<title>{status}</title>
</svelte:head>
<h1>{status}</h1>
<p>{error.message}</p>
{#if dev && error.stack}
<pre>{error.stack}</pre>
{/if}
<style>
h1,
p {
......@@ -27,15 +39,3 @@
}
}
</style>
<svelte:head>
<title>{status}</title>
</svelte:head>
<h1>{status}</h1>
<p>{error.message}</p>
{#if dev && error.stack}
<pre>{error.stack}</pre>
{/if}
......@@ -45,49 +45,57 @@
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="TGIR PROGEDO: Production et gestion des données d’enquêtes pour la recherche en sciences sociales"
href="http://www.progedo.fr/">Progedo
href="http://www.progedo.fr/"
>Progedo
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="Archives de Données Issues de la Statistique Publique"
href="http://www.progedo-adisp.fr/">Adisp
href="http://www.progedo-adisp.fr/"
>Adisp
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="L'Institut national d'études démographiques"
href="http://www.ined.fr">Ined
href="http://www.ined.fr"
>Ined
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="Centre de Données Socio-Politiques"
href="https://cdsp.sciences-po.fr/fr/le-cdsp/presentation/">Cdsp
href="https://cdsp.sciences-po.fr/fr/le-cdsp/presentation/"
>Cdsp
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="Consortium of European Social Science Data Archives"
href="https://www.cessda.eu">Cessda
href="https://www.cessda.eu"
>Cessda
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="TGIR CEPREMAP: Centre pour la recherche économique et ses applications"
href="https://www.cepremap.fr/">Cepremap
href="https://www.cepremap.fr/"
>Cepremap
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="The Largest Free Economic Database"
href="https://db.nomics.world">DBnomics
href="https://db.nomics.world"
>DBnomics
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
<a
title="La TGIR des humanités numériques"
href="https://www.huma-num.fr/">Huma-Num
href="https://www.huma-num.fr/"
>Huma-Num
</a>
</p>
<p class="portal ~neutral px-2 py-2 hover:text-grey-700">
......@@ -177,7 +185,8 @@
© Copyright 2020, 2021
<a
title="TGIR PROGEDO: Production et gestion des données d’enquêtes pour la recherche en sciences sociales"
href="https://www.progedo.fr">Progedo &
href="https://www.progedo.fr"
>Progedo &
</a>
<a
title="TGIR CEPREMAP: Centre pour la recherche économique et ses applications"
......
......@@ -8,6 +8,7 @@ import {
auditTrimString,
cleanAudit,
} from "@auditors/core"
import assert from "assert"
import commandLineArgs from "command-line-args"
import he from "he"
import xmlParser from "fast-xml-parser"
......@@ -31,7 +32,7 @@ const optionsDefinitions = [
{
alias: "t",
help:
"path of TypeScript file to generate (default: src/raw_types/code_books.ts)",
"path of TypeScript file to generate (default: src/raw_types/codebooks.ts)",
name: "target",
type: String,
},
......@@ -49,7 +50,7 @@ const optionsDefinitions = [
},
{
defaultOption: true,
help: "directory containing XML DDI Codebook files",
help: "directory containing XML DDI CodeBook files",
multiple: true,
name: "xmlDirs",
type: String,
......@@ -126,7 +127,7 @@ async function main() {
path.join(
"src",
"raw_types",
version ? `code_books_${version}.ts` : `code_books.ts`,
version ? `codebooks_${version}.ts` : `codebooks.ts`,
)
const codeBooksJson = []
......@@ -151,6 +152,7 @@ async function main() {
},
true,
).codeBook
assert.notStrictEqual(codeBook, undefined)
if (version == null || codeBook["@version"] === version) {
codeBooksJson.push(JSON.stringify(codeBook))
}
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment