... | ... | @@ -23,7 +23,7 @@ me@mylaptop:~$ source nomics_env/bin/activate |
|
|
### Architecture of a DBNOMICS Fetcher
|
|
|
|
|
|
A Fetcher is composed of modules or bricks:
|
|
|
* Converter: dbnomics-converter
|
|
|
* Data-model: dbnomics-data-model
|
|
|
* Source-data: dbnomics-source-data
|
|
|
* JSON-data: dbnomics-json-data
|
|
|
* Fetcher: dbnomics-fetcher
|
... | ... | @@ -33,8 +33,8 @@ For pedagogical purpose we will create the tree for dbnomics project by creating |
|
|
- dbnomics-source-data
|
|
|
- dbnomics-json-data
|
|
|
- dbnomics-fetcher
|
|
|
and by downloading:
|
|
|
- dbnomics-converter
|
|
|
and by cloning:
|
|
|
- dbnomics-data-model
|
|
|
|
|
|
```bash
|
|
|
(nomics_env) me@mylaptop:~$ mkdir dbnomics-source-data
|
... | ... | @@ -47,8 +47,8 @@ At the end of the procedure a DBNOMICS fetcher should be ordered in your compute |
|
|
```bash
|
|
|
(nomics_env) me@mylaptop:~$ tree . -L 2
|
|
|
.
|
|
|
├── dbnomics-converters
|
|
|
│ ├── dbnomics_converters
|
|
|
├── dbnomics-data_model
|
|
|
│ ├── dbnomics_data_model
|
|
|
│ ├── setup.cfg
|
|
|
│ └── setup.py
|
|
|
├── dbnomics-fetchers
|
... | ... | @@ -65,17 +65,19 @@ All folders inside source-data and json-data MUST follow the naming conventions: |
|
|
- `<provider_slug>-json-data`
|
|
|
- `<provider_slug>-fetcher`
|
|
|
|
|
|
#### Converter
|
|
|
#### Data_model :tools: FIXME
|
|
|
|
|
|
Converters are a specific library swith a et of utils, helpers and shortcodes to convert raw data (source-data) to dbnomics formatted data (json-data).
|
|
|
|
|
|
* `clone` the `repository` dbnomics/converter
|
|
|
Data_model consists of a specific library with a set of utils, helpers and shortcodes to convert raw data (source-data) to dbnomics formatted data (json-data).
|
|
|
using json-schema
|
|
|
|
|
|
* `clone` the `repository` dbnomics/data_model
|
|
|
|
|
|
* In ssh mode: you have to previously add your ssh_key to your profile on git.nomics.world.
|
|
|
|
|
|
```bash
|
|
|
|
|
|
(nomics_env) me@mylaptop:~$ git clone git@git.nomics.world:dbnomics/dbnomics-converters.git
|
|
|
(nomics_env) me@mylaptop:~$ git clone git@git.nomics.world:dbnomics/dbnomics-data-model.git
|
|
|
|
|
|
```
|
|
|
You will have to add your fingerprint to the server
|
... | ... | @@ -84,27 +86,29 @@ Converters are a specific library swith a et of utils, helpers and shortcodes t |
|
|
|
|
|
```bash
|
|
|
|
|
|
(nomics_env) me@mylaptop:~$ git clone https://git.nomics.world/dbnomics/dbnomics-converters
|
|
|
(nomics_env) me@mylaptop:~$ git clone https://git.nomics.world/dbnomics/dbnomics-data_model
|
|
|
|
|
|
```
|
|
|
* Check if clone is ok
|
|
|
|
|
|
```bash
|
|
|
|
|
|
(nomics_env) me@mylaptop:~$ ls dbnomics-converters/
|
|
|
(nomics_env) me@mylaptop:~$ dbnomics_converters setup.cfg setup.py
|
|
|
(nomics_env) me@mylaptop:~$ ls dbnomics-data_model/
|
|
|
(nomics_env) me@mylaptop:~$ dbnomics_data_model setup.cfg setup.py
|
|
|
```
|
|
|
|
|
|
* Install the library
|
|
|
* Install the library :tools: FIXME
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
(nomics_env) me@mylaptop:~$ pip -e dbnomics-converters/
|
|
|
(nomics_env) me@mylaptop:~$ pip -e dbnomics-data_model/
|
|
|
```
|
|
|
Add it the packags requirements with the current version
|
|
|
|
|
|
> When pulling dbnomics-converters think about reinstalling the current version
|
|
|
with :
|
|
|
```pip -e dbnomics-converters/```
|
|
|
```pip -e dbnomics-data_model/```
|
|
|
|
|
|
|
|
|
#### Source Data
|
... | ... | |