UNDATA fetcher for DBnomics

Fetch time series from the United Nations (UNDATA).

Download

API

• Gets all dataflows with structure ref: http://data.un.org/ws/rest/dataflow/
• Gets data: http://data.un.org/ws/rest/{DataflowId}
• Gets structure: http://data.un.org/ws/rest/datastructure/{AgencyId}/{StructureId}/?references=children

Dataflows

• download all but DF_UNDATA_SDG_PILOT "Sustainable Development Goals Pilot Exchange"
• data files are encoded on one single line
• In undata-source-data:
  • data files are named '{dataflow_id}.xml' and saved in data subdirectory
  • structure files are named '{structure_agencyid}_{structure_id}.xml' in structure subdirectory.
• Timestamp information are replaced by XXX in downloaded files

Install

It is recommended to use pyenv to use the same Python version than the one recommended for this project (cf .python-version).

To install this package:

# Check if the Python version specified in .python-version is installed
python -V
# If it is not, run `pyenv install`.
pyenv virtualenv bdf-fetcher
pyenv activate bdf-fetcher
pip install -r requirements-dev.txt
pip install -e .[dev]
pre-commit install

Then you can run the scripts of the fetcher:

python download.py ...
python convert.py ...

To deactivate the virtualenv:

pyenv deactivate

To run the dev tools:

pre-commit run --all-files

Visual Studio Code specific points

As this template uses the "sort imports" feature of Ruff, it is required to disable or uninstall the isort extension (cf docs).