Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • D documentation
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • dbnomics-fetchers
  • documentation
  • Wiki
  • Home

Last edited by Christophe Benz May 06, 2022
Page history
This is an old version of this page. You can view the most recent version or browse the history.

Home

This wiki documents how to write a fetcher.

  • DBnomics tutorial notebook
  • Glossary
  • Fetcher design rules
  • Write a new fetcher
  • Write a new converter
  • Things to think about when finishing writing a converter
  • General acceptance criteria
  • JSON data model
  • Web API

Tooling

  • Development tools
  • Code style
  • Git and gitlab workflow
  • Code Optimization (profiling...)

Maintainance & Operations

  • Maintaining fetchers
  • Publishing Python packages
  • Solr indexation
  • Failure handling procedures
  • Troubleshooting (what to do if a problem occurs)

Infrastructure

  • Servers and services (architecture)
  • Setup CI jobs: see https://git.nomics.world/dbnomics/dbnomics-gitlab-ci/blob/master/README.md
  • CI jobs and runners
  • [download-convert runners migration(runners-migration)
  • Repositories organization
  • Test a fetcher on the pre-production server
  • Test the API or the UI on the development server
  • Download big git repositories statically
  • Production configuration
  • Monitoring
  • E-mails

Data

  • Special datasets

Other fetcher libraries

SDMX libraries:

  • https://github.com/amattioc/SDMX
  • https://pandasdmx.readthedocs.io/
  • https://github.com/26fe/jsonstat.py
  • https://github.com/sdmx-twg/sdmx-rest

Fun

  • DBnomics pearls

Project management

See this project and wiki

Clone repository
  • Code style
  • Git and Gitlab workflow
  • acceptance criteria
    • fetchers
  • ci jobs and runners
  • code optimization
  • dev tools
  • e mails
  • failure handling procedures
  • Home
  • librairies
  • maintaining fetchers
  • monitoring
  • presentation
  • production configuration
  • publishing python packages
View All Pages