Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
D
documentation
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • dbnomics-fetchers
  • documentation
  • Wiki
  • Home

Last edited by Christophe Benz Dec 22, 2020
Page history
This is an old version of this page. You can view the most recent version or browse the history.

Home

This wiki documents how to write a fetcher.

  • DBnomics tutorial notebook
  • Glossary
  • Fetcher design rules
  • Write a new fetcher
  • Write a new converter
  • Things to think about when finishing writing a converter
  • General acceptance criteria
  • JSON data model
  • Web API

Tooling

  • Development tools
  • Code style
  • Git and gitlab workflow
  • Code Optimization (profiling...)

Maintainance & Operations

  • Maintaining fetchers
  • Publishing Python packages
  • Solr indexation
  • Failure handling procedures
  • Troubleshooting (what to do if a problem occurs)

Infrastructure

  • Servers and services (architecture)
  • Setup CI jobs: see https://git.nomics.world/dbnomics/dbnomics-gitlab-ci/blob/master/README.md
  • CI jobs and runners
  • download-convert runners migration
  • Repositories organization
  • Test a fetcher on the pre-production server
  • Test the API or the UI on the development server
  • Download big git repositories statically
  • Production configuration
  • Monitoring
  • E-mails

Data

  • Special datasets and series

Other fetcher libraries

SDMX libraries:

  • https://github.com/amattioc/SDMX
  • https://pandasdmx.readthedocs.io/
  • https://github.com/26fe/jsonstat.py
  • https://github.com/sdmx-twg/sdmx-rest

Fun

  • DBnomics pearls

Project management

See this project and wiki

Clone repository
  • Code style
  • Git and Gitlab workflow
  • acceptance criteria
    • fetchers
  • ci jobs and runners
  • code optimization
  • dev tools
  • e mails
  • failure handling procedures
  • Home
  • librairies
  • maintaining fetchers
  • monitoring
  • presentation
  • production configuration
  • publishing python packages
View All Pages