[EPIC] Fetchers DX
DX = developer experience
This EPIC lists issues aiming to simplify development of fetchers by providing higher-level tools/libraries/services, harmonize fetchers by reducing their specificities, and improve technical documentation. It does not deal with data model.
This would prevent fetcher authors to reinvent something, having to copy/paste recipies, reduce the risk of forgetting something and give directives about how to write a fetcher in general.
The common tasks could be implemented as a library of functions (i.e. dbnomics_fetchers
Python package), or fetchers could implement some method, inherit a class, etc. Each solution will be discussed in specific issues, and we'll decide about the implementation once we will be sure that it satisfies the majority of the needs.
Goals
- for each fetcher, write the mapping between the source data model and the dbnomics data model, for example in the README
- propose an example to fill-in
Issues
- #642 (closed) Create a documentation website
- #507 (closed) Harmonization of CI jobs
- #523 (closed) Improve CI pipeline
- #516 Have a pre-production environment for fetchers
- #504 (closed) CONTRIBUTING.md
- #520 Ensure commits consistency for datasets
- #498 Some fetchers generate too many commits
-
#531 (closed) Harmonization of storage
- would solve #498
- #554 Simplify how to write fetchers
- #542 (closed) Accumulating data accross revisions