Improve CI pipeline of fetchers
Related to #507 (closed) (maybe do it first)
This issue is to be detailed...
Description
Goals:
- currently CI convert and download jobs are defined by one job
- we should use 2 different jobs
- performance: avoid cloning Git repositories, especially big ones
- takes > 45min for "big" fetchers (WB, ECB...)
- note: we already have a stateful environment for Eurostat and IMF (cf wiki)
- implement locks on pipelines
- for example to prevent running INSEE download twice in parallel... otherwise
git push
of one of them will fail... - cf https://gitlab.com/gitlab-org/gitlab-foss/issues/20481
- for example to prevent running INSEE download twice in parallel... otherwise
Edited by Christophe Benz