Improve CI pipeline of fetchers

Related to #507 (closed) (maybe do it first)

This issue is to be detailed...

Description

Goals:

  • currently CI convert and download jobs are defined by one job
    • we should use 2 different jobs
  • performance: avoid cloning Git repositories, especially big ones
    • takes > 45min for "big" fetchers (WB, ECB...)
    • note: we already have a stateful environment for Eurostat and IMF (cf wiki)
  • implement locks on pipelines
    • for example to prevent running INSEE download twice in parallel... otherwise git push of one of them will fail...
    • cf https://gitlab.com/gitlab-org/gitlab-foss/issues/20481
Edited Nov 27, 2020 by Christophe Benz
Assignee Loading
Time tracking Loading