note: the dolos-shell tag should have been named index or indexation
Runner configuration is spread between:
configuration files on runner host server
settings done in runner admin UI
This "configuration as code" would be awesome:
# This does not exist, this is pure imagination!runners:-server:dolosenvironment:type:dockerimage:debian/stretchprojects:-dbnomics/dbnomics-dashboardtags:-dashboard
Jobs execution environments
Most of jobs have a stateless environment, recreated each time they run. They clone Git repositories and commit to Git repositories.
Some of them are exceptions due to their size: Eurostat and IMF. Cloning Git repositories for each job would be too long so we decided to give them a stateful environment by telling them to use this runner using the eros-shell tag. This stateful environment is physically stored on the eros server in the directory /home/gitlab-runner/fetchers-envs (referenced by this line).
download job is triggered on a daily basis (via fetcher pipeline settings)
sequenceDiagramparticipant DJ as Download jobparticipant GS as GitLab serverparticipant SDW as source-data working treeparticipant PS as providerDJ ->> GS: clone / pullactivate GSGS ->> SDW: creating/updatingdeactivate GSDJ -->> SDW: clean files (optional)loop download process DJ ->> PS: fetching data DJ ->> SDW: writing dataendDJ ->> SDW: commit and pushactivate SDWSDW ->> GS: pushdeactivate SDW
convert job is configured to run when a commit occurred on *-source-data repository
sequenceDiagramparticipant DJ as Convert jobparticipant GS as GitLab serverparticipant SDW as source-data working treeparticipant JDW as json-data working treeDJ ->> GS: clone / pull source-dataactivate GSGS ->> SDW: creating/updatingdeactivate GSDJ ->> GS: clone / pull json-dataactivate GSGS ->> JDW: creating/updatingdeactivate GSDJ ->> JDW: delete filesloop convert process DJ ->> SDW: reading data DJ ->> JDW: writing dataendDJ ->> JDW: commit and pushactivate JDWJDW ->> GS: pushdeactivate JDW
graph LR gls[Gitlab server] jdr[json-data repo] ij[index job<br/>on dbnomics-importer] ss[Solr server] si[Solr index] gls -- starts -->ij ij -- read --> jdr ij -- publish --> ss ss -- update --> si
index job is configured to run when a commit occurred on *-json-data repository
sequenceDiagramparticipant IJ as Index jobparticipant GS as GitLab serverparticipant JDW as json-data working treeparticipant SS as Solr serverparticipant SI as Solr indexIJ ->> GS: clone / pull json-dataactivate GSGS ->> JDW: creating/updatingdeactivate GSloop index process IJ ->> JDW: reading data IJ ->> SS: publish SS ->> SI: updateendIJ ->> SS: commitSS ->> SI: commit / optimize
Disabling stuck jobs elimination
By default GitLab CI removes jobs that have been in a pending state for more than 2 hours.