Documentation
https://git.nomics.world/cepremap-private/servers-and-services
https://git.nomics.world/dbnomics-fetchers/documentation/wikis/ci-jobs-and-runners
Services
Note: bold services are public.
graph LR
%% services
api[API]
website[Web site]
solr[Solr]
runners1[CI runners]
runners2[CI runners]
%% style
style api stroke-width:4px
style website stroke-width:4px
%% resources
jsondata((JSON data<br>520G))
solrindex((Solr index<br>104G))
dockervolumes1((Docker volumes))
dockervolumesnote1>purged]
dockervolumes2((Docker volumes))
dockervolumesnote2>purged]
%% links
api --> solr
api --- jsondata
website --> api
solr --- solrindex
runners1 --- dockervolumes1
dockervolumes1 --- dockervolumesnote1
runners2 --- dockervolumes2
dockervolumes2 --- dockervolumesnote2
%% servers
subgraph dolos
api
jsondata
solr
solrindex
runners1
dockervolumes1
dockervolumesnote1
end
subgraph kalos
website
end
subgraph eros
runners2
dockervolumes2
dockervolumesnote2
end
Runners
- download and convert jobs on eros runner and nomos runner
- index job on dolos runner
- validate job on ioke runner
Resources:
- Docker volumes directory:
/var/lib/docker/volumes/
- is purged every night by "cron" job:
/etc/systemd/system/docker-system-prune.timer
- is purged every night by "cron" job:
Web site
- server:
kalos
- website directory:
/home/dbnomics_website/builds/
API
- server:
dolos
- two versions served by nginx:
/etc/nginx/sites-enabled/api.db.nomics.world.conf
- two Python venv:
/home/cepremap/virtualenvs/dbnomics-api-v2[1,2]
- v22 code is here:
/home/cepremap/dbnomics-api-v22/dbnomics-api
Resources:
- JSON data directory:
/home/gitlab-runner/json-data/
Solr
See also: Solr indexation
pre-production website
- See also
- pre-prod domain is declared in registrar (Gandi CEPREMAP - DC3428-GANDI) and points to
eros
- because we want to use the huge disk storage available on
eros
for pre-prod - pre-prod website is hosted on
eros
, same machine than gitlab server - gitlab server already provides nginx on port 80
- it's not the Debian package of nginx (
apt-cache policy nginx
=> not installed, it's a specific shipping of nginx done by gitlab (i.e./opt/gitlab/embedded/sbin/nginx
)
- it's not the Debian package of nginx (
- so we can't install another webserver on this machine, it would conflict on port 80, and using another port is not suited
- but gitlab server does not use classic
/etc/nginx
directory for config, it provides its own system- but it's misleading:
/etc/nginx
exists and is not empty- it belongs to Debian package
nginx-common
- it belongs to Debian package
- under the hood it (omnibus) compiles (using Chef?) templates of config files when doing
gitlab-ctl restart
, reading keys from/etc/gitlab/gitlab.rb
- but it's misleading:
- so we have to customize the nginx config of gitlab via
/etc/gitlab/gitlab.rb
(see doc)- we find
pre.db.nomics.world.conf
andapi.pre.db.nomics.world.conf
in/etc/nginx/conf.d
- but how to use
https
with this custom config? - gitlab generates SSL certificates for itself (
git.nomics.world
) but not for any includedconf.d
- we accept to use
http
and nothttps
- we accept to use
- we find
Past incidents:
Servers
Online servers
-
dolos
- PROD - Dedibox MD SSD 2017 - dolos.nomics.world 51.15.185.159- DBnomics Web API: https://api.db.nomics.world/v21
- DBnomics Web UI: https://db.nomics.world
- Solr (soft:
/opt/solr
- data:/var/solr/
)
-
eros
: Dedibox ST 18 2017- GitLab: https://git.nomics.world (conf: /etc/gitlab/gitlab.rb)
- Gitlab git repos (/var/opt/gitlab/git-data/repositories)
- Pre-production server (production API and UI with pre-production data): http://pre.db.nomics.world
- GitLab runners
- GitLab: https://git.nomics.world (conf: /etc/gitlab/gitlab.rb)
-
ioke
: Start-3-L- Integration server (development API and UI with production data): https://dev.db.nomics.world
-
nomos
: Store-2-M- GitLab runners
Scaleway servers
- mail.nomics.world (ARM64-2GB)
- forum.dynare.org (VC1S)
- forum.db.nomics.world (VC1S)
- analytics.nomics.world (VC1S)