servers

Documentation

https://git.nomics.world/cepremap-private/servers-and-services

https://git.nomics.world/dbnomics-fetchers/documentation/wikis/ci-jobs-and-runners

Services

Note: bold services are public.

graph LR
  %% services
  api[API]
  website[Web site]
  solr[Solr]
  runners1[CI runners]
  runners2[CI runners]

  %% style
  style api stroke-width:4px
  style website stroke-width:4px

  %% resources
  jsondata((JSON data<br>520G))
  solrindex((Solr index<br>104G))
  dockervolumes1((Docker volumes))
  dockervolumesnote1>purged]
  dockervolumes2((Docker volumes))
  dockervolumesnote2>purged]

  %% links
  api --> solr
  api --- jsondata
  website --> api
  solr --- solrindex
  runners1 --- dockervolumes1
  dockervolumes1 --- dockervolumesnote1
  runners2 --- dockervolumes2
  dockervolumes2 --- dockervolumesnote2

  %% servers
  subgraph dolos
    api
    jsondata
    solr
    solrindex
    runners1
    dockervolumes1
    dockervolumesnote1
  end
  subgraph kalos
    website
  end
  subgraph eros
    runners2
    dockervolumes2
    dockervolumesnote2
  end

Runners

download and convert jobs on eros runner and nomos runner
index job on dolos runner
validate job on ioke runner

Resources:

Docker volumes directory: /var/lib/docker/volumes/
- is purged every night by "cron" job: /etc/systemd/system/docker-system-prune.timer

Web site

server: kalos
website directory: /home/dbnomics_website/builds/

API

server: dolos
two versions served by nginx: /etc/nginx/sites-enabled/api.db.nomics.world.conf
two Python venv: /home/cepremap/virtualenvs/dbnomics-api-v2[1,2]
v22 code is here: /home/cepremap/dbnomics-api-v22/dbnomics-api

Resources:

JSON data directory: /home/gitlab-runner/json-data/

Solr

pre-production website

See also
- https://git.nomics.world/cepremap-private/servers-and-services/blob/master/gitlab-server-eros.nomics.world.md#web-ui-installation
pre-prod domain is declared in registrar (Gandi CEPREMAP - DC3428-GANDI) and points to eros
because we want to use the huge disk storage available on eros for pre-prod
pre-prod website is hosted on eros, same machine than gitlab server
gitlab server already provides nginx on port 80
- it's not the Debian package of nginx (apt-cache policy nginx => not installed, it's a specific shipping of nginx done by gitlab (i.e. /opt/gitlab/embedded/sbin/nginx)
so we can't install another webserver on this machine, it would conflict on port 80, and using another port is not suited
but gitlab server does not use classic /etc/nginx directory for config, it provides its own system
- but it's misleading: /etc/nginx exists and is not empty
  - it belongs to Debian package nginx-common
- under the hood it (omnibus) compiles (using Chef?) templates of config files when doing gitlab-ctl restart, reading keys from /etc/gitlab/gitlab.rb
so we have to customize the nginx config of gitlab via /etc/gitlab/gitlab.rb (see doc)
- we find pre.db.nomics.world.conf and api.pre.db.nomics.world.conf in /etc/nginx/conf.d
- but how to use https with this custom config?
- gitlab generates SSL certificates for itself (git.nomics.world) but not for any included conf.d
  - we accept to use http and not https

Past incidents:

management#506 (closed)

Online servers

dolos - PROD - Dedibox MD SSD 2017 - dolos.nomics.world 51.15.185.159
- DBnomics Web API: https://api.db.nomics.world/v21
- DBnomics Web UI: https://db.nomics.world
- Solr (soft: /opt/solr - data: /var/solr/)
eros: Dedibox ST 18 2017
- GitLab: https://git.nomics.world (conf: /etc/gitlab/gitlab.rb)
  - Gitlab git repos (/var/opt/gitlab/git-data/repositories)
- Pre-production server (production API and UI with pre-production data): http://pre.db.nomics.world
- GitLab runners
ioke: Start-3-L
- Integration server (development API and UI with production data): https://dev.db.nomics.world
nomos: Store-2-M
- GitLab runners

Scaleway servers

mail.nomics.world (ARM64-2GB)
forum.dynare.org (VC1S)
forum.db.nomics.world (VC1S)
analytics.nomics.world (VC1S)