Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
D
documentation
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • dbnomics-fetchers
  • documentation
  • Wiki
  • servers

Last edited by Christophe Benz Dec 30, 2019
Page history

servers

Documentation

https://git.nomics.world/cepremap-private/servers-and-services

https://git.nomics.world/dbnomics-fetchers/documentation/wikis/ci-jobs-and-runners

Services

Note: bold services are public.

graph LR
  %% services
  api[API]
  website[Web site]
  solr[Solr]
  runners1[CI runners]
  runners2[CI runners]

  %% style
  style api stroke-width:4px
  style website stroke-width:4px

  %% resources
  jsondata((JSON data<br>520G))
  solrindex((Solr index<br>104G))
  dockervolumes1((Docker volumes))
  dockervolumesnote1>purged]
  dockervolumes2((Docker volumes))
  dockervolumesnote2>purged]

  %% links
  api --> solr
  api --- jsondata
  website --> api
  solr --- solrindex
  runners1 --- dockervolumes1
  dockervolumes1 --- dockervolumesnote1
  runners2 --- dockervolumes2
  dockervolumes2 --- dockervolumesnote2

  %% servers
  subgraph dolos
    api
    jsondata
    solr
    solrindex
    runners1
    dockervolumes1
    dockervolumesnote1
  end
  subgraph kalos
    website
  end
  subgraph eros
    runners2
    dockervolumes2
    dockervolumesnote2
  end

Runners

  • download and convert jobs on eros runner and nomos runner
  • index job on dolos runner
  • validate job on ioke runner

Resources:

  • Docker volumes directory: /var/lib/docker/volumes/
    • is purged every night by "cron" job: /etc/systemd/system/docker-system-prune.timer

Web site

  • server: kalos
  • website directory: /home/dbnomics_website/builds/

API

  • server: dolos
  • two versions served by nginx: /etc/nginx/sites-enabled/api.db.nomics.world.conf
  • two Python venv: /home/cepremap/virtualenvs/dbnomics-api-v2[1,2]
  • v22 code is here: /home/cepremap/dbnomics-api-v22/dbnomics-api

Resources:

  • JSON data directory: /home/gitlab-runner/json-data/

Solr

See also: Solr indexation

pre-production website

  • See also
    • https://git.nomics.world/cepremap-private/servers-and-services/blob/master/gitlab-server-eros.nomics.world.md#web-ui-installation
  • pre-prod domain is declared in registrar (Gandi CEPREMAP - DC3428-GANDI) and points to eros
  • because we want to use the huge disk storage available on eros for pre-prod
  • pre-prod website is hosted on eros, same machine than gitlab server
  • gitlab server already provides nginx on port 80
    • it's not the Debian package of nginx (apt-cache policy nginx => not installed, it's a specific shipping of nginx done by gitlab (i.e. /opt/gitlab/embedded/sbin/nginx)
  • so we can't install another webserver on this machine, it would conflict on port 80, and using another port is not suited
  • but gitlab server does not use classic /etc/nginx directory for config, it provides its own system
    • but it's misleading: /etc/nginx exists and is not empty
      • it belongs to Debian package nginx-common
    • under the hood it (omnibus) compiles (using Chef?) templates of config files when doing gitlab-ctl restart, reading keys from /etc/gitlab/gitlab.rb
  • so we have to customize the nginx config of gitlab via /etc/gitlab/gitlab.rb (see doc)
    • we find pre.db.nomics.world.conf and api.pre.db.nomics.world.conf in /etc/nginx/conf.d
    • but how to use https with this custom config?
    • gitlab generates SSL certificates for itself (git.nomics.world) but not for any included conf.d
      • we accept to use http and not https

Past incidents:

  • management#506 (closed)

Servers

Online servers

  • dolos - PROD - Dedibox MD SSD 2017 - dolos.nomics.world 51.15.185.159
    • DBnomics Web API: https://api.db.nomics.world/v21
    • DBnomics Web UI: https://db.nomics.world
    • Solr (soft: /opt/solr - data: /var/solr/)
  • eros: Dedibox ST 18 2017
    • GitLab: https://git.nomics.world (conf: /etc/gitlab/gitlab.rb)
      • Gitlab git repos (/var/opt/gitlab/git-data/repositories)
    • Pre-production server (production API and UI with pre-production data): http://pre.db.nomics.world
    • GitLab runners
  • ioke: Start-3-L
    • Integration server (development API and UI with production data): https://dev.db.nomics.world
  • nomos: Store-2-M
    • GitLab runners

Scaleway servers

  • mail.nomics.world (ARM64-2GB)
  • forum.dynare.org (VC1S)
  • forum.db.nomics.world (VC1S)
  • analytics.nomics.world (VC1S)
Clone repository
  • Code style
  • Git and Gitlab workflow
  • acceptance criteria
    • fetchers
  • ci jobs and runners
  • code optimization
  • dev tools
  • e mails
  • failure handling procedures
  • Home
  • librairies
  • maintaining fetchers
  • monitoring
  • presentation
  • production configuration
  • publishing python packages
View All Pages