GitLab

Metrics about providers

Provider	Nb datasets	Disk usage (JSON data)	Disk usage (source data)	Download time	Convert time
Eurostat	~5000	123G	269G	?	~1.5d

Notes:

• disk usage includes .git directory

Prometheus metrics

• prometheus is running on Eros, shipped with Gitlab, and collects metrics exposed in Eros and Dolos
• metrics are text files, updated regularly my deamons
• Grafana is an UI for viewing metrics, shipped with Gitlab, available here: https://git.nomics.world/-/grafana/

Services Summary

Eros

• Gitlab
  • service
    • Prometheus (via Gitlab)
      • metrics: http://localhost:9090/
    • Grafana (via Gitlab)
      • metrics: https://git.nomics.world/-/grafana/
      • auto-generated conf file: /var/opt/gitlab/grafana/grafana.ini
    • node-exporter (via GItlab)
      • metrics: http://localhost:9100/
      • auto-generated conf file: /var/opt/gitlab/
  • Howto edit conf and re-generate conf files:
    • vi /etc/gitlab/gitlab.rb
    • gitlab-ctl reconfigure
    • don't forget to restart services (ex: gitlab-ctl restart grafana)
• Other services
  • systemd timer: folder-sizes-metrics
    • metrics: dans node-exporter: http://localhost:9100/
      • clés: node_directory_size_bytes

Dolos

• uwsgi
  • https://github.com/timonwong/uwsgi_exporter
  • installed: via Docker (timonwong/uwsgi-exporter)
  • start: docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'
  • metrics: http://dolos.nomics.world:9117/metrics
• solr-exporter
  • shipped with Solr: /opt/solr/contrib/prometheus-exporter/bin/solr-exporter
  • start: root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
  • metrics: http://dolos.nomics.world:9854
• node-exporter
  • https://github.com/prometheus/node_exporter
  • installed via Debian prometheus-node-exporter (stretch-backports)
  • start: systemd service: prometheus-node-exporter
  • conf: /etc/default/prometheus-node-exporter
  • metrics: http://dolos.nomics.world:9100/metrics
• process-exporter
  • https://github.com/ncabatoff/process-exporter
  • installed via Docker
  • start: root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v pwd:/config ncabatoff/process-exporter --procfs /host/proc
  • metrics: http://dolos.nomics.world:9256/metrics
• custom metrics (python script)
  • https://git.nomics.world/dbnomics/dbnomics-metrics
  • folder-sizes-metrics
    • systemd timer: folder-sizes-metrics
    • metrics via node-exporter: http://dolos.nomics.world:9100/metrics
      • clés: node_directory_size_bytes
  • DBnomics metrics (Solr counts + Gitlab jobs)
    • systemd timer: dbnomics-custom-metrics
      • metrics via node-exporter: http://dolos.nomics.world:9100/metrics
        • clés:
          • dbnomics_datasets_total
          • dbnomics_series_total
          • dbnomics_providers_total
          • dbnomics_running_jobs
          • dbnomics_running_jobs_total

Installation notes

Gitlab metrics

• https://git.nomics.world/admin/application_settings/metrics_and_profiling
  • Metrics - Prometheus -> enabled
  • metrics: https://git.nomics.world/-/metrics?toke =Vy7HgyXTVc5hiHqnkyxJ
    • => access token deprecated https://docs.gitlab.com/ce/user/admin_area/monitoring/health_check.html#access-token-deprecated

Gitlab prometheus

• by default listening on 9090 root@eros:~# netstat -apn | grep 9090 tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN 39910/prometheus
• connection à prometheus via l'UI
  • ssh -N -L 9090:localhost:9090 eros
  • http://localhost:9090/
• endpoints: https://git.nomics.world/help/administration/monitoring/prometheus/index#gitlab-metrics
  • node -> infos système

Grafana

• https://docs.gitlab.com/omnibus/settings/grafana.html
  • https://git.nomics.world/-/grafana/
    • admin / password in jailbreak keypassx file
  • changed all Jailbreak team members to admin rights
• "Failed to get settings" bug makes it impossible to work
  • how to access to grafana logs throught Gitlab ?
    • gitlab-ctl tail grafana
    • on voit plein de logge =context error="database is locked"
  • /var/opt/gitlab/grafana/grafana.ini
  • DB: /var/opt/gitlab/grafana/data/grafana.db
    • let's reinit DB
      • gitlab-ctl stop grafana
      • mv /var/opt/gitlab/grafana/data/grafana.db{,.bck}
      • Add to grafana.ini: ''' [database] type=sqlite3 connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL '''
      • gitlab-ctl start grafana
      • /var/opt/gitlab/grafana/data/grafana.db have been recreated
    • => seems to work great ☺
• copy paste panel
  • trying to copy/paste Gitlab node "Node Available Memory"
    • it doesn't works as it. In formula there's variable $node that's not defined in the other context, I suppose. Replacing with localhost makes it works: ''' ( ) / node_memory_MemTotal_bytes{instanc =~"localhost:9100"} '''

Solr

• https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html
• /opt/solr/contrib/prometheus-exporter/bin
  • working dir: /opt/solr/contrib/prometheus-exporter
  • conf: ./conf/solr-exporter-config.xml
  • starting:
    • root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
    • => http://dolos.nomics.world:9854
  • Adding http://dolos.nomics.world:9854 to Grafana
    • is there a pre-configured dashboard ?
      • yes: https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#sample-grafana-dashboard
        • https://github.com/apache/lucene-solr/blob/master/solr/contrib/prometheus-exporter/conf/grafana-solr-dashboard.json
      • imported this dashboard
    • => errors. I sould have added it to Prometeus
  • adding to prometheus
    • https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#prometheus-configuration
      • /etc/gitlab/gitlab.rb
      • gitlab-ctl reconfigure
    • => I can see it on promotheus: http://localhost:9090/targets
  • I can access to data from prometheus:
    • rate(solr_metrics_core_requests_total[30m])
  • I can finaly access this data from "explore" tool in Grafana
    • rate(solr_metrics_core_requests_total{categor ="QUERY", handler="/query"}[5m])

Dolos node exporter

• https://github.com/prometheus/node_exporter

  • apt install golang-go
  • go get github.com/prometheus/node_exporter
    • package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
  • FInaly, contrary to what apt search node_exporter search gave me, there's a Debian package :s
    • apt install prometheus-node-exporter
    • => systemd service
    • listening on :9100
      • => http://dolos.nomics.world:9100/metrics
  • in promoethus conf: }, 'job_name': 'Dolos node', 'static_configs' => [ 'targets' => ['dolos.nomics.world:9100'], ], },
  • intersting exported metrics
    • node_filesystem_free
      • in promotheus: node_filesystem_free{devic ="/dev/md1"} => 173293019136
      • but df -h reports 114G free 🤔
  • import pre-configured dashboard: https://grafana.com/grafana/dashboards/1860
    • => doesn't show anything. Why ?
      • example: node_cpu_seconds_total{instanc =~"$node:$port",job=~"$job"}
      • maybe because job name = "node"
        • https://github.com/rfrail3/grafana-dashboards#node-exporter-full
    • looks like Debian version is vveerryyy old
  • trying to install from stretch-backports
    • root@dolos:~# apt install -t stretch-backports prometheus-node-exporter
    • => yes ☺
  • view RAM per process
    • https://github.com/ncabatoff/process-exporter
      • install: docker pull ncabatoff/process-exporter
      • run:
        • root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v pwd:/config ncabatoff/process-exporter --procfs /host/proc
      • => http://dolos.nomics.world:9256/metrics
    • add to gitlab conf + reload gitlab conf '''job_name': 'process-exporter-dolos', 'static_configs' => [ 'targets' => ['dolos.nomics.world:9256'], ],`
    • dashboard: https://grafana.com/grafana/dashboards/249
  • view requests count to DBnomics web API
    • uWSGI Exporter https://github.com/timonwong/uwsgi_exporter
      • docker pull timonwong/uwsgi-exporter
      • docker run -d -p 9117:9117 timonwong/uwsgi-exporter
        • => ça ne se lance pas
      • docker logs a6ee70d98c61ac22b124334eb31d364adc0a8daded4de263acb376fd952cf35a
        • leve =fatal msg="Incompatible uri " source="exporter.go:136"
      • https://github.com/timonwong/uwsgi_exporter/issues/32
      • => missing --stats.uri param
        • what to gives to this param ?
        • /etc/uwsgi/apps-enabled/dbnomics-api-uwsgi-v22.ini
          • stats = :8002
      • so let's try
        • `docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'''
        • => it works ☺
          • http://dolos.nomics.world:9117/metrics
    • add to promethus
      • existing dashboard: https://grafana.com/grafana/dashboards/3435
        • => oh no, it's for the other one: https://github.com/AndreaGreco/prometeus_uwsgi_exporter
    • how to graph ?
      • en existing dashboard exists but for the other exporter
        • => I can't make it run on Dolos (GOPATH variable problem)
      • let pick some values that seems interesting and graph them
        • go_memstats_alloc_bytes{jo ="uwsgi-exporter-dolos"}
        • rate(process_cpu_seconds_total{instanc =~"localhost:9100"}[2m])
        • rate(http_requests_total{instanc ="dolos.nomics.world:9117"}[10m])

• custom folders size

  • via node-exporter
    • dolos
      • https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector
      • dans le fichier de conf: /var/lib/prometheus/node-exporter

         #  --collector.textfile.director ="/var/lib/prometheus/node-exporter"
         #                            Directory to read text files with metrics from.

      • test: ajout d'un fichier test contenant: test_bduye 0
        • => OK
      • vrais stats
      • systemd service: echo "# Generated on $(date)" > /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && du -sb /home/gitlab-runner/json-data/*-json-data* | sed -ne "s/^\([0-9]\+\)\t\(.*\)$/node_directory_size_bytes{director ="\2"} \1/p" >> /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && mv /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom
    • eros

• déploiement

  • comment faire pour avoir de jolis labels ?
    • label_replace(v instant-vector, dst_label string, replacement string, src_label string, "")
      • => impossible: https://github.com/grafana/grafana/pull/6627