monitoring

Metrics about providers

Provider	Nb datasets	Disk usage (JSON data)	Disk usage (source data)	Download time	Convert time
Eurostat	~5000	123G	269G	?	~1.5d

Notes:

disk usage includes .git directory

Prometheus metrics

prometheus is running on Eros, shipped with Gitlab, and collects metrics exposed in Eros and Dolos
metrics are text files, updated regularly my deamons
Grafana is an UI for viewing metrics, shipped with Gitlab, available here: https://git.nomics.world/-/grafana/

Services Summary

Eros

Gitlab
- service
  - Prometheus (via Gitlab)
    - metrics: http://localhost:9090/
  - Grafana (via Gitlab)
    - metrics: https://git.nomics.world/-/grafana/
    - auto-generated conf file: /var/opt/gitlab/grafana/grafana.ini
  - node-exporter (via GItlab)
    - metrics: http://localhost:9100/
    - auto-generated conf file: /var/opt/gitlab/
- Howto edit conf and re-generate conf files:
  - vi /etc/gitlab/gitlab.rb
  - gitlab-ctl reconfigure
  - don't forget to restart services (ex: gitlab-ctl restart grafana)
Other services
- systemd timer: folder-sizes-metrics
  - metrics: dans node-exporter: http://localhost:9100/
    - clés: node_directory_size_bytes

Dolos

uwsgi
- https://github.com/timonwong/uwsgi_exporter
- installed: via Docker (timonwong/uwsgi-exporter)
- start: docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'
- metrics: http://dolos.nomics.world:9117/metrics
solr-exporter
- shipped with Solr: /opt/solr/contrib/prometheus-exporter/bin/solr-exporter
- start: root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
- metrics: http://dolos.nomics.world:9854
node-exporter
- https://github.com/prometheus/node_exporter
- installed via Debian prometheus-node-exporter (stretch-backports)
- start: systemd service: prometheus-node-exporter
- conf: /etc/default/prometheus-node-exporter
- metrics: http://dolos.nomics.world:9100/metrics
process-exporter
- https://github.com/ncabatoff/process-exporter
- installed via Docker
- start: root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v pwd:/config ncabatoff/process-exporter --procfs /host/proc
- metrics: http://dolos.nomics.world:9256/metrics
custom metrics (python script)
- https://git.nomics.world/dbnomics/dbnomics-metrics
- folder-sizes-metrics
  - systemd timer: folder-sizes-metrics
  - metrics via node-exporter: http://dolos.nomics.world:9100/metrics
    - clés: node_directory_size_bytes
- DBnomics metrics (Solr counts + Gitlab jobs)
  - systemd timer: dbnomics-custom-metrics
    - metrics via node-exporter: http://dolos.nomics.world:9100/metrics
      - clés:
        
        dbnomics_datasets_total
        
        dbnomics_series_total
        
        dbnomics_providers_total
        
        dbnomics_running_jobs
        
        dbnomics_running_jobs_total

Installation notes

Gitlab metrics

https://git.nomics.world/admin/application_settings/metrics_and_profiling
- Metrics - Prometheus -> enabled
- metrics: https://git.nomics.world/-/metrics?toke =Vy7HgyXTVc5hiHqnkyxJ
  - => access token deprecated https://docs.gitlab.com/ce/user/admin_area/monitoring/health_check.html#access-token-deprecated

Gitlab prometheus

by default listening on 9090 root@eros:~# netstat -apn | grep 9090 tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN 39910/prometheus
connection à prometheus via l'UI
- ssh -N -L 9090:localhost:9090 eros
- http://localhost:9090/
endpoints: https://git.nomics.world/help/administration/monitoring/prometheus/index#gitlab-metrics
- node -> infos système

Grafana

https://docs.gitlab.com/omnibus/settings/grafana.html
- https://git.nomics.world/-/grafana/
  - admin / password in jailbreak keypassx file
- changed all Jailbreak team members to admin rights
"Failed to get settings" bug makes it impossible to work
- how to access to grafana logs throught Gitlab ?
  - gitlab-ctl tail grafana
  - on voit plein de logge =context error="database is locked"
- /var/opt/gitlab/grafana/grafana.ini
- DB: /var/opt/gitlab/grafana/data/grafana.db
  - let's reinit DB
    - gitlab-ctl stop grafana
    - mv /var/opt/gitlab/grafana/data/grafana.db{,.bck}
    - Add to grafana.ini: ''' [database] type=sqlite3 connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL '''
    - gitlab-ctl start grafana
    - /var/opt/gitlab/grafana/data/grafana.db have been recreated
  - => seems to work great ☺
copy paste panel
- trying to copy/paste Gitlab node "Node Available Memory"
  - it doesn't works as it. In formula there's variable $node that's not defined in the other context, I suppose. Replacing with localhost makes it works: ''' ( ) / node_memory_MemTotal_bytes{instanc =~"localhost:9100"} '''

Solr

https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html
/opt/solr/contrib/prometheus-exporter/bin
- working dir: /opt/solr/contrib/prometheus-exporter
- conf: ./conf/solr-exporter-config.xml
- starting:
  - root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
  - => http://dolos.nomics.world:9854
- Adding http://dolos.nomics.world:9854 to Grafana
  - is there a pre-configured dashboard ?
    - yes: https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#sample-grafana-dashboard
      - https://github.com/apache/lucene-solr/blob/master/solr/contrib/prometheus-exporter/conf/grafana-solr-dashboard.json
    - imported this dashboard
  - => errors. I sould have added it to Prometeus
- adding to prometheus
  - https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#prometheus-configuration
    - /etc/gitlab/gitlab.rb
    - gitlab-ctl reconfigure
  - => I can see it on promotheus: http://localhost:9090/targets
- I can access to data from prometheus:
  - rate(solr_metrics_core_requests_total[30m])
- I can finaly access this data from "explore" tool in Grafana
  - rate(solr_metrics_core_requests_total{categor ="QUERY", handler="/query"}[5m])

Dolos node exporter

https://github.com/prometheus/node_exporter
- apt install golang-go
- go get github.com/prometheus/node_exporter
  - package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
- FInaly, contrary to what apt search node_exporter search gave me, there's a Debian package :s
  - apt install prometheus-node-exporter
  - => systemd service
  - listening on :9100
    - => http://dolos.nomics.world:9100/metrics
- in promoethus conf: }, 'job_name': 'Dolos node', 'static_configs' => [ 'targets' => ['dolos.nomics.world:9100'], ], },
- intersting exported metrics
  - node_filesystem_free
    - in promotheus: node_filesystem_free{devic ="/dev/md1"} => 173293019136
    - but df -h reports 114G free 🤔
- import pre-configured dashboard: https://grafana.com/grafana/dashboards/1860
  - => doesn't show anything. Why ?
    - example: node_cpu_seconds_total{instanc =~~"$node:$port",job=~~"$job"}
    - maybe because job name = "node"
      - https://github.com/rfrail3/grafana-dashboards#node-exporter-full
  - looks like Debian version is vveerryyy old
- trying to install from stretch-backports
  - root@dolos:~# apt install -t stretch-backports prometheus-node-exporter
  - => yes ☺
- view RAM per process
  - https://github.com/ncabatoff/process-exporter
    - install: docker pull ncabatoff/process-exporter
    - run:
      - root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v pwd:/config ncabatoff/process-exporter --procfs /host/proc
    - => http://dolos.nomics.world:9256/metrics
  - add to gitlab conf + reload gitlab conf '''job_name': 'process-exporter-dolos', 'static_configs' => [ 'targets' => ['dolos.nomics.world:9256'], ],`
  - dashboard: https://grafana.com/grafana/dashboards/249
- view requests count to DBnomics web API
  - uWSGI Exporter https://github.com/timonwong/uwsgi_exporter
    - docker pull timonwong/uwsgi-exporter
    - docker run -d -p 9117:9117 timonwong/uwsgi-exporter
      - => ça ne se lance pas
    - docker logs a6ee70d98c61ac22b124334eb31d364adc0a8daded4de263acb376fd952cf35a
      - leve =fatal msg="Incompatible uri " source="exporter.go:136"
    - https://github.com/timonwong/uwsgi_exporter/issues/32
    - => missing --stats.uri param
      - what to gives to this param ?
      - /etc/uwsgi/apps-enabled/dbnomics-api-uwsgi-v22.ini
        
        stats = :8002
    - so let's try
      - `docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'''
      - => it works ☺
        
        http://dolos.nomics.world:9117/metrics
  - add to promethus
    - existing dashboard: https://grafana.com/grafana/dashboards/3435
      - => oh no, it's for the other one: https://github.com/AndreaGreco/prometeus_uwsgi_exporter
  - how to graph ?
    - en existing dashboard exists but for the other exporter
      - => I can't make it run on Dolos (GOPATH variable problem)
    - let pick some values that seems interesting and graph them
      - go_memstats_alloc_bytes{jo ="uwsgi-exporter-dolos"}
      - rate(process_cpu_seconds_total{instanc =~"localhost:9100"}[2m])
      - rate(http_requests_total{instanc ="dolos.nomics.world:9117"}[10m])
custom folders size
- via node-exporter
  - dolos
    - https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector
    - dans le fichier de conf: /var/lib/prometheus/node-exporter
      # --collector.textfile.director ="/var/lib/prometheus/node-exporter" # Directory to read text files with metrics from.
    - test: ajout d'un fichier test contenant: test_bduye 0
      - => OK
    - vrais stats
    - systemd service: echo "# Generated on $(date)" > /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && du -sb /home/gitlab-runner/json-data/*-json-data* | sed -ne "s/^$[0-9]\+$\t$.*$$/node_directory_size_bytes{director ="\2"} \1/p" >> /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && mv /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom
  - eros
déploiement
- comment faire pour avoir de jolis labels ?
  - label_replace(v instant-vector, dst_label string, replacement string, src_label string, "")
    - => impossible: https://github.com/grafana/grafana/pull/6627