Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • D documentation
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • dbnomics-fetchers
  • documentation
  • Wiki
  • monitoring

Last edited by Bruno Duyé Oct 24, 2019
Page history

monitoring

Metrics about providers

Provider Nb datasets Disk usage (JSON data) Disk usage (source data) Download time Convert time
Eurostat ~5000 123G 269G ? ~1.5d

Notes:

  • disk usage includes .git directory

Prometheus metrics

  • prometheus is running on Eros, shipped with Gitlab, and collects metrics exposed in Eros and Dolos
  • metrics are text files, updated regularly my deamons
  • Grafana is an UI for viewing metrics, shipped with Gitlab, available here: https://git.nomics.world/-/grafana/

Services Summary

Eros

  • Gitlab
    • service
      • Prometheus (via Gitlab)
        • metrics: http://localhost:9090/
      • Grafana (via Gitlab)
        • metrics: https://git.nomics.world/-/grafana/
        • auto-generated conf file: /var/opt/gitlab/grafana/grafana.ini
      • node-exporter (via GItlab)
        • metrics: http://localhost:9100/
        • auto-generated conf file: /var/opt/gitlab/
    • Howto edit conf and re-generate conf files:
      • vi /etc/gitlab/gitlab.rb
      • gitlab-ctl reconfigure
      • don't forget to restart services (ex: gitlab-ctl restart grafana)
  • Other services
    • systemd timer: folder-sizes-metrics
      • metrics: dans node-exporter: http://localhost:9100/
        • clés: node_directory_size_bytes

Dolos

  • uwsgi
    • https://github.com/timonwong/uwsgi_exporter
    • installed: via Docker (timonwong/uwsgi-exporter)
    • start: docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'
    • metrics: http://dolos.nomics.world:9117/metrics
  • solr-exporter
    • shipped with Solr: /opt/solr/contrib/prometheus-exporter/bin/solr-exporter
    • start: root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
    • metrics: http://dolos.nomics.world:9854
  • node-exporter
    • https://github.com/prometheus/node_exporter
    • installed via Debian prometheus-node-exporter (stretch-backports)
    • start: systemd service: prometheus-node-exporter
    • conf: /etc/default/prometheus-node-exporter
    • metrics: http://dolos.nomics.world:9100/metrics
  • process-exporter
    • https://github.com/ncabatoff/process-exporter
    • installed via Docker
    • start: root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v pwd:/config ncabatoff/process-exporter --procfs /host/proc
    • metrics: http://dolos.nomics.world:9256/metrics
  • custom metrics (python script)
    • https://git.nomics.world/dbnomics/dbnomics-metrics
    • folder-sizes-metrics
      • systemd timer: folder-sizes-metrics
      • metrics via node-exporter: http://dolos.nomics.world:9100/metrics
        • clés: node_directory_size_bytes
    • DBnomics metrics (Solr counts + Gitlab jobs)
      • systemd timer: dbnomics-custom-metrics
        • metrics via node-exporter: http://dolos.nomics.world:9100/metrics
          • clés:
            • dbnomics_datasets_total
            • dbnomics_series_total
            • dbnomics_providers_total
            • dbnomics_running_jobs
            • dbnomics_running_jobs_total

Installation notes

Gitlab metrics

  • https://git.nomics.world/admin/application_settings/metrics_and_profiling
    • Metrics - Prometheus -> enabled
    • metrics: https://git.nomics.world/-/metrics?toke =Vy7HgyXTVc5hiHqnkyxJ
      • => access token deprecated https://docs.gitlab.com/ce/user/admin_area/monitoring/health_check.html#access-token-deprecated

Gitlab prometheus

  • by default listening on 9090 root@eros:~# netstat -apn | grep 9090 tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN 39910/prometheus
  • connection à prometheus via l'UI
    • ssh -N -L 9090:localhost:9090 eros
    • http://localhost:9090/
  • endpoints: https://git.nomics.world/help/administration/monitoring/prometheus/index#gitlab-metrics
    • node -> infos système

Grafana

  • https://docs.gitlab.com/omnibus/settings/grafana.html
    • https://git.nomics.world/-/grafana/
      • admin / password in jailbreak keypassx file
    • changed all Jailbreak team members to admin rights
  • "Failed to get settings" bug makes it impossible to work
    • how to access to grafana logs throught Gitlab ?
      • gitlab-ctl tail grafana
      • on voit plein de logge =context error="database is locked"
    • /var/opt/gitlab/grafana/grafana.ini
    • DB: /var/opt/gitlab/grafana/data/grafana.db
      • let's reinit DB
        • gitlab-ctl stop grafana
        • mv /var/opt/gitlab/grafana/data/grafana.db{,.bck}
        • Add to grafana.ini: ''' [database] type=sqlite3 connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL '''
        • gitlab-ctl start grafana
        • /var/opt/gitlab/grafana/data/grafana.db have been recreated
      • => seems to work great ☺
  • copy paste panel
    • trying to copy/paste Gitlab node "Node Available Memory"
      • it doesn't works as it. In formula there's variable $node that's not defined in the other context, I suppose. Replacing with localhost makes it works: ''' ( ) / node_memory_MemTotal_bytes{instanc =~"localhost:9100"} '''

Solr

  • https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html
  • /opt/solr/contrib/prometheus-exporter/bin
    • working dir: /opt/solr/contrib/prometheus-exporter
    • conf: ./conf/solr-exporter-config.xml
    • starting:
      • root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
      • => http://dolos.nomics.world:9854
    • Adding http://dolos.nomics.world:9854 to Grafana
      • is there a pre-configured dashboard ?
        • yes: https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#sample-grafana-dashboard
          • https://github.com/apache/lucene-solr/blob/master/solr/contrib/prometheus-exporter/conf/grafana-solr-dashboard.json
        • imported this dashboard
      • => errors. I sould have added it to Prometeus
    • adding to prometheus
      • https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#prometheus-configuration
        • /etc/gitlab/gitlab.rb
        • gitlab-ctl reconfigure
      • => I can see it on promotheus: http://localhost:9090/targets
    • I can access to data from prometheus:
      • rate(solr_metrics_core_requests_total[30m])
    • I can finaly access this data from "explore" tool in Grafana
      • rate(solr_metrics_core_requests_total{categor ="QUERY", handler="/query"}[5m])

Dolos node exporter

  • https://github.com/prometheus/node_exporter

    • apt install golang-go
    • go get github.com/prometheus/node_exporter
      • package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
    • FInaly, contrary to what apt search node_exporter search gave me, there's a Debian package :s
      • apt install prometheus-node-exporter
      • => systemd service
      • listening on :9100
        • => http://dolos.nomics.world:9100/metrics
    • in promoethus conf: }, 'job_name': 'Dolos node', 'static_configs' => [ 'targets' => ['dolos.nomics.world:9100'], ], },
    • intersting exported metrics
      • node_filesystem_free
        • in promotheus: node_filesystem_free{devic ="/dev/md1"} => 173293019136
        • but df -h reports 114G free 🤔
    • import pre-configured dashboard: https://grafana.com/grafana/dashboards/1860
      • => doesn't show anything. Why ?
        • example: node_cpu_seconds_total{instanc ="$node:$port",job="$job"}
        • maybe because job name = "node"
          • https://github.com/rfrail3/grafana-dashboards#node-exporter-full
      • looks like Debian version is vveerryyy old
    • trying to install from stretch-backports
      • root@dolos:~# apt install -t stretch-backports prometheus-node-exporter
      • => yes ☺
    • view RAM per process
      • https://github.com/ncabatoff/process-exporter
        • install: docker pull ncabatoff/process-exporter
        • run:
          • root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v pwd:/config ncabatoff/process-exporter --procfs /host/proc
        • => http://dolos.nomics.world:9256/metrics
      • add to gitlab conf + reload gitlab conf '''job_name': 'process-exporter-dolos', 'static_configs' => [ 'targets' => ['dolos.nomics.world:9256'], ],`
      • dashboard: https://grafana.com/grafana/dashboards/249
    • view requests count to DBnomics web API
      • uWSGI Exporter https://github.com/timonwong/uwsgi_exporter
        • docker pull timonwong/uwsgi-exporter
        • docker run -d -p 9117:9117 timonwong/uwsgi-exporter
          • => ça ne se lance pas
        • docker logs a6ee70d98c61ac22b124334eb31d364adc0a8daded4de263acb376fd952cf35a
          • leve =fatal msg="Incompatible uri " source="exporter.go:136"
        • https://github.com/timonwong/uwsgi_exporter/issues/32
        • => missing --stats.uri param
          • what to gives to this param ?
          • /etc/uwsgi/apps-enabled/dbnomics-api-uwsgi-v22.ini
            • stats = :8002
        • so let's try
          • `docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'''
          • => it works ☺
            • http://dolos.nomics.world:9117/metrics
      • add to promethus
        • existing dashboard: https://grafana.com/grafana/dashboards/3435
          • => oh no, it's for the other one: https://github.com/AndreaGreco/prometeus_uwsgi_exporter
      • how to graph ?
        • en existing dashboard exists but for the other exporter
          • => I can't make it run on Dolos (GOPATH variable problem)
        • let pick some values that seems interesting and graph them
          • go_memstats_alloc_bytes{jo ="uwsgi-exporter-dolos"}
          • rate(process_cpu_seconds_total{instanc =~"localhost:9100"}[2m])
          • rate(http_requests_total{instanc ="dolos.nomics.world:9117"}[10m])
  • custom folders size

    • via node-exporter
      • dolos
        • https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector
        • dans le fichier de conf: /var/lib/prometheus/node-exporter
           #  --collector.textfile.director ="/var/lib/prometheus/node-exporter"
           #                            Directory to read text files with metrics from.
        • test: ajout d'un fichier test contenant: test_bduye 0
          • => OK
        • vrais stats
        • systemd service: echo "# Generated on $(date)" > /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && du -sb /home/gitlab-runner/json-data/*-json-data* | sed -ne "s/^\([0-9]\+\)\t\(.*\)$/node_directory_size_bytes{director ="\2"} \1/p" >> /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && mv /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom
      • eros
  • déploiement

    • comment faire pour avoir de jolis labels ?
      • label_replace(v instant-vector, dst_label string, replacement string, src_label string, "")
        • => impossible: https://github.com/grafana/grafana/pull/6627
Clone repository
  • Code style
  • Git and Gitlab workflow
  • acceptance criteria
    • fetchers
  • ci jobs and runners
  • code optimization
  • dev tools
  • e mails
  • failure handling procedures
  • Home
  • librairies
  • maintaining fetchers
  • monitoring
  • presentation
  • production configuration
  • publishing python packages
View All Pages