Metrics about providers
Provider | Nb datasets | Disk usage (JSON data) | Disk usage (source data) | Download time | Convert time |
---|---|---|---|---|---|
Eurostat | ~5000 | 123G | 269G | ? | ~1.5d |
Notes:
- disk usage includes
.git
directory
Prometheus metrics
- prometheus is running on Eros, shipped with Gitlab, and collects metrics exposed in Eros and Dolos
- metrics are text files, updated regularly my deamons
- Grafana is an UI for viewing metrics, shipped with Gitlab, available here: https://git.nomics.world/-/grafana/
Services Summary
Eros
- Gitlab
- service
- Prometheus (via Gitlab)
- metrics: http://localhost:9090/
- Grafana (via Gitlab)
- metrics: https://git.nomics.world/-/grafana/
- auto-generated conf file: /var/opt/gitlab/grafana/grafana.ini
- node-exporter (via GItlab)
- metrics: http://localhost:9100/
- auto-generated conf file: /var/opt/gitlab/
- Prometheus (via Gitlab)
- Howto edit conf and re-generate conf files:
- vi
/etc/gitlab/gitlab.rb
gitlab-ctl reconfigure
- don't forget to restart services (ex:
gitlab-ctl restart grafana)
- vi
- service
- Other services
- systemd timer: folder-sizes-metrics
- metrics: dans node-exporter: http://localhost:9100/
- clés: node_directory_size_bytes
- metrics: dans node-exporter: http://localhost:9100/
- systemd timer: folder-sizes-metrics
Dolos
- uwsgi
- https://github.com/timonwong/uwsgi_exporter
- installed: via Docker (
timonwong/uwsgi-exporter)
- start:
docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002
' - metrics: http://dolos.nomics.world:9117/metrics
- solr-exporter
- shipped with Solr:
/opt/solr/contrib/prometheus-exporter/bin/solr-exporter
- start: root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
- metrics: http://dolos.nomics.world:9854
- shipped with Solr:
- node-exporter
- https://github.com/prometheus/node_exporter
- installed via Debian
prometheus-node-exporter (stretch-backports)
- start: systemd service: prometheus-node-exporter
- conf: /etc/default/prometheus-node-exporter
- metrics: http://dolos.nomics.world:9100/metrics
- process-exporter
- https://github.com/ncabatoff/process-exporter
- installed via Docker
- start:
root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v
pwd:/config ncabatoff/process-exporter --procfs /host/proc
- metrics: http://dolos.nomics.world:9256/metrics
- custom metrics (python script)
- https://git.nomics.world/dbnomics/dbnomics-metrics
- folder-sizes-metrics
- systemd timer: folder-sizes-metrics
- metrics via node-exporter: http://dolos.nomics.world:9100/metrics
- clés: node_directory_size_bytes
- DBnomics metrics (Solr counts + Gitlab jobs)
- systemd timer: dbnomics-custom-metrics
- metrics via node-exporter: http://dolos.nomics.world:9100/metrics
- clés:
- dbnomics_datasets_total
- dbnomics_series_total
- dbnomics_providers_total
- dbnomics_running_jobs
- dbnomics_running_jobs_total
- clés:
- metrics via node-exporter: http://dolos.nomics.world:9100/metrics
- systemd timer: dbnomics-custom-metrics
Installation notes
Gitlab metrics
-
https://git.nomics.world/admin/application_settings/metrics_and_profiling
- Metrics - Prometheus -> enabled
- metrics: https://git.nomics.world/-/metrics?toke =Vy7HgyXTVc5hiHqnkyxJ
- => access token deprecated https://docs.gitlab.com/ce/user/admin_area/monitoring/health_check.html#access-token-deprecated
Gitlab prometheus
- by default listening on 9090 root@eros:~# netstat -apn | grep 9090 tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN 39910/prometheus
- connection à prometheus via l'UI
ssh -N -L 9090:localhost:9090 eros
- http://localhost:9090/
- endpoints: https://git.nomics.world/help/administration/monitoring/prometheus/index#gitlab-metrics
- node -> infos système
Grafana
-
https://docs.gitlab.com/omnibus/settings/grafana.html
-
https://git.nomics.world/-/grafana/
- admin / password in jailbreak keypassx file
- changed all Jailbreak team members to admin rights
-
https://git.nomics.world/-/grafana/
- "Failed to get settings" bug makes it impossible to work
- how to access to grafana logs throught Gitlab ?
gitlab-ctl tail grafana
- on voit plein de
logge =context error="database is locked"
- /var/opt/gitlab/grafana/grafana.ini
- DB: /var/opt/gitlab/grafana/data/grafana.db
- let's reinit DB
gitlab-ctl stop grafana
mv /var/opt/gitlab/grafana/data/grafana.db{,.bck}
- Add to grafana.ini: ''' [database] type=sqlite3 connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL '''
gitlab-ctl start grafana
-
/var/opt/gitlab/grafana/data/grafana.db
have been recreated
- => seems to work great
☺
- let's reinit DB
- how to access to grafana logs throught Gitlab ?
- copy paste panel
- trying to copy/paste Gitlab node "Node Available Memory"
- it doesn't works as it. In formula there's variable $node that's not defined in the other context, I suppose. Replacing with localhost makes it works: ''' ( ) / node_memory_MemTotal_bytes{instanc =~"localhost:9100"} '''
- trying to copy/paste Gitlab node "Node Available Memory"
Solr
- https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html
-
/opt/solr/contrib/prometheus-exporter/bin
- working dir:
/opt/solr/contrib/prometheus-exporter
- conf:
./conf/solr-exporter-config.xml
- starting:
- root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
- => http://dolos.nomics.world:9854
- Adding http://dolos.nomics.world:9854 to Grafana
- is there a pre-configured dashboard ?
- => errors. I sould have added it to Prometeus
- adding to prometheus
-
https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#prometheus-configuration
- /etc/gitlab/gitlab.rb
gitlab-ctl reconfigure
- => I can see it on promotheus: http://localhost:9090/targets
-
https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#prometheus-configuration
- I can access to data from prometheus:
- rate(solr_metrics_core_requests_total[30m])
- I can finaly access this data from "explore" tool in Grafana
- rate(solr_metrics_core_requests_total{categor ="QUERY", handler="/query"}[5m])
- working dir:
Dolos node exporter
-
https://github.com/prometheus/node_exporter
apt install golang-go
-
go get github.com/prometheus/node_exporter
package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
- FInaly, contrary to what apt search node_exporter search gave me, there's a Debian package :s
apt install prometheus-node-exporter
- => systemd service
- listening on :9100
- in promoethus conf:
},
'job_name': 'Dolos node',
'static_configs' => [
'targets' => ['dolos.nomics.world:9100'],
],
},
- intersting exported metrics
- node_filesystem_free
- in promotheus:
node_filesystem_free{devic ="/dev/md1"} => 173293019136
- but df -h reports 114G free
🤔
- in promotheus:
- node_filesystem_free
- import pre-configured dashboard: https://grafana.com/grafana/dashboards/1860
- => doesn't show anything. Why ?
- example: node_cpu_seconds_total{instanc =
"$node:$port",job="$job"} - maybe because job name = "node"
- example: node_cpu_seconds_total{instanc =
- looks like Debian version is vveerryyy old
- => doesn't show anything. Why ?
- trying to install from stretch-backports
root@dolos:~# apt install -t stretch-backports prometheus-node-exporter
- => yes
☺
-
view RAM per process
-
https://github.com/ncabatoff/process-exporter
- install: docker pull ncabatoff/process-exporter
- run:
-
root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v
pwd:/config ncabatoff/process-exporter --procfs /host/proc
-
- => http://dolos.nomics.world:9256/metrics
- add to gitlab conf + reload gitlab conf
'''job_name': 'process-exporter-dolos',
- dashboard: https://grafana.com/grafana/dashboards/249
-
https://github.com/ncabatoff/process-exporter
-
view requests count to DBnomics web API
- uWSGI Exporter https://github.com/timonwong/uwsgi_exporter
docker pull timonwong/uwsgi-exporter
-
docker run -d -p 9117:9117 timonwong/uwsgi-exporter
- => ça ne se lance pas
- docker logs a6ee70d98c61ac22b124334eb31d364adc0a8daded4de263acb376fd952cf35a
leve =fatal msg="Incompatible uri " source="exporter.go:136"
- https://github.com/timonwong/uwsgi_exporter/issues/32
- => missing --stats.uri param
- what to gives to this param ?
-
/etc/uwsgi/apps-enabled/dbnomics-api-uwsgi-v22.ini
- stats = :8002
- so let's try
- `docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'''
- => it works
☺
- add to promethus
- existing dashboard: https://grafana.com/grafana/dashboards/3435
- => oh no, it's for the other one: https://github.com/AndreaGreco/prometeus_uwsgi_exporter
- existing dashboard: https://grafana.com/grafana/dashboards/3435
- how to graph ?
- en existing dashboard exists but for the other exporter
- => I can't make it run on Dolos (GOPATH variable problem)
- let pick some values that seems interesting and graph them
- go_memstats_alloc_bytes{jo ="uwsgi-exporter-dolos"}
- rate(process_cpu_seconds_total{instanc =~"localhost:9100"}[2m])
- rate(http_requests_total{instanc ="dolos.nomics.world:9117"}[10m])
- en existing dashboard exists but for the other exporter
- uWSGI Exporter https://github.com/timonwong/uwsgi_exporter
-
custom folders size
- via node-exporter
-
dolos
- https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector
- dans le fichier de conf: /var/lib/prometheus/node-exporter
# --collector.textfile.director ="/var/lib/prometheus/node-exporter" # Directory to read text files with metrics from.
- test: ajout d'un fichier test contenant:
test_bduye 0
- => OK
- vrais stats
- systemd service:
echo "# Generated on $(date)" > /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && du -sb /home/gitlab-runner/json-data/*-json-data* | sed -ne "s/^\([0-9]\+\)\t\(.*\)$/node_directory_size_bytes{director ="\2"} \1/p" >> /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && mv /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom
- eros
-
dolos
- via node-exporter
-
déploiement
- comment faire pour avoir de jolis labels ?
- label_replace(v instant-vector, dst_label string, replacement string, src_label string, "")
- => impossible: https://github.com/grafana/grafana/pull/6627
- label_replace(v instant-vector, dst_label string, replacement string, src_label string, "")
- comment faire pour avoir de jolis labels ?