... | ... | @@ -6,3 +6,230 @@ Eurostat | ~5000 | 123G | 269G | ? |
|
|
|
|
|
Notes:
|
|
|
- disk usage includes `.git` directory
|
|
|
|
|
|
## Prometheus metrics
|
|
|
|
|
|
- prometheus is running on Eros, shipped with Gitlab, and collects metrics exposed in Eros and Dolos
|
|
|
- metrics are text files, updated regularly my deamons
|
|
|
- Grafana is an UI for viewing metrics, shipped with Gitlab, available here: https://git.nomics.world/-/grafana/
|
|
|
|
|
|
### Services Summary
|
|
|
|
|
|
#### Eros
|
|
|
|
|
|
* Gitlab
|
|
|
* service
|
|
|
* Prometheus (via Gitlab)
|
|
|
* metrics: http://localhost:9090/
|
|
|
* Grafana (via Gitlab)
|
|
|
* metrics: https://git.nomics.world/-/grafana/
|
|
|
* auto-generated conf file: **/var/opt/gitlab/grafana/grafana.ini**
|
|
|
* node-exporter (via GItlab)
|
|
|
* metrics: http://localhost:9100/
|
|
|
* auto-generated conf file: **/var/opt/gitlab/**
|
|
|
* Howto edit conf and re-generate conf files:
|
|
|
* vi `/etc/gitlab/gitlab.rb`
|
|
|
* `gitlab-ctl reconfigure`
|
|
|
* don't forget to restart services (ex: `gitlab-ctl restart grafana)`
|
|
|
* Other services
|
|
|
* systemd timer: folder-sizes-metrics
|
|
|
* metrics: dans node-exporter: http://localhost:9100/
|
|
|
* clés: node_directory_size_bytes
|
|
|
|
|
|
#### Dolos
|
|
|
|
|
|
* uwsgi
|
|
|
* https://github.com/timonwong/uwsgi_exporter
|
|
|
* installed: via Docker (`timonwong/uwsgi-exporter)`
|
|
|
* start: `docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002`'
|
|
|
* metrics: http://dolos.nomics.world:9117/metrics
|
|
|
* solr-exporter
|
|
|
* shipped with Solr: `/opt/solr/contrib/prometheus-exporter/bin/solr-exporter`
|
|
|
* start: root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
|
|
|
* metrics: http://dolos.nomics.world:9854
|
|
|
* node-exporter
|
|
|
* https://github.com/prometheus/node_exporter
|
|
|
* installed via Debian `prometheus-node-exporter (stretch-backports)`
|
|
|
* start: systemd service: prometheus-node-exporter
|
|
|
* conf: /etc/default/prometheus-node-exporter
|
|
|
* metrics: http://dolos.nomics.world:9100/metrics
|
|
|
* process-exporter `https://github.com/ncabatoff/process-exporter`
|
|
|
* https://github.com/ncabatoff/process-exporter
|
|
|
* installed via Docker
|
|
|
* start: `root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc`
|
|
|
* metrics: http://dolos.nomics.world:9256/metrics
|
|
|
* folder-sizes-metrics
|
|
|
* systemd timer: folder-sizes-metrics
|
|
|
* metrics via node-exporter: http://dolos.nomics.world:9100/metrics
|
|
|
* clés: node_directory_size_bytes
|
|
|
* custom metrics (python script)
|
|
|
* systemd timer: dbnomics-custom-metrics
|
|
|
* metrics via node-exporter: http://dolos.nomics.world:9100/metrics
|
|
|
* clés:
|
|
|
* dbnomics_datasets_total
|
|
|
* dbnomics_series_total
|
|
|
* dbnomics_providers_total
|
|
|
* dbnomics_running_jobs
|
|
|
* dbnomics_running_jobs_total
|
|
|
|
|
|
### Installation notes
|
|
|
|
|
|
#### Gitlab metrics
|
|
|
* https://git.nomics.world/admin/application_settings/metrics_and_profiling
|
|
|
* Metrics - Prometheus -> enabled
|
|
|
* metrics: https://git.nomics.world/-/metrics?toke =Vy7HgyXTVc5hiHqnkyxJ
|
|
|
* => access token deprecated https://docs.gitlab.com/ce/user/admin_area/monitoring/health_check.html#access-token-deprecated
|
|
|
|
|
|
#### Gitlab prometheus
|
|
|
* by default listening on 9090
|
|
|
root@eros:~# netstat -apn | grep 9090
|
|
|
tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN 39910/prometheus
|
|
|
* connection à prometheus via l'UI
|
|
|
* `ssh -N -L 9090:localhost:9090 eros`
|
|
|
* http://localhost:9090/
|
|
|
* endpoints: https://git.nomics.world/help/administration/monitoring/prometheus/index#gitlab-metrics
|
|
|
* node -> infos système
|
|
|
|
|
|
#### Grafana
|
|
|
* https://docs.gitlab.com/omnibus/settings/grafana.html
|
|
|
* https://git.nomics.world/-/grafana/
|
|
|
* admin / WqDbwBcmmNk4J4AJ
|
|
|
* changed all Jailbreak team members to admin rights
|
|
|
* "Failed to get settings" bug makes it impossible to work
|
|
|
* how to access to grafana logs throught Gitlab ?
|
|
|
* `gitlab-ctl tail grafana`
|
|
|
* on voit plein de `logge =context error="database is locked"`
|
|
|
* **/var/opt/gitlab/grafana/grafana.ini**
|
|
|
* DB: **/var/opt/gitlab/grafana/data/grafana.db**
|
|
|
* let's reinit DB
|
|
|
* `gitlab-ctl stop grafana`
|
|
|
* `mv /var/opt/gitlab/grafana/data/grafana.db{,.bck}`
|
|
|
* Add to grafana.ini:
|
|
|
'''
|
|
|
[database]
|
|
|
type=sqlite3
|
|
|
connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL
|
|
|
'''
|
|
|
* `gitlab-ctl start grafana`
|
|
|
* `/var/opt/gitlab/grafana/data/grafana.db` have been recreated
|
|
|
* => seems to work great ☺
|
|
|
* copy paste panel
|
|
|
* trying to copy/paste Gitlab node "Node Available Memory"
|
|
|
* it doesn't works as it. In formula there's variable $node that's not defined in the other context, I suppose. Replacing with localhost makes it works:
|
|
|
'''
|
|
|
(
|
|
|
) / node_memory_MemTotal_bytes{instanc =~"localhost:9100"}
|
|
|
'''
|
|
|
|
|
|
#### Solr
|
|
|
* https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html
|
|
|
* solr-exporter is here: [[/opt/solr/contrib/prometheus-exporter/bin]]
|
|
|
* working dir: `/opt/solr/contrib/prometheus-exporter`
|
|
|
* conf: `./conf/solr-exporter-config.xml`
|
|
|
* starting:
|
|
|
* root@dolos:/opt/solr/contrib/prometheus-exporter# ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8
|
|
|
* => http://dolos.nomics.world:9854
|
|
|
* Adding http://dolos.nomics.world:9854 to Grafana
|
|
|
* is there a pre-configured dashboard ?
|
|
|
* yes: https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#sample-grafana-dashboard
|
|
|
* https://github.com/apache/lucene-solr/blob/master/solr/contrib/prometheus-exporter/conf/grafana-solr-dashboard.json
|
|
|
* imported this dashboard
|
|
|
* => errors. I sould have added it to Prometeus
|
|
|
* adding to prometheus
|
|
|
* https://lucene.apache.org/solr/guide/8_1/monitoring-solr-with-prometheus-and-grafana.html#prometheus-configuration
|
|
|
* vi [[/etc/gitlab/gitlab.rb]]
|
|
|
* `gitlab-ctl reconfigure`
|
|
|
* => I can see it on promotheus: http://localhost:9090/targets
|
|
|
* I can access to data from prometheus:
|
|
|
* rate(solr_metrics_core_requests_total[30m])
|
|
|
* I can finaly access this data from "explore" tool in Grafana
|
|
|
* rate(solr_metrics_core_requests_total{categor ="QUERY", handler="/query"}[5m])
|
|
|
|
|
|
#### Dolos node exporter
|
|
|
* https://github.com/prometheus/node_exporter
|
|
|
* `apt install golang-go `
|
|
|
* `go get github.com/prometheus/node_exporter`
|
|
|
* `package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)`
|
|
|
* FInaly, contrary to what apt search node_exporter search gave me, there's a Debian package :s
|
|
|
* `apt install prometheus-node-exporter`
|
|
|
* => systemd service
|
|
|
* listening on :9100
|
|
|
* => http://dolos.nomics.world:9100/metrics
|
|
|
* in promoethus conf:
|
|
|
`},`
|
|
|
` 'job_name': 'Dolos node',`
|
|
|
` 'static_configs' => [`
|
|
|
` 'targets' => ['dolos.nomics.world:9100'],`
|
|
|
` ],`
|
|
|
` },`
|
|
|
* intersting exported metrics
|
|
|
* node_filesystem_free
|
|
|
* in promotheus: `node_filesystem_free{devic ="/dev/md1"} => 173293019136`
|
|
|
* but df -h reports 114G free 🤔
|
|
|
* import pre-configured dashboard: https://grafana.com/grafana/dashboards/1860
|
|
|
* => doesn't show anything. Why ?
|
|
|
* example: node_cpu_seconds_total{instanc =~"$node:$port",job=~"$job"}
|
|
|
* maybe because job name = "node"
|
|
|
* https://github.com/rfrail3/grafana-dashboards#node-exporter-full
|
|
|
* looks like Debian version is vveerryyy old
|
|
|
* trying to install from stretch-backports
|
|
|
* `root@dolos:~# apt install -t stretch-backports prometheus-node-exporter`
|
|
|
* => yes ☺
|
|
|
* **view RAM per process**
|
|
|
* https://github.com/ncabatoff/process-exporter
|
|
|
* install: docker pull ncabatoff/process-exporter
|
|
|
* run:
|
|
|
* `root@dolos:~# docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc`
|
|
|
* => http://dolos.nomics.world:9256/metrics
|
|
|
* add to gitlab conf + reload gitlab conf
|
|
|
'''job_name': 'process-exporter-dolos',`
|
|
|
` 'static_configs' => [`
|
|
|
` 'targets' => ['dolos.nomics.world:9256'],`
|
|
|
` ],`
|
|
|
* dashboard: https://grafana.com/grafana/dashboards/249
|
|
|
* **view requests count to DBnomics web API**
|
|
|
* uWSGI Exporter https://github.com/timonwong/uwsgi_exporter
|
|
|
* `docker pull timonwong/uwsgi-exporter`
|
|
|
* `docker run -d -p 9117:9117 timonwong/uwsgi-exporter`
|
|
|
* => ça ne se lance pas
|
|
|
* docker logs a6ee70d98c61ac22b124334eb31d364adc0a8daded4de263acb376fd952cf35a
|
|
|
* `leve =fatal msg="Incompatible uri " source="exporter.go:136"`
|
|
|
* https://github.com/timonwong/uwsgi_exporter/issues/32
|
|
|
* => missing --stats.uri param
|
|
|
* what to gives to this param ?
|
|
|
* in [[/etc/uwsgi/apps-enabled/dbnomics-api-uwsgi-v22.ini]] we see
|
|
|
* stats = :8002
|
|
|
* so let's try
|
|
|
* `docker run -d -p 9117:9117 timonwong/uwsgi-exporter --stats.uri 'http://localhost:8002'''
|
|
|
* => it works ☺
|
|
|
* http://dolos.nomics.world:9117/metrics
|
|
|
* add to promethus
|
|
|
* existing dashboard: https://grafana.com/grafana/dashboards/3435
|
|
|
* => oh no, it's for the other one: https://github.com/AndreaGreco/prometeus_uwsgi_exporter
|
|
|
* how to graph ?
|
|
|
* en existing dashboard exists but for the other exporter
|
|
|
* => I can't make it run on Dolos (GOPATH variable problem)
|
|
|
* let pick some values that seems interesting and graph them
|
|
|
* go_memstats_alloc_bytes{jo ="uwsgi-exporter-dolos"}
|
|
|
* rate(process_cpu_seconds_total{instanc =~"localhost:9100"}[2m])
|
|
|
* rate(http_requests_total{instanc ="dolos.nomics.world:9117"}[10m])
|
|
|
|
|
|
|
|
|
* custom folders size
|
|
|
* via node-exporter
|
|
|
* **dolos**
|
|
|
* https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector
|
|
|
* dans le fichier de conf: /var/lib/prometheus/node-exporter
|
|
|
# --collector.textfile.director ="/var/lib/prometheus/node-exporter"
|
|
|
# Directory to read text files with metrics from.
|
|
|
* test: ajout d'un fichier test contenant:
|
|
|
test_bduye 0
|
|
|
* => OK
|
|
|
* vrais stats
|
|
|
* systemd service:
|
|
|
`echo "# Generated on $(date)" > /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && du -sb /home/gitlab-runner/json-data/*-json-data* | sed -ne "s/^\([0-9]\+\)\t\(.*\)$/node_directory_size_bytes{director ="\2"} \1/p" >> /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ && mv /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom.$$ /var/lib/prometheus/node-exporter/gitlab-runner-sizes.prom`
|
|
|
* **eros**
|
|
|
* déploiement
|
|
|
* comment faire pour avoir de jolis labels ?
|
|
|
* label_replace(v instant-vector, dst_label string, replacement string, src_label string, "")
|
|
|
* => impossible: https://github.com/grafana/grafana/pull/6627 |