troubleshooting

To connect to servers, either use the machine domain name, or the service domain name:

ssh cepremap@dolos.nomics.world # or db.nomics.world
ssh cepremap@ioke.nomics.world # or dev.db.nomics.world
ssh cepremap@eros.nomics.world # or pre.db.nomics.world

What to do if ...

Solr is down

When the website or the API displays "Internal server error" (see below) it may be useful to restart Solr (indexer), uwsgi (DBnomics API) and Nginx (web server) on dolos (as root):

systemctl restart solr.service
systemctl restart uwsgi.service
systemctl restart nginx.service

Note: always restart uwsgi and nginx after solr otherwise the web API process could keep dangling connections and fail to connect to the new Solr process (I don't understand exactly what happens).

Disk is full

Docker

Note: the following actions are done every night by a systemd timer (see dolos doc and eros doc)

If a server uses Docker, sometimes the disk is full because of old Docker containers and volumes. This can be cleaned like so:

docker system prune

Check that unused volumes were deleted also (otherwise try with --volumes option).

This operation requires the unix user to root, or be in the docker group (for Debian).

Website displays "Internal server error"

This message is either caused by:

Solr is down (see previous section)
wrong data (or no data at all) fetched by the UI (from the API) causing a problem rendering the page
a bug in the UI

The first thing to do is to check if the API is OK when loading the page. See section "Find errors in Web API" to do that.

If the API does not fail, the problem is in the UI. To find its source, open the JavaScript console of the browser, and look for a stack trace. The reproduce locally, fix and deploy (see README of dbnomics-website).

Web API returns errors

The error is probably caused by a Python exception in dbnomics-api. The exception is not visible in the JSON response of the URL, but is visible in the application log on the server.

Search for errors in the logs (dbnomics-api is deployed using uwsgi so the logs are in /var/log/uwsgi/app/):

ssh root@api.db.nomics.world
tail -f /var/log/uwsgi/app/dbnomics-api-uwsgi-v22.log

If you find an internal server error (HTTP status code 500): reproduce locally, fix and deploy (see README of dbnomics-api).

If you find a connection error to Solr (i.e. urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8983): Max retries exceeded with url: /solr/dbnomics/query):

systemctl restart solr.service
systemctl restart uwsgi.service

I can't see a dataset on the website

Example: I open https://db.nomics.world/SCSMICH/MICS and it is "404 not found".

check it is indexed in Solr
- in production:
  - ssh -N -L 8983:localhost:8983 cepremap@dolos.nomics.world
- for pre-production:
  - ssh -N -L 8983:localhost:8983 cepremap@eros.nomics.world
- open http://localhost:8983/solr/dbnomics/query?q=type:dataset%20AND%20provider_code:SCSMICH%20AND%20code:MICS (the Solr query is more readable as "type:dataset AND provider_code:SCSMICH AND code:MICS")
if a JSON document is displayed, it's OK
else
- before reindexing check that there is disk space on the server
- try to index in full mode (not incremental mode): see Solr indexation

Jobs stuck / Runner not responding

A runner is considered as stuck when the time of last contact is longer than one hour. You can find this information in the admin panel, by looking at the "Last contact" column in list of the runners registered for the Gitlab instance you're working with, for example: https://git.nomics.world/admin/runners.

First, connect to the server on which the runner is installed, for example:

ssh root@163.172.67.63

Then restart the runner service:

systemctl restart gitlab-runner.service

Gitlab should be able to "contact" the runner again and the jobs can be restarted as normal.