Skip to content

GitLab

D
documentation

troubleshooting

Last edited by Christophe Benz
Page history

To connect to servers, either use the machine domain name, or the service domain name:

ssh cepremap@dolos.nomics.world # or db.nomics.world
ssh cepremap@ioke.nomics.world # or dev.db.nomics.world
ssh cepremap@eros.nomics.world # or pre.db.nomics.world

What to do if ...

Solr is down

When the website or the API displays "Internal server error" (see below) it may be useful to restart Solr (indexer), uwsgi (DBnomics API) and Nginx (web server) on dolos (as root):

systemctl restart solr.service
systemctl restart uwsgi.service
systemctl restart nginx.service

Note: always restart uwsgi and nginx after solr otherwise the web API process could keep dangling connections and fail to connect to the new Solr process (I don't understand exactly what happens).

Disk is full

Docker

Note: the following actions are done every night by a systemd timer (see dolos doc and eros doc)

If a server uses Docker, sometimes the disk is full because of old Docker containers and volumes. This can be cleaned like so:

docker system prune

Check that unused volumes were deleted also (otherwise try with --volumes option).

This operation requires the unix user to root, or be in the docker group (for Debian).

Website displays "Internal server error"

This message is either caused by:

  • Solr is down (see previous section)
  • wrong data (or no data at all) fetched by the UI (from the API) causing a problem rendering the page
  • a bug in the UI

The first thing to do is to check if the API is OK when loading the page. See section "Find errors in Web API" to do that.

If the API does not fail, the problem is in the UI. To find its source, open the JavaScript console of the browser, and look for a stack trace. The reproduce locally, fix and deploy (see README of dbnomics-website).

Web API returns errors

The error is probably caused by a Python exception in dbnomics-api. The exception is not visible in the JSON response of the URL, but is visible in the application log on the server.

See also: dolos server install doc and production configuration.

Search for errors in the logs (dbnomics-api is deployed using uwsgi so the logs are in /var/log/uwsgi/app/):

ssh root@api.db.nomics.world
tail -f /var/log/uwsgi/app/dbnomics-api-uwsgi-v22.log

If you find an internal server error (HTTP status code 500): reproduce locally, fix and deploy (see README of dbnomics-api).

If you find a connection error to Solr (i.e. urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8983): Max retries exceeded with url: /solr/dbnomics/query):

systemctl restart solr.service
systemctl restart uwsgi.service

I can't see a dataset on the website

Example: I open https://db.nomics.world/SCSMICH/MICS and it is "404 not found".