To connect to servers, either use the machine domain name, or the service domain name:
ssh firstname.lastname@example.org # or db.nomics.world ssh email@example.com # or dev.db.nomics.world ssh firstname.lastname@example.org # or pre.db.nomics.world
What to do if ...
Solr is down
When the website or the API displays "Internal server error" (see below) it may be useful to restart Solr (indexer), uwsgi (DBnomics API) and Nginx (web server) on dolos (as root):
systemctl restart solr.service systemctl restart uwsgi.service systemctl restart nginx.service
Note: always restart
solr otherwise the web API process could keep dangling connections and fail to connect to the new Solr process (I don't understand exactly what happens).
Disk is full
If a server uses Docker, sometimes the disk is full because of old Docker containers and volumes. This can be cleaned like so:
docker system prune
Check that unused volumes were deleted also (otherwise try with
This operation requires the unix user to
root, or be in the
docker group (for Debian).
Website displays "Internal server error"
This message is either caused by:
- Solr is down (see previous section)
- wrong data (or no data at all) fetched by the UI (from the API) causing a problem rendering the page
- a bug in the UI
The first thing to do is to check if the API is OK when loading the page. See section "Find errors in Web API" to do that.
Web API returns errors
The error is probably caused by a Python exception in dbnomics-api. The exception is not visible in the JSON response of the URL, but is visible in the application log on the server.
Search for errors in the logs (dbnomics-api is deployed using uwsgi so the logs are in
ssh email@example.com tail -f /var/log/uwsgi/app/dbnomics-api-uwsgi-v22.log
If you find an internal server error (HTTP status code 500): reproduce locally, fix and deploy (see README of dbnomics-api).
If you find a connection error to Solr (i.e.
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8983): Max retries exceeded with url: /solr/dbnomics/query):
systemctl restart solr.service systemctl restart uwsgi.service
I can't see a dataset on the website
Example: I open https://db.nomics.world/SCSMICH/MICS and it is "404 not found".
- check it is indexed in Solr
- in production:
ssh -N -L 8983:localhost:8983 firstname.lastname@example.org
- for pre-production:
ssh -N -L 8983:localhost:8983 email@example.com
- open http://localhost:8983/solr/dbnomics/query?q=type:dataset%20AND%20provider_code:SCSMICH%20AND%20code:MICS (the Solr query is more readable as
"type:dataset AND provider_code:SCSMICH AND code:MICS")
- in production:
- if a JSON document is displayed, it's OK
- before reindexing check that there is disk space on the server
- try to index in full mode (not incremental mode): see Solr indexation
Jobs stuck / Runner not responding
A runner is considered as stuck when the time of last contact is longer than one hour. You can find this information in the admin panel, by looking at the "Last contact" column in list of the runners registered for the Gitlab instance you're working with, for example: https://git.nomics.world/admin/runners.
First, connect to the server on which the runner is installed, for example:
Then restart the runner service:
systemctl restart gitlab-runner.service
Gitlab should be able to "contact" the runner again and the jobs can be restarted as normal.