... | ... | @@ -2,13 +2,68 @@ |
|
|
|
|
|
How to deal with failures in production? What to do if something happens?
|
|
|
|
|
|
## Criticity level
|
|
|
|
|
|
Depending on what is affected, the criticity level of the failure is different:
|
|
|
|
|
|
- UI or API: critical
|
|
|
- data or jobs:
|
|
|
- star-provider (see the [dashboard](https://db.nomics.world/dashboard/)): urgent
|
|
|
- other providers: normal
|
|
|
|
|
|
Critical failures MUST be reported via a SMS, e-mail, chat message or direct call to the person in charge of the maintenance, besides the normal procedure (see sections below).
|
|
|
|
|
|
## Monitoring
|
|
|
|
|
|
TODO
|
|
|
|
|
|
## Failure with UI
|
|
|
|
|
|
## Failure with API
|
|
|
When something goes wrong with production [UI](https://db.nomics.world/). This does not includes problems with data (see other section).
|
|
|
|
|
|
Typical symptoms:
|
|
|
|
|
|
- "Error loading page" message displayed on the web page. This means that the data needed to initialize the page could not be fetched by the UI.
|
|
|
|
|
|
How to report bug: create an issue on the [board](https://git.nomics.world/dbnomics-fetchers/management/boards):
|
|
|
|
|
|
- labels: "UI"
|
|
|
- mentioning the URL showing the problem in the description
|
|
|
- a description of what is wrong, and what was expected
|
|
|
- optional: a screenshot or copy/paste of the error
|
|
|
|
|
|
How to investigate: see [troubleshooting](./troubleshooting)
|
|
|
|
|
|
## Failure with data
|
|
|
|
|
|
When something goes wrong with data, as seen in the production [UI](https://db.nomics.world/) or any other programming language package (like [DBnomics-Python](https://pypi.org/project/DBnomics/)).
|
|
|
|
|
|
Typical symptoms:
|
|
|
|
|
|
- data seems incomplete or wrong
|
|
|
|
|
|
How to report bug: create an issue on the [board](https://git.nomics.world/dbnomics-fetchers/management/boards):
|
|
|
|
|
|
- title starting by the name of the provider
|
|
|
- labels: "Maintenance" and "Fetcher"
|
|
|
- mentioning the URL showing the problem in the description
|
|
|
- a description of what is wrong, and what was expected
|
|
|
|
|
|
## Failure with jobs
|
|
|
|
|
|
## Failure with Solr
|
|
|
When something goes wrong with download, conversion or indexation jobs, visible from the [dashboard](https://db.nomics.world/dashboard/).
|
|
|
|
|
|
Typical symptoms:
|
|
|
|
|
|
- job status is "failed"
|
|
|
|
|
|
How to report bug: create an issue on the [board](https://git.nomics.world/dbnomics-fetchers/management/boards):
|
|
|
|
|
|
- title starting by the name of the provider
|
|
|
- labels: "Maintenance" and "Fetcher"
|
|
|
- mentioning the URL of the failed job in the description
|
|
|
|
|
|
How to investigate:
|
|
|
|
|
|
- Open the job URL; the following depends on what you'll see.
|
|
|
- Maybe use a [pre-production fetcher environment](test-fetcher-on-pre-prod) |