README.md 4.28 KB
Newer Older
Christophe Benz's avatar
Christophe Benz committed
1
# DBnomics GitLab-CI
2

Christophe Benz's avatar
Christophe Benz committed
3 4
**Warning**: if you're here to deploy a new fetcher to production, you must now use https://git.nomics.world/dbnomics/dbnomics-fetcher-ops

5
This repository contains scripts interacting with the GitLab Continuous Integration of DBnomics.
6

Christophe Benz's avatar
Christophe Benz committed
7
It also defines a `Dockerfile` allowing Docker Hub to build a container image for us.
Christophe Benz's avatar
Christophe Benz committed
8
Unfortunately, Docker Hub is only compatible with GitHub for now, and DBnomics is hosted on its own [GitLab platform](https://git.nomics.world/). That's why we created a mirror of this project [on GitHub](https://github.com/dbnomics/dbnomics-gitlab-ci), but the real home is [on DBnomics GitLab](https://git.nomics.world/dbnomics/dbnomics-gitlab-ci).
Christophe Benz's avatar
Christophe Benz committed
9

Christophe Benz's avatar
Christophe Benz committed
10
The Docker image is referenced in `.gitlab-ci.yml` files of each fetcher, with the line `image: dbnomics/dbnomics-gitlab-ci:latest`.
Christophe Benz's avatar
Christophe Benz committed
11

12
## Configuration
13

14 15 16 17 18 19
### Requirements

Install the Python requirements.

```sh
pip install -r requirements.txt
20 21
# when developing
pip install -r requirements-dev.txt
22 23 24 25 26 27 28 29 30 31 32
```

### Private token

The scripts of this repository use GitLab API in a way that requires authentication.
They read a "private token" from an environment variable which can be also configured via a `.env` file.

First, copy the private token from this private file: https://git.nomics.world/cepremap-private/servers-and-services/blob/master/dbnomics-gitlab-private-tokens.md

If you don't have access, ask DBnomics team.

33
Copy the `.env.example` file to `.env` and replace the value of `GITLAB_PRIVATE_TOKEN`.
34 35

Now you can use the scripts of this repository.
36

Christophe Benz's avatar
Christophe Benz committed
37
## Other scripts
Christophe Benz's avatar
Christophe Benz committed
38

Christophe Benz's avatar
Christophe Benz committed
39
- `open-urls-for-provider.py` opens all URLs related to GitLab-CI management for a provider. It's a quick helper meant to help debugging the CI.
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

## What to do after changing a provider code

Example: rename `bank-of-england` to `BOE`.

- Rename the fetcher, source data and JSON data repositories in advanced settings
- Rename your local directories and update Git remote URLs with:
  ```
  git remote set-url origin git@git.nomics.world:dbnomics-fetchers/boe-fetcher.git
  ```
  (this time by ssh)
- Remove all documents from Solr index with `./delete_provider.sh bank-of-england`
- In the fetcher repo:
  - change the code in `convert.py` in `PROVIDER_JSON` constant (use `BOE`)
  - change the `PROVIDER_SLUG` variable in `.gitlab-ci.yml` (use `boe`)
- On the server `dolos`, rename the JSON data directory under `/home/gitlab-runner/json-data` and update thew Git remote URL:
  ```
  git remote set-url origin https://git.nomics.world/dbnomics-json-data/boe-json-data.git
  ```
  (this time by `https`)
- Update the `PROVIDER_SLUG` job variable in the [webhook](https://git.nomics.world/dbnomics-json-data/boe-json-data/settings/integrations)
61
- Trigger the pipeline of the fetcher with `SKIP_DOWNLOAD=1`, or for legacy pipelines use `./trigger-job-for-provider-legacy.py convert boe`
Christophe Benz's avatar
Christophe Benz committed
62 63 64 65 66 67 68 69 70 71 72 73 74

## Legacy

The following sections only apply to legacy GitLab CI pipelines.
Now, a new shared pipeline exists, cf [dbnomics-fetcher-pipeline](https://git.nomics.world/dbnomics/dbnomics-fetcher-pipeline).

### Configure CI for a provider

- Use [dbnomics-fetcher-cookiecutter](https://git.nomics.world/dbnomics/dbnomics-fetcher-cookiecutter), or copy its `.gitlab-ci.yml` to the fetcher directory, and subtitute `{{ }}` placeholders by real values.
- Set the PROVIDER_SLUG variable in the `variables` section of the YAML file.
- [optional] If the fetcher scripts are designed to read/write Git objects (blobs and trees) instead of files and create the commit, for example because they implement incremental mode, adapt the CI in consequence. Look at [IMF CI config file](https://git.nomics.world/dbnomics-fetchers/imf-fetcher/blob/master/.gitlab-ci.yml) for an example.
- Check that you keep lines of `.gitlab-ci.yml` specific to this fetcher (for example `apt install wget`).
- Commit `.gitlab-ci.yml` and push.
Christophe Benz's avatar
Christophe Benz committed
75
- Run `configure-fetcher-legacy.py` and follow the instructions.
Christophe Benz's avatar
Christophe Benz committed
76
  ```sh
Christophe Benz's avatar
Christophe Benz committed
77
  ./configure-fetcher-legacy.py --purge -v <provider_slug>
Christophe Benz's avatar
Christophe Benz committed
78
  ```
79
- To test averything is okay, trigger a job using `trigger-job-for-provider-legacy.py` (see below).
Christophe Benz's avatar
Christophe Benz committed
80 81 82 83 84 85

### Trigger a job for a provider

This script runs a job in GitLab-CI using the configured webhooks. The triggered job can be followed by clicking on the link printed by the script.

```sh
86
./trigger-job-for-provider-legacy.py <download|convert|index> <provider_slug>
Christophe Benz's avatar
Christophe Benz committed
87
```