Commit d97cfcaf authored by Christophe Benz's avatar Christophe Benz

Add support for new pipelines

parent 8f9e74b0
Pipeline #158503 passed with stage
in 7 minutes and 44 seconds
FETCHERS_YML=fetchers.yml
FETCHERS_YML=https://git.nomics.world/dbnomics-fetchers/management/raw/master/fetchers.yml
GITLAB_BASE_URL=https://git.nomics.world
GITLAB_PRIVATE_TOKEN=
......@@ -6,8 +6,12 @@ GITLAB_PRIVATE_TOKEN=
UI_BASE_URL=https://db.nomics.world
SOLR_URL=https://solr.db.nomics.world/solr/dbnomics
NB_DATA_MODEL_JOBS=100
NB_INDEXER_JOBS=100
NB_PIPELINES=5
MAX_NB_PIPELINES_FETCHED=100
ALL_BRANCHES=0
NO_SOLR_INFO=0
# Legacy variables
NB_DATA_MODEL_JOBS=100
NB_INDEXER_JOBS=100
# DBnomics dashboard
## Configuration
This repo contains a script generating the [DBnomics dashboard](https://db.nomics.world/dashboard/).
The dashboard is a static HTML file.
## Development
### Requirements
Install the Python requirements.
Install the Python requirements in a Python virtual env:
```sh
pip install -r requirements.txt
pip install -r requirements-dev.txt
```
When developing, install also `requirements-dev.txt`.
### Private token
The `generate-dashboard.py` script uses GitLab API in a way that requires authentication.
......@@ -20,6 +23,7 @@ It reads a "private token" from an environment variable which can be also config
If you are part of DBnomics team you can use a private token accessible in that [private file](https://git.nomics.world/cepremap-private/servers-and-services/blob/master/dbnomics-gitlab-private-tokens.md).
If not, you can:
- create an account on our GitLab platform (https://git.nomics.world)
- go to your [personal tokens page](https://git.nomics.world/profile/personal_access_tokens)
- create a token, filling the "name" field with for example "dbnomics dashboard", and checking the "api" checkbox, and no other ones
......@@ -29,8 +33,28 @@ Copy the `.env.example` file to `.env` and replace the value of `GITLAB_PRIVATE_
Now you can use the scripts of this repository.
## Debug
### Configuration
Environment variables can be defined in `.env`. Copy the default file:
```bash
cp .env.example .env
```
### Usage
Example:
```bash
python generate-dashboard.py --providers AFDB,AMECO --debug > dashboard.html
```
Example: `./generate-dashboard.py --providers ecb --disable-solr-info > index.html`
Then open the generated `dashboard.html` file in your browser.
Export `DEBUG_PYSOLR=1` to see URLs generated by `pysolr`.
## Production
The dashboard is generated every 10 minutes by a Systemd timer (like a Cron job) that triggers the [CI pipeline job](./.gitlab-ci.yml).
It is deployed [here](https://db.nomics.world/dashboard/) as a [GitLab page](https://git.nomics.world/dbnomics/dbnomics-dashboard/pages) that is reverse-proxied by the nginx server.
This diff is collapsed.
{%- macro local_time(datetime_str) %}
{# I'm using single quotes in HTML attributes below because `local_time` macro is sometimes embedded in a `title="..."` HTML attribute, and I don't know how to escape quotes in Jinja2. #}
<local-time datetime='{{ datetime_str }}' day='numeric' month='short' year='numeric' hour='numeric' minute='numeric'>
{{ datetime_str }}
</local-time>
{% endmacro -%}
{%- macro schedule_tooltip(schedule) %}
status: {{ "active" if schedule.active else "inactive" }}
<br>
{%- if schedule.active -%}
next run: {{ local_time(schedule.next_run_at) }}
<br>
{%- endif -%}
cron expression: {{ schedule.cron }}
{% endmacro -%}
{%- macro job_tooltip(job) %}
{% if job.gitlab_job.attributes["ref"] != MASTER %}
branch: {{ job.gitlab_job.attributes["ref"] }}
<br>
{% endif %}
status: {{ job.get_status() }}
<br>
{% if job.errors_description %}
completion: {{ job.errors_description }}
<br>
{% endif %}
duration: {{ humanfriendly.format_timespan(job.gitlab_job.duration) if job.gitlab_job.duration else "?" }}
<br>
created: {{ local_time(job.gitlab_job.created_at) if job.gitlab_job.created_at else "?" }}
<br>
started: {{ local_time(job.gitlab_job.started_at) if job.gitlab_job.started_at else "?" }}
<br>
finished: {{ local_time(job.gitlab_job.finished_at) if job.gitlab_job.finished_at else "?" }}
<br>
git push: {{ str(job.git_push).lower() }}
{% endmacro -%}
{%- macro job_icon(project, job) %}
{# Add background when jobs are not triggered from master #}
{% set style = "background-color: yellow;" if job.gitlab_job.attributes["ref"] != MASTER else "" %}
{% if job.is_fetcher_job and job.success_rate is not none and job.success_rate < 100 %}
{# When completion information is available and fetcher did not process all data, display a progress bar. #}
<a href="{{ job_url(args.gitlab_base_url, project, job) }}" class="w-100 mr-2" style="{{ style }}">
<div class="progress border border-dark" title="{{ job_tooltip(job) }}" data-toggle="tooltip" data-html="true">
<div role="progressbar" style="width: {{ job.success_rate }}%" aria-valuenow="{{ job.success_rate }}" aria-valuemin="0" aria-valuemax="100" class="progress-bar bg-success"></div>
</div>
</a>
{% else %}
<a href="{{ job_url(args.gitlab_base_url, project, job) }}" class="mr-1" target="_blank" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ job_tooltip(job) }}" style="{{ style }}">
<i class="fas {{ job.get_job_icon_classes() }}"></i>
</a>
{%- endif -%}
{% endmacro -%}
{% from "macros.html" import local_time, render_pipeline, schedule_tooltip %}
{% from "legacy_pipeline.html" import render_legacy_pipeline_jobs %}
<!doctype html>
<html lang="en">
......@@ -78,13 +25,12 @@
</head>
<body>
<main class="container-fluid">
<h1 class="my-4">DBnomics dashboard</h1>
<p>
Dashboard was generated on {{ local_time(generation_datetime_str) }} in {{ generation_duration }} from
{% if validators.url(args.fetchers_yml) %}
<a href="{{ args.fetchers_yml }}" target="_blank">fetchers.yml</a>.
<a href="{{ args.fetchers_yml }}">fetchers.yml</a>.
{% else %}
{{ args.fetchers_yml }}.
{% endif %}
......@@ -97,10 +43,7 @@
<th scope="col">Star</th>
<th scope="col">Provider</th>
<th scope="col">Schedules</th>
<th scope="col">Download</th>
<th scope="col">Convert</th>
<th scope="col">Index</th>
<th scope="col">Validate</th>
<th scope="col">Pipelines</th>
<th scope="col">Nb datasets</th>
<th scope="col">Nb series</th>
<th scope="col">Maintainer</th>
......@@ -115,76 +58,45 @@
{% endif %}
</th>
<th scope="row">
<a href="{{ urljoin(args.ui_base_url, fetcher.provider_code) }}" target="_blank">
<a href="{{ urljoin(args.ui_base_url, fetcher.provider_code) }}">
{{ fetcher.provider_code }}
</a>
<a href="{{ project_url(args.gitlab_base_url, fetcher.project) }}" class="ml-2 small" target="_blank">
<a href="{{ fetcher.project.web_url }}" class="ml-2 small">
fetcher
</a>
<a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" class="ml-2 small" target="_blank">
<a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" class="ml-2 small">
jobs
</a>
<a href="{{ source_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small" target="_blank">
<a href="{{ source_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small">
source-data
</a>
<a href="{{ json_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small" target="_blank">
<a href="{{ json_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small">
json-data
</a>
</th>
<td>
{% for schedule in fetcher.schedules %}
<a href="{{ schedule_url(args.gitlab_base_url, fetcher.project, schedule) }}" target="_blank" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ schedule_tooltip(schedule) }}">
<a href="{{ schedule_url(args.gitlab_base_url, fetcher.project, schedule) }}" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ schedule_tooltip(schedule) }}">
<i class="fas {{ 'fa-check text-success' if schedule.active else 'fa-times text-warning' }}"></i>
</a>
{% else %}
<a href="{{ schedule_url(args.gitlab_base_url, fetcher.project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No schedule defined">
<a href="{{ schedule_url(args.gitlab_base_url, fetcher.project) }}" data-toggle="tooltip" data-placement="auto" title="No schedule defined">
<i class="fas fa-exclamation-triangle text-danger"></i>
</a>
{% endfor %}
</td>
<td>
<div class="d-flex align-items-center">
{% for job in fetcher.download_jobs %}
{{ job_icon(fetcher.project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No download job found in the latest jobs of {{ fetcher.project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% if fetcher.pipelines %}
{% for pipeline, jobs in fetcher.pipelines %}
{{ render_pipeline(fetcher, pipeline, jobs) }}
{% endfor %}
</div>
</td>
<td>
{% elif fetcher.download_jobs or fetcher.convert_jobs or fetcher.index_jobs or fetcher.validate_jobs %}
<div class="d-flex align-items-center">
{% for job in fetcher.convert_jobs %}
{{ job_icon(fetcher.project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No convert job found in the latest jobs of {{ fetcher.project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
</div>
</td>
<td>
<div class="d-flex align-items-center">
{% for job in fetcher.index_jobs %}
{{ job_icon(solr_project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, solr_project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No index job found in the latest jobs of {{ solr_project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
</div>
</td>
<td>
<div class="d-flex align-items-center">
{% for job in fetcher.validate_jobs %}
{{ job_icon(data_model_project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, data_model_project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No validate job found in the latest jobs of {{ data_model_project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
{{ render_legacy_pipeline_jobs(fetcher, solr_project, data_model_project) }}
</div>
{% else %}
<div class="text-muted">No data could be retrieved about this fetcher.</div>
{% endif %}
</td>
<td data-order="{{fetcher.nb_datasets or 0 }}">
{{ humanfriendly.format_number(fetcher.nb_datasets) if fetcher.nb_datasets is not none else "?" }}
......@@ -194,7 +106,7 @@
</td>
<td>
{% if fetcher.maintainer is not none %}
<a href="{{ urljoin(args.gitlab_base_url, fetcher.maintainer) }}" target="_blank">
<a href="{{ urljoin(args.gitlab_base_url, fetcher.maintainer) }}">
@{{ fetcher.maintainer }}
</a>
{% endif %}
......
{% from "macros.html" import job_icon %}
{% macro render_legacy_pipeline_jobs(fetcher, solr_project, data_model_project) %}
<div class="d-flex align-items-center mr-4">
<span class="mr-2">Download</span>
{% for job in fetcher.download_jobs %}
{{ job_icon(fetcher.project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" data-toggle="tooltip" data-placement="auto" title="No download job found in the latest jobs of {{ fetcher.project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
</div>
<div class="d-flex align-items-center mr-4">
<span class="mr-2">Convert</span>
{% for job in fetcher.convert_jobs %}
{{ job_icon(fetcher.project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" data-toggle="tooltip" data-placement="auto" title="No convert job found in the latest jobs of {{ fetcher.project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
</div>
<div class="d-flex align-items-center mr-4">
<span class="mr-2">Index</span>
{% for job in fetcher.index_jobs %}
{{ job_icon(solr_project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, solr_project) }}" data-toggle="tooltip" data-placement="auto" title="No index job found in the latest jobs of {{ solr_project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
</div>
<div class="d-flex align-items-center">
<span class="mr-2">Validate</span>
{% for job in fetcher.validate_jobs %}
{{ job_icon(data_model_project, job) }}
{% else %}
<a href="{{ jobs_url(args.gitlab_base_url, data_model_project) }}" data-toggle="tooltip" data-placement="auto" title="No validate job found in the latest jobs of {{ data_model_project.path }}">
<i class="fas fa-question-circle text-warning"></i>
</a>
{% endfor %}
</div>
{% endmacro %}
{% macro local_time(datetime_str) %}
{# I'm using single quotes in HTML attributes below because `local_time` macro is sometimes embedded in a `title="..."` HTML attribute, and I don't know how to escape quotes in Jinja2. #}
<local-time datetime='{{ datetime_str }}' day='numeric' month='short' year='numeric' hour='numeric' minute='numeric'>
{{ datetime_str }}
</local-time>
{% endmacro %}
{% macro schedule_tooltip(schedule) %}
status: {{ "active" if schedule.active else "inactive" }}
<br>
{% if schedule.active %}
next run: {{ local_time(schedule.next_run_at) }}
<br>
{% endif %}
cron expression: {{ schedule.cron }}
{% endmacro %}
{% macro job_tooltip(job) %}
{% if job.gitlab_job.ref != MASTER %}
branch: {{ job.gitlab_job.ref }}
<br>
{% endif %}
name: {{ job.gitlab_job.name }}
<br>
status: {{ job.get_status() }}
<br>
{% if job.errors_description %}
completion: {{ job.errors_description }}
<br>
{% endif %}
duration: {{ humanfriendly.format_timespan(job.gitlab_job.duration) if job.gitlab_job.duration else "?" }}
<br>
created: {{ local_time(job.gitlab_job.created_at) if job.gitlab_job.created_at else "?" }}
<br>
started: {{ local_time(job.gitlab_job.started_at) if job.gitlab_job.started_at else "?" }}
<br>
finished: {{ local_time(job.gitlab_job.finished_at) if job.gitlab_job.finished_at else "?" }}
<br>
{% if job.git_push is not none %}
git push: {{ str(job.git_push).lower() }}
<br>
{% endif %}
{% endmacro %}
{% macro pipeline_tooltip(pipeline) %}
{% if pipeline.ref != MASTER %}
branch: {{ pipeline.ref }}
<br>
{% endif %}
id: {{ pipeline.id }}
<br>
status: {{ pipeline.status }}
<br>
created: {{ local_time(pipeline.created_at) if pipeline.created_at else "?" }}
<br>
updated: {{ local_time(pipeline.updated_at) if pipeline.updated_at else "?" }}
{% endmacro %}
{% macro job_icon(project, job) %}
{# Add background when jobs are not triggered from master #}
{% set style = "background-color: yellow;" if job.gitlab_job.ref != MASTER else "" %}
{% if job.is_fetcher_job and job.success_rate is not none and job.success_rate < 100 %}
{# When completion information is available and fetcher did not process all data, display a progress bar. #}
<a href="{{ job.gitlab_job.web_url }}" class="w-100 mr-2" style="{{ style }}">
<div class="progress border border-dark" title="{{ job_tooltip(job) }}" data-toggle="tooltip" data-html="true" style="min-width: 3em;">
<div role="progressbar" style="width: {{ job.success_rate }}%" aria-valuenow="{{ job.success_rate }}" aria-valuemin="0" aria-valuemax="100" class="progress-bar bg-success"></div>
</div>
</a>
{% else %}
<a href="{{ job.gitlab_job.web_url }}" class="mr-1" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ job_tooltip(job) }}" style="{{ style }}">
<i class="fas {{ job.get_job_icon_classes() }}"></i>
</a>
{% endif %}
{% endmacro %}
{% macro render_pipeline(fetcher, pipeline, jobs) %}
<div class="d-flex align-items-center mb-2">
<a href="{{ pipeline.web_url }}" class="mr-2" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ pipeline_tooltip(pipeline) }}" >
#{{pipeline.id}}
</a>
{% for job in jobs %}
{{ job_icon(fetcher.project, job) }}
{% endfor %}
</div>
{% endmacro %}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment