Skip to content

GitLab

Commit d97cfcaf authored by Christophe Benz's avatar Christophe Benz
1 Browse files
Options

Add support for new pipelines

parent 8f9e74b0
Pipeline #158503 passed with stage
    in 7 minutes and 44 seconds
    Expand all Hide whitespace changes
    Inline Side-by-side
    Showing with 403 additions and 248 deletions
    .env.example
    View file @ d97cfcaf
    FETCHERS_YML=fetchers.yml
    FETCHERS_YML=https://git.nomics.world/dbnomics-fetchers/management/raw/master/fetchers.yml
    GITLAB_BASE_URL=https://git.nomics.world
    GITLAB_PRIVATE_TOKEN=
    ......@@ -6,8 +6,12 @@ GITLAB_PRIVATE_TOKEN=
    UI_BASE_URL=https://db.nomics.world
    SOLR_URL=https://solr.db.nomics.world/solr/dbnomics
    NB_DATA_MODEL_JOBS=100
    NB_INDEXER_JOBS=100
    NB_PIPELINES=5
    MAX_NB_PIPELINES_FETCHED=100
    ALL_BRANCHES=0
    NO_SOLR_INFO=0
    # Legacy variables
    NB_DATA_MODEL_JOBS=100
    NB_INDEXER_JOBS=100
    README.md
    View file @ d97cfcaf
    # DBnomics dashboard
    ## Configuration
    This repo contains a script generating the [DBnomics dashboard](https://db.nomics.world/dashboard/).
    The dashboard is a static HTML file.
    ## Development
    ### Requirements
    Install the Python requirements.
    Install the Python requirements in a Python virtual env:
    ```sh
    pip install -r requirements.txt
    pip install -r requirements-dev.txt
    ```
    When developing, install also `requirements-dev.txt`.
    ### Private token
    The `generate-dashboard.py` script uses GitLab API in a way that requires authentication.
    ......@@ -20,6 +23,7 @@ It reads a "private token" from an environment variable which can be also config
    If you are part of DBnomics team you can use a private token accessible in that [private file](https://git.nomics.world/cepremap-private/servers-and-services/blob/master/dbnomics-gitlab-private-tokens.md).
    If not, you can:
    - create an account on our GitLab platform (https://git.nomics.world)
    - go to your [personal tokens page](https://git.nomics.world/profile/personal_access_tokens)
    - create a token, filling the "name" field with for example "dbnomics dashboard", and checking the "api" checkbox, and no other ones
    ......@@ -29,8 +33,28 @@ Copy the `.env.example` file to `.env` and replace the value of `GITLAB_PRIVATE_
    Now you can use the scripts of this repository.
    ## Debug
    ### Configuration
    Environment variables can be defined in `.env`. Copy the default file:
    ```bash
    cp .env.example .env
    ```
    ### Usage
    Example:
    ```bash
    python generate-dashboard.py --providers AFDB,AMECO --debug > dashboard.html
    ```
    Example: `./generate-dashboard.py --providers ecb --disable-solr-info > index.html`
    Then open the generated `dashboard.html` file in your browser.
    Export `DEBUG_PYSOLR=1` to see URLs generated by `pysolr`.
    ## Production
    The dashboard is generated every 10 minutes by a Systemd timer (like a Cron job) that triggers the [CI pipeline job](./.gitlab-ci.yml).
    It is deployed [here](https://db.nomics.world/dashboard/) as a [GitLab page](https://git.nomics.world/dbnomics/dbnomics-dashboard/pages) that is reverse-proxied by the nginx server.
    generate-dashboard.py
    View file @ d97cfcaf
    This diff is collapsed.
    templates/dashboard.html
    View file @ d97cfcaf
    {%- macro local_time(datetime_str) %}
    {# I'm using single quotes in HTML attributes below because `local_time` macro is sometimes embedded in a `title="..."` HTML attribute, and I don't know how to escape quotes in Jinja2. #}
    <local-time datetime='{{ datetime_str }}' day='numeric' month='short' year='numeric' hour='numeric' minute='numeric'>
    {{ datetime_str }}
    </local-time>
    {% endmacro -%}
    {%- macro schedule_tooltip(schedule) %}
    status: {{ "active" if schedule.active else "inactive" }}
    <br>
    {%- if schedule.active -%}
    next run: {{ local_time(schedule.next_run_at) }}
    <br>
    {%- endif -%}
    cron expression: {{ schedule.cron }}
    {% endmacro -%}
    {%- macro job_tooltip(job) %}
    {% if job.gitlab_job.attributes["ref"] != MASTER %}
    branch: {{ job.gitlab_job.attributes["ref"] }}
    <br>
    {% endif %}
    status: {{ job.get_status() }}
    <br>
    {% if job.errors_description %}
    completion: {{ job.errors_description }}
    <br>
    {% endif %}
    duration: {{ humanfriendly.format_timespan(job.gitlab_job.duration) if job.gitlab_job.duration else "?" }}
    <br>
    created: {{ local_time(job.gitlab_job.created_at) if job.gitlab_job.created_at else "?" }}
    <br>
    started: {{ local_time(job.gitlab_job.started_at) if job.gitlab_job.started_at else "?" }}
    <br>
    finished: {{ local_time(job.gitlab_job.finished_at) if job.gitlab_job.finished_at else "?" }}
    <br>
    git push: {{ str(job.git_push).lower() }}
    {% endmacro -%}
    {%- macro job_icon(project, job) %}
    {# Add background when jobs are not triggered from master #}
    {% set style = "background-color: yellow;" if job.gitlab_job.attributes["ref"] != MASTER else "" %}
    {% if job.is_fetcher_job and job.success_rate is not none and job.success_rate < 100 %}
    {# When completion information is available and fetcher did not process all data, display a progress bar. #}
    <a href="{{ job_url(args.gitlab_base_url, project, job) }}" class="w-100 mr-2" style="{{ style }}">
    <div class="progress border border-dark" title="{{ job_tooltip(job) }}" data-toggle="tooltip" data-html="true">
    <div role="progressbar" style="width: {{ job.success_rate }}%" aria-valuenow="{{ job.success_rate }}" aria-valuemin="0" aria-valuemax="100" class="progress-bar bg-success"></div>
    </div>
    </a>
    {% else %}
    <a href="{{ job_url(args.gitlab_base_url, project, job) }}" class="mr-1" target="_blank" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ job_tooltip(job) }}" style="{{ style }}">
    <i class="fas {{ job.get_job_icon_classes() }}"></i>
    </a>
    {%- endif -%}
    {% endmacro -%}
    {% from "macros.html" import local_time, render_pipeline, schedule_tooltip %}
    {% from "legacy_pipeline.html" import render_legacy_pipeline_jobs %}
    <!doctype html>
    <html lang="en">
    ......@@ -78,13 +25,12 @@
    </head>
    <body>
    <main class="container-fluid">
    <h1 class="my-4">DBnomics dashboard</h1>
    <p>
    Dashboard was generated on {{ local_time(generation_datetime_str) }} in {{ generation_duration }} from
    {% if validators.url(args.fetchers_yml) %}
    <a href="{{ args.fetchers_yml }}" target="_blank">fetchers.yml</a>.
    <a href="{{ args.fetchers_yml }}">fetchers.yml</a>.
    {% else %}
    {{ args.fetchers_yml }}.
    {% endif %}
    ......@@ -97,10 +43,7 @@
    <th scope="col">Star</th>
    <th scope="col">Provider</th>
    <th scope="col">Schedules</th>
    <th scope="col">Download</th>
    <th scope="col">Convert</th>
    <th scope="col">Index</th>
    <th scope="col">Validate</th>
    <th scope="col">Pipelines</th>
    <th scope="col">Nb datasets</th>
    <th scope="col">Nb series</th>
    <th scope="col">Maintainer</th>
    ......@@ -115,76 +58,45 @@
    {% endif %}
    </th>
    <th scope="row">
    <a href="{{ urljoin(args.ui_base_url, fetcher.provider_code) }}" target="_blank">
    <a href="{{ urljoin(args.ui_base_url, fetcher.provider_code) }}">
    {{ fetcher.provider_code }}
    </a>
    <a href="{{ project_url(args.gitlab_base_url, fetcher.project) }}" class="ml-2 small" target="_blank">
    <a href="{{ fetcher.project.web_url }}" class="ml-2 small">
    fetcher
    </a>
    <a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" class="ml-2 small" target="_blank">
    <a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" class="ml-2 small">
    jobs
    </a>
    <a href="{{ source_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small" target="_blank">
    <a href="{{ source_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small">
    source-data
    </a>
    <a href="{{ json_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small" target="_blank">
    <a href="{{ json_data_url(args.gitlab_base_url, fetcher.provider_slug) }}" class="ml-2 small">
    json-data
    </a>
    </th>
    <td>
    {% for schedule in fetcher.schedules %}
    <a href="{{ schedule_url(args.gitlab_base_url, fetcher.project, schedule) }}" target="_blank" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ schedule_tooltip(schedule) }}">
    <a href="{{ schedule_url(args.gitlab_base_url, fetcher.project, schedule) }}" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ schedule_tooltip(schedule) }}">
    <i class="fas {{ 'fa-check text-success' if schedule.active else 'fa-times text-warning' }}"></i>
    </a>
    {% else %}
    <a href="{{ schedule_url(args.gitlab_base_url, fetcher.project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No schedule defined">
    <a href="{{ schedule_url(args.gitlab_base_url, fetcher.project) }}" data-toggle="tooltip" data-placement="auto" title="No schedule defined">
    <i class="fas fa-exclamation-triangle text-danger"></i>
    </a>
    {% endfor %}
    </td>
    <td>
    <div class="d-flex align-items-center">
    {% for job in fetcher.download_jobs %}
    {{ job_icon(fetcher.project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No download job found in the latest jobs of {{ fetcher.project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% if fetcher.pipelines %}
    {% for pipeline, jobs in fetcher.pipelines %}
    {{ render_pipeline(fetcher, pipeline, jobs) }}
    {% endfor %}
    </div>
    </td>
    <td>
    {% elif fetcher.download_jobs or fetcher.convert_jobs or fetcher.index_jobs or fetcher.validate_jobs %}
    <div class="d-flex align-items-center">
    {% for job in fetcher.convert_jobs %}
    {{ job_icon(fetcher.project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No convert job found in the latest jobs of {{ fetcher.project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    </div>
    </td>
    <td>
    <div class="d-flex align-items-center">
    {% for job in fetcher.index_jobs %}
    {{ job_icon(solr_project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, solr_project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No index job found in the latest jobs of {{ solr_project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    </div>
    </td>
    <td>
    <div class="d-flex align-items-center">
    {% for job in fetcher.validate_jobs %}
    {{ job_icon(data_model_project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, data_model_project) }}" target="_blank" data-toggle="tooltip" data-placement="auto" title="No validate job found in the latest jobs of {{ data_model_project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    {{ render_legacy_pipeline_jobs(fetcher, solr_project, data_model_project) }}
    </div>
    {% else %}
    <div class="text-muted">No data could be retrieved about this fetcher.</div>
    {% endif %}
    </td>
    <td data-order="{{fetcher.nb_datasets or 0 }}">
    {{ humanfriendly.format_number(fetcher.nb_datasets) if fetcher.nb_datasets is not none else "?" }}
    ......@@ -194,7 +106,7 @@
    </td>
    <td>
    {% if fetcher.maintainer is not none %}
    <a href="{{ urljoin(args.gitlab_base_url, fetcher.maintainer) }}" target="_blank">
    <a href="{{ urljoin(args.gitlab_base_url, fetcher.maintainer) }}">
    @{{ fetcher.maintainer }}
    </a>
    {% endif %}
    ......
    templates/legacy_pipeline.html 0 → 100644
    View file @ d97cfcaf
    {% from "macros.html" import job_icon %}
    {% macro render_legacy_pipeline_jobs(fetcher, solr_project, data_model_project) %}
    <div class="d-flex align-items-center mr-4">
    <span class="mr-2">Download</span>
    {% for job in fetcher.download_jobs %}
    {{ job_icon(fetcher.project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" data-toggle="tooltip" data-placement="auto" title="No download job found in the latest jobs of {{ fetcher.project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    </div>
    <div class="d-flex align-items-center mr-4">
    <span class="mr-2">Convert</span>
    {% for job in fetcher.convert_jobs %}
    {{ job_icon(fetcher.project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, fetcher.project) }}" data-toggle="tooltip" data-placement="auto" title="No convert job found in the latest jobs of {{ fetcher.project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    </div>
    <div class="d-flex align-items-center mr-4">
    <span class="mr-2">Index</span>
    {% for job in fetcher.index_jobs %}
    {{ job_icon(solr_project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, solr_project) }}" data-toggle="tooltip" data-placement="auto" title="No index job found in the latest jobs of {{ solr_project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    </div>
    <div class="d-flex align-items-center">
    <span class="mr-2">Validate</span>
    {% for job in fetcher.validate_jobs %}
    {{ job_icon(data_model_project, job) }}
    {% else %}
    <a href="{{ jobs_url(args.gitlab_base_url, data_model_project) }}" data-toggle="tooltip" data-placement="auto" title="No validate job found in the latest jobs of {{ data_model_project.path }}">
    <i class="fas fa-question-circle text-warning"></i>
    </a>
    {% endfor %}
    </div>
    {% endmacro %}
    templates/macros.html 0 → 100644
    View file @ d97cfcaf
    {% macro local_time(datetime_str) %}
    {# I'm using single quotes in HTML attributes below because `local_time` macro is sometimes embedded in a `title="..."` HTML attribute, and I don't know how to escape quotes in Jinja2. #}
    <local-time datetime='{{ datetime_str }}' day='numeric' month='short' year='numeric' hour='numeric' minute='numeric'>
    {{ datetime_str }}
    </local-time>
    {% endmacro %}
    {% macro schedule_tooltip(schedule) %}
    status: {{ "active" if schedule.active else "inactive" }}
    <br>
    {% if schedule.active %}
    next run: {{ local_time(schedule.next_run_at) }}
    <br>
    {% endif %}
    cron expression: {{ schedule.cron }}
    {% endmacro %}
    {% macro job_tooltip(job) %}
    {% if job.gitlab_job.ref != MASTER %}
    branch: {{ job.gitlab_job.ref }}
    <br>
    {% endif %}
    name: {{ job.gitlab_job.name }}
    <br>
    status: {{ job.get_status() }}
    <br>
    {% if job.errors_description %}
    completion: {{ job.errors_description }}
    <br>
    {% endif %}
    duration: {{ humanfriendly.format_timespan(job.gitlab_job.duration) if job.gitlab_job.duration else "?" }}
    <br>
    created: {{ local_time(job.gitlab_job.created_at) if job.gitlab_job.created_at else "?" }}
    <br>
    started: {{ local_time(job.gitlab_job.started_at) if job.gitlab_job.started_at else "?" }}
    <br>
    finished: {{ local_time(job.gitlab_job.finished_at) if job.gitlab_job.finished_at else "?" }}
    <br>
    {% if job.git_push is not none %}
    git push: {{ str(job.git_push).lower() }}
    <br>
    {% endif %}
    {% endmacro %}
    {% macro pipeline_tooltip(pipeline) %}
    {% if pipeline.ref != MASTER %}
    branch: {{ pipeline.ref }}
    <br>
    {% endif %}
    id: {{ pipeline.id }}
    <br>
    status: {{ pipeline.status }}
    <br>
    created: {{ local_time(pipeline.created_at) if pipeline.created_at else "?" }}
    <br>
    updated: {{ local_time(pipeline.updated_at) if pipeline.updated_at else "?" }}
    {% endmacro %}
    {% macro job_icon(project, job) %}
    {# Add background when jobs are not triggered from master #}
    {% set style = "background-color: yellow;" if job.gitlab_job.ref != MASTER else "" %}
    {% if job.is_fetcher_job and job.success_rate is not none and job.success_rate < 100 %}
    {# When completion information is available and fetcher did not process all data, display a progress bar. #}
    <a href="{{ job.gitlab_job.web_url }}" class="w-100 mr-2" style="{{ style }}">
    <div class="progress border border-dark" title="{{ job_tooltip(job) }}" data-toggle="tooltip" data-html="true" style="min-width: 3em;">
    <div role="progressbar" style="width: {{ job.success_rate }}%" aria-valuenow="{{ job.success_rate }}" aria-valuemin="0" aria-valuemax="100" class="progress-bar bg-success"></div>
    </div>
    </a>
    {% else %}
    <a href="{{ job.gitlab_job.web_url }}" class="mr-1" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ job_tooltip(job) }}" style="{{ style }}">
    <i class="fas {{ job.get_job_icon_classes() }}"></i>
    </a>
    {% endif %}
    {% endmacro %}
    {% macro render_pipeline(fetcher, pipeline, jobs) %}
    <div class="d-flex align-items-center mb-2">
    <a href="{{ pipeline.web_url }}" class="mr-2" data-toggle="tooltip" data-html="true" data-placement="auto" title="{{ pipeline_tooltip(pipeline) }}" >
    #{{pipeline.id}}
    </a>
    {% for job in jobs %}
    {{ job_icon(fetcher.project, job) }}
    {% endfor %}
    </div>
    {% endmacro %}
    Markdown is supported
    0% or .
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment