Commit 92dbcd5e authored by Christophe Benz's avatar Christophe Benz

Update

parent 3883db46
Pipeline #120213 passed with stage
in 54 seconds
# Architecture
## Services
![Diagram of DBnomics services](./assets/services.svg)
## GitLab as a platform
- host repositories
- continuous integration
- pages
This diff is collapsed.
This diff is collapsed.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="731px" height="161px" viewBox="-0.5 -0.5 731 161" content="&lt;mxfile host=&quot;www.draw.io&quot; modified=&quot;2020-02-07T18:25:30.823Z&quot; agent=&quot;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36&quot; etag=&quot;d9gjbn80tiwVxHSPq9Dx&quot; version=&quot;12.6.6&quot; type=&quot;google&quot;&gt;&lt;diagram id=&quot;9imAEOg_j1ddoVbDTOxJ&quot; name=&quot;Page-1&quot;&gt;3Vdtb9owEP41fBwicQvpR6CwbuokJjp1+zSZ5EjcOTnqXEror5+dOITMa9VKXWnHB+R7fOeX596cHpum5UfFN8kXjED2/EFU9th5z/dHga//DbCrgZPhqAZiJaIa8lpgKe7BggOLFiKCvKNIiJLEpguGmGUQUgfjSuG2q7ZG2d11w2NwgGXIpYtei4iSGg38UYtfgIiTZmdveFbPpLxRtjfJEx7h9gBisx6bKkSqR2k5BWm4a3ip7eYPzO4PpiCjpxjIkv28urwPdmez6OZE0tfP0/zDab3KHZeFvfDF1dVCIwpuC8gpt2enXUMIRJofK6KiBGPMuJy16ERhkUVgdh1oqdW5RNxo0NPgDRDtrLN5QaihhFJpZ+s9zUYPXtVCORYqhEfu14QMVzHQI3r+3iE6kAFTILWrWJCcxF33HNyGVLzXa1nXA0v8M5zgOU4YLz45xHdp3SaCYLnh1e23OvX+RuEdKILycRLdS1sDFtjAtZnrD628bfPAa4I7OciBRu/FefIdnpbAVZgYO01N2fOHUp9hslJ6FJvREqU6OpFD9taIPPnPs549Mesf8NvrZD1znHANq1wHoxvGDcJNgChYV+rD28K0Dk0hmV44Nqfx59Gqn2Eqwry/RaV73V6tWctVkPX0nLfbHTljvDeXMZ7nkPK+UiJ4YkqwY6aEHxyDZSgFfTfm/VMr/ehI56VduxJ2B8IClNA3B2WxI/irfogey2GBU8O+5eB2XP303ZhhkcpxSKjpmpj6IPQz+5KvQC5Q1z2BmVZZIRGmWkGaiQkPf8WVg6cojZ1eja2r38EaYyliY0v4RynCgqTIYLr/Nhi8TH3yu+WJeW55Yq/6MBo5fnhOy+B5skKuor7TG+Zu/zhvtN9F52D/sHNosf16q+YOPoHZ7Dc=&lt;/diagram&gt;&lt;/mxfile&gt;" style="background-color: rgb(255, 255, 255);"><defs/><g><path d="M 480 30 L 603.63 30" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 608.88 30 L 601.88 33.5 L 603.63 30 L 601.88 26.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 30px; margin-left: 545px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">HTTP requests</div></div></div></foreignObject><text x="545" y="33" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">HTTP requests</text></switch></g><rect x="360" y="0" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 30px; margin-left: 361px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">API</div></div></div></foreignObject><text x="420" y="34" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">API</text></switch></g><rect x="610" y="0" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 30px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Search index<br />Solr</div></div></div></foreignObject><text x="670" y="34" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Search index...</text></switch></g><path d="M 230 30 L 353.63 30" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 358.88 30 L 351.88 33.5 L 353.63 30 L 351.88 26.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 30px; margin-left: 295px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">HTTP requests</div></div></div></foreignObject><text x="295" y="33" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">HTTP requests</text></switch></g><rect x="110" y="0" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 30px; margin-left: 111px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Website<br /><a href="https://db.nomics.world">db.nomics.world</a></div></div></div></foreignObject><text x="170" y="34" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Website...</text></switch></g><path d="M 30 80 L 70 80 L 70 30 L 103.63 30" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 108.88 30 L 101.88 33.5 L 103.63 30 L 101.88 26.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 15 80 L 70 80 L 70 130 L 103.63 130" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 108.88 130 L 101.88 133.5 L 103.63 130 L 101.88 126.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><ellipse cx="15" cy="57.5" rx="7.5" ry="7.5" fill="#ffffff" stroke="#000000" pointer-events="all"/><path d="M 15 65 L 15 90 M 15 70 L 0 70 M 15 70 L 30 70 M 15 90 L 0 110 M 15 90 L 30 110" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 117px; margin-left: 15px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">User</div></div></div></foreignObject><text x="15" y="129" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">User</text></switch></g><rect x="110" y="100" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 130px; margin-left: 111px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><a href="https://dashboard.db.nomics.world/">Dashboard</a></div></div></div></foreignObject><text x="170" y="134" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dashboard</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file
# Features
# Design goals
## Redistribute data from providers as-is
We want our users to be aware that the original provider distributes the data as it is found on DBnomics.
We want our users to be aware that the data found on DBnomics is similar to the provider data.
On the other hand, we want our users to avoid dealing with data representation specificities.
As a consequence, DBnomics distinguishes data from its format, and simplifies format only.
......
......@@ -3,11 +3,17 @@
In DBnomics, data acquisition is done by *fetchers*, small programs that download data from the provider infrastructure, and convert it to a common data model and format.
Each fetcher covers a particular data provider.
## Scheduling
## Components
![Diagram of the components of a fetcher](./assets/fetcher-components.svg)
## Pipeline
Fetchers are scheduled on a regular basis in order to keep DBnomics data up to date.
Every data download or conversion exists as a job, and the [dashboard](https://db.nomics.world/dashboard) shows them.
As explained on the [architecture](./architecture) page, the DBnomics project relies heavily on [GitLab](http://gitlab.com/) and its continuous integration system as a development platform.
![Diagram of the pipeline of a fetcher](./assets/fetcher-pipeline.svg)
## Contribute
......
# Infrastructure
info technique pour bruno et pdi : j'ai ouvert en lecture seule le domaine solr.db.nomics.world pour faciliter le requêtage Solr et éviter d'avoir à faire un tunnel SSH. Par exemple pour obtenir les datasets d'Eurostat : https://solr.db.nomics.world/solr/dbnomics/select?q=provider_code:Eurostat%20AND%20type:dataset
# Quick start
The purpose of the DBnomics project is to provide a unique website accessible for all users, allowing them to free download public economic data as released by national producers (national institutes of statistics, central banks) as well as international ones (IMF, World Bank, OECD, Eurostat, ECB…).
DBnomics goals:
- aggregate data from many providers
......
......@@ -2,11 +2,11 @@ site_name: DBnomics
nav:
- Home: index.md
- Quick start: quickstart.md
- Features: features.md
- Design goals: design-goals.md
- Data model: data-model.md
- Fetchers: fetchers.md
- Web API: web-api.md
- Infrastructure: infrastructure.md
- Architecture: architecture.md
site_dir: public
repo_url: https://git.nomics.world/dbnomics/dbnomics-docs
repo_name: GitLab
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment