management issueshttps://git.nomics.world/dbnomics-fetchers/management/-/issues2021-08-11T14:36:47Zhttps://git.nomics.world/dbnomics-fetchers/management/-/issues/1015WEBAPI should return error if a dimension name or value doesn't exist2021-08-11T14:36:47ZMichel JuillardWEBAPI should return error if a dimension name or value doesn't existI would be helpful to the users, if the API returned an error message when a query uses a dimension name or value that doesn't exist for the dataset.
This improvement could be then leveraged in DBnomics clients.I would be helpful to the users, if the API returned an error message when a query uses a dimension name or value that doesn't exist for the dataset.
This improvement could be then leveraged in DBnomics clients.Christophe Benzchristophe.benz@nomics.worldChristophe Benzchristophe.benz@nomics.worldhttps://git.nomics.world/dbnomics-fetchers/management/-/issues/618BEA : not comprehensive2020-01-27T15:17:44ZThomas BrandBEA : not comprehensiveWe should add new datasets in the BEA API https://apps.bea.gov/API/signup/index.cfm.We should add new datasets in the BEA API https://apps.bea.gov/API/signup/index.cfm.https://git.nomics.world/dbnomics-fetchers/management/-/issues/617UNCTAD : not comprehensive2020-01-27T15:34:06ZThomas BrandUNCTAD : not comprehensiveThere are few missing datasets in DBnomics when we compare with the category of the provider : https://unctadstat.unctad.org/wds/ReportFolders/reportFolders.aspx . For example, on the provider's website, International trade in goods and ...There are few missing datasets in DBnomics when we compare with the category of the provider : https://unctadstat.unctad.org/wds/ReportFolders/reportFolders.aspx . For example, on the provider's website, International trade in goods and services > Trade structure by partner, product or service-category, there are 7 datasets, but only one in DBnomics UI.https://git.nomics.world/dbnomics-fetchers/management/-/issues/615INEGI : not comprehensive2020-01-27T15:34:25ZThomas BrandINEGI : not comprehensiveDespite an SDMX portal (http://en.www.inegi.org.mx/default.html), we have only 49 data series. Despite an SDMX portal (http://en.www.inegi.org.mx/default.html), we have only 49 data series. https://git.nomics.world/dbnomics-fetchers/management/-/issues/612BLS : not comprehensive2020-01-27T15:34:18ZThomas BrandBLS : not comprehensiveAmong all the datasets available in bulk download (https://download.bls.gov/pub/time.series/), we should take at least all the non-discontinued datasets. Based on this page https://www.bls.gov/bls/proghome.htm, we deduce a more comprehen...Among all the datasets available in bulk download (https://download.bls.gov/pub/time.series/), we should take at least all the non-discontinued datasets. Based on this page https://www.bls.gov/bls/proghome.htm, we deduce a more comprehensive category tree (see below).https://git.nomics.world/dbnomics-fetchers/management/-/issues/611BOJ : not comprehensive2020-01-27T15:34:41ZThomas BrandBOJ : not comprehensiveWe have here http://www.stat-search.boj.or.jp/info/dload_en.html a list of all the flat files (datasets) that should be included in DBnomics. We only have a part of it.We have here http://www.stat-search.boj.or.jp/info/dload_en.html a list of all the flat files (datasets) that should be included in DBnomics. We only have a part of it.https://git.nomics.world/dbnomics-fetchers/management/-/issues/573NBS : no series names in UI2021-01-28T10:35:14ZThomas BrandNBS : no series names in UIDepends on: #463
Only the code of the series appears, for example https://db.nomics.world/NBS/A_A020201. We don't know the corresponding name, so we can't use these series.Depends on: #463
Only the code of the series appears, for example https://db.nomics.world/NBS/A_A020201. We don't know the corresponding name, so we can't use these series.https://git.nomics.world/dbnomics-fetchers/management/-/issues/457BLS json provided by API contains too much2019-06-13T17:22:36ZMichel JuillardBLS json provided by API contains too muchThe request https://api.db.nomics.world/v22/series/BLS/cu/CUSR0000SETE?observations=1 sends back a json that contains all the values of the dimensions instead of just the ones correcponding to that particular series.The request https://api.db.nomics.world/v22/series/BLS/cu/CUSR0000SETE?observations=1 sends back a json that contains all the values of the dimensions instead of just the ones correcponding to that particular series.https://git.nomics.world/dbnomics-fetchers/management/-/issues/431Check all fetchers for datasets with names including time2021-03-30T11:22:30ZMichel JuillardCheck all fetchers for datasets with names including timeThere are a some providers where the name of a dataset changes with updates (for example IMF/WEO or ESRI)
We need to make sure that each fetcher is made robust by parsing the html page describing available datasets.
In no circumstances, ...There are a some providers where the name of a dataset changes with updates (for example IMF/WEO or ESRI)
We need to make sure that each fetcher is made robust by parsing the html page describing available datasets.
In no circumstances, a fetcher can have hard coded URL of datasets that change thru time.https://git.nomics.world/dbnomics-fetchers/management/-/issues/348Implement period intervals like "2002-2006"2019-01-21T11:15:45ZBruno DuyéImplement period intervals like "2002-2006"In Sweeden Statistics, some datasets (15% of total) uses periods like: `2002-2006`.
For now we skip those datasets. @MichelJuillard proposed to implement this later (discussion here [here](https://git.nomics.world/dbnomics-fetchers/mana...In Sweeden Statistics, some datasets (15% of total) uses periods like: `2002-2006`.
For now we skip those datasets. @MichelJuillard proposed to implement this later (discussion here [here](https://git.nomics.world/dbnomics-fetchers/management/issues/325#note_9185)).https://git.nomics.world/dbnomics-fetchers/management/-/issues/284Handle hierarchical dimensions2020-03-16T16:28:04ZChristophe Benzchristophe.benz@nomics.worldHandle hierarchical dimensionsThis is called hierarchical dimensions in SDMX.
### Source data
We can have in source data either:
- a tree with fixed depth (e.g. /continent/country)
- a tree with variable depth (e.g. /path/to/a/node)
### Data model
The idea is to...This is called hierarchical dimensions in SDMX.
### Source data
We can have in source data either:
- a tree with fixed depth (e.g. /continent/country)
- a tree with variable depth (e.g. /path/to/a/node)
### Data model
The idea is to encode the dimension tree in `dataset.json`, by adding a new key, keeping the existing data as-is.
The dimension tree can either:
- be encoded by a single dimension (e.g. 1 dimension country with continents and countries mixed)
- (not choosed) be spanned on many dimensions (e.g. 1 dimension continent and 1 dimension country)
We choose the first solution.
Example:
```json
# dataset
{
"code": "dataset1",
"dimensions_codes_order": ["SUBJECT", "GEO"],
"dimensions_labels": {
"SUBJECT": "Subject",
"GEO": "Geo"
},
"dimensions_values_labels": {
"SUBJECT": {
"S1": "Subject 1",
"S2": "Subject 2"
},
"GEO": {
"AF": "Africa",
"AL": "Algeria",
"DE": "Germany",
"EU": "Europe",
"FR": "France"
}
},
"dimension_tree": {
"GEO": {
"EU": ["DE", "FR"],
"AF": ["AL"]
}
}
}
```
```json
# series
{
"code": "S1.FR",
"dimensions": {
"SUBJECT": "S1",
"GEO": "FR"
},
"observations": []
}
```
Note: depending on the provider data, the intermediary tree nodes ("EU", "AF") can have series or just encode a classification. For example some providers may distribute time series for Europe or Africa, and some others may just distribute series for countries.
### Faceted search
Indexation script to Solr has to be updated. A series must know which intermediary nodes of the tree it is classified in.
Example:
```json
# Solr document for a series
{
[...]
"code": "S1.FR",
"type": "series",
"dimension_value_code.SUBJECT": "S1",
"dimension_value_code.GEO": ["FR", "EU"]
}
```
Question:
- [ ] is it feasible? should the schema be evolved?
### UX
The facet widget has to become a tree, visually indented, where:
- each node and leaves is selectionnable via a checkbox
- each node is collapsable/extensible
When selecting a node or a leaf:
- the normal mechanism of facets would be used
- for now we keep displaying the selected node in the select box (and not its full path)
Optimizations:
- lazy DOM nodes creation
### Inspiration
- https://www.econdb.com/dataset/NAMQ_10_GDP/gdp-and-main-components-output-expenditure-and-income/
![image](/uploads/3d386c4176bccd06033c1da7cb944974/image.png)
- an example of JSON-stat using tree-like dimensions: https://json-stat.org/samples/hierarchy.json
### Tasks
- [ ] encode dimension tree in `dataset.json` (cf above)
- [ ] propose an evolution of `dataset.json` schema in data model
- [ ] enhance the indexation script to set multiple values for hierarchical dimensions
- [ ] enhance the web API to output this tree when requesting a dataset
- [ ] release a new version of the API, deployed under `/v22.1`
- [ ] enhance the dimension widget in the UI