ISTAT - Italian National Institute of Statistics

• As a system
• I want to acquire data from ISTAT
• in order to diffuse them on DBnomics

Acceptance criteria

• MUST obtain data from ISTAT by SDMX (https://www.istat.it/it/metodi-e-strumenti/web-service)
• MUST convert them to DBnomics

Resources

Fetcher description: https://git.nomics.world/dbnomics-fetchers/management/wikis/Fetcher-Description/ISTAT

Technical steps

Look at http://www.istat.it/it/files/2013/07/Queries.zip

added Fetcher label

changed the description

changed milestone to %Fetchers Batch #2

changed the description

changed the description

added Doing label

assigned to @cbenz

Looking at amattioc/SDMX implementation of istat I see a REST API. That's way easier than doing SOAP.

@cbenz Is there a method to obtain the dataset list? That would be the first step.

To: GitLab noreply@git.nomics.world

Yes, I used these URLs in download.py:

INFO:__main__:2018-03-09 17:18:48,844:Fetching dataflow http://sdmx.istat.it/SDMXWS/rest/dataflow/
INFO:__main__:2018-03-09 17:18:49,770:Fetching datastructure http://sdmx.istat.it/SDMXWS/rest/datastructure/
INFO:__main__:2018-03-09 17:18:57,500:Fetching conceptscheme http://sdmx.istat.it/SDMXWS/rest/conceptscheme/
INFO:__main__:2018-03-09 17:19:04,759:Fetching categoryscheme http://sdmx.istat.it/SDMXWS/rest/categoryscheme/
INFO:__main__:2018-03-09 17:19:12,145:Fetching categorisation http://sdmx.istat.it/SDMXWS/rest/categorisation/
INFO:__main__:2018-03-09 17:19:14,727:Fetching codelist CL_ABITAZ_PROBL http://sdmx.istat.it/SDMXWS/rest/codelist/all/CL_ABITAZ_PROBL/
INFO:__main__:2018-03-09 17:19:15,283:Fetching codelist CL_ABITAZ_TITGOD http://sdmx.istat.it/SDMXWS/rest/codelist/all/CL_ABITAZ_TITGOD/
INFO:__main__:2018-03-09 17:19:20,857:Fetching codelist CL_ACCESSOMUSEO http://sdmx.istat.it/SDMXWS/rest/codelist/all/CL_ACCESSOMUSEO/

etc.

added ~141 and removed Doing labels

changed title from ISTAT (Italy) fetcher to ISTAT - Italian National Institute of Statistics

added Doing and removed ~141 labels

changed milestone to %Fetchers Batch #3

added 1 deleted label and removed Doing label

added Doing label and removed 1 deleted label

changed the description

For the record I have achieved to download data by skipping the datasets that are too long to respond. I am testing the conversion.

Deployed in pre-production: http://pre.db.nomics.world/ISTAT/174_321

It's good to see ISTAT included as a provider, but there's something odd about their time series: they're not in time-order. For example, take the retail sales series ISTAT/120_337/M.RTD_TURN3.1.TOTAL.N.1.IT. The monthly "period" array in the JSON starts like this:

"period" : [
  "2005-01",
  "2005-10",
  "2005-11",
  "2005-12",
  "2005-02",
  "2005-03",
  "2005-04",
  "2005-05",
  "2005-06",
  "2005-07",
  "2005-08",
  "2005-09",

And other years are the same: January, October, ...! This places a rather awkward burden on clients (to get the data into the right order), which we haven't seen for any other providers.

https://forum.db.nomics.world/t/istat-data-problem/228

Source data contains time series not sorted by period (cf this example)

This has been fixed by istat-fetcher@fed001d8 and a new conversion is ongoing.