Eurostat: category tree is different than provider's

As reported by user

On DBnomics

https://db.nomics.world/Eurostat

On Eurostat

https://ec.europa.eu/eurostat/data/database

added 1 deleted label

added Fetcher label

changed the description

added Doing label

removed Doing label

added Doing label

assigned to @bduye

mentioned in merge request eurostat-fetcher!1 (merged)

This is exact: some categories and datasets are currently missing in Eurostat.

Technical explanation:

For now downloader and convert scripts only takes in account XML leafs of type dataset in table_of_contents.xml file. But there are some datasets that have type table (while they do not contain only one table, as name would suggest).

Proposed solution

I adapted downloder and converter code to take those datasets onto account too. The merge request is here: eurostat-fetcher!1 (merged)

added 6h of time spent at 2019-08-02

added 1h of time spent at 2019-08-26

For people who follow this issue, until eurostat-fetcher!1 (merged) is merged, all updates are in this merge request

This is fixed now, preprod is accessible

added 1 deleted label and removed Doing label

added Doing label and removed 1 deleted label

removed Doing label

added 3h of time spent at 2019-09-17

added 1 deleted label

added Doing label and removed 1 deleted label

In preprod we can check that the missing parts of category tree are now available:

Also, 1064 new datasets have been added. Only those new datasets are available in preprod (for disk space and computing time issues).

Some random "new" datasets:

• http://pre.db.nomics.world/Eurostat/tqoe1c2
• http://pre.db.nomics.world/Eurostat/tps00172
• http://pre.db.nomics.world/Eurostat/teicp290

added 1 deleted label and removed Doing label

changed the description

@MichelJuillard here's the flat list of all new datasets that have been added: new_datasets_in_Eurostat.txt

added Doing label and removed 1 deleted label

I merged eurostat-fetcher!1 (merged)

Before closing @bduye can you verify tomorrow that datasets appear in production?

Hopefully you didn't closed this issue by yourself because there's a "Things todo before closing" section in the issue description.

added 1 deleted label and removed Doing label

The last download job after merging didn't downloaded all "new" datasets, so not all "new" datasets are available in production.

After investigation, it appears that this is because the download script download in incremental mode by default, and to decide the datasets to download it compares the lastUpdate attributes in table_of_contents.xml with previously downloaded table_of_contents.xml file.

In our case, only a part (162) of "new" datasets have changed, so they have been downloaded.

What I have done:

• generated list of "new" datasets to download:

  source_data = Path("/home/bruno/dev/jailbreak/dbnomics/fetchers/eurostat/eurostat-source-data-local/")
  all = (source_data / "list_all_downloaded").read_text().split('\n')
  new = (source_data / "new_datasets").read_text().split('\n')
  to_download_datasets = set(new) - set(all)
  (source_data / 'new_datasets_to_download').write_text(' '.join(to_download_datasets))

• copied the generated file to eros
• manually downloaded those datasets: (eurostat-venv) cepremap@eros:~/fetchers-envs/eurostat/eurostat-fetcher$ python download.py ../eurostat-source-data/ --datasets $( cat ~/new_datasets_to_download )
• added this downloaded data to /home/gitlab-runner/fetchers-envs/eurostat/eurostat-source-data
• commited it and pushed it:
  • git add . --ignore-removal
  • git commit -m '#448 - manual add of missing "new" datasets'
• it triggered a convert and index
• now all "new" datasets are in production

added 1 deleted label and removed 1 deleted label

@MichelJuillard I was on the point of answering to the user that reported the issue, but I don't find the thread in forum. Maybe it have been reported by mail ? Do you still have the contact of the reporter ? Thanks

I can't find the email. Maybe it was me who reported this.

@bduye I didn't report the issue

Indeed @MichelJuillard, sorry ! So this question is to @cbenz

I don't find the info. I'm surprised I did not put it in the issue...

Maybe it's @thomasbrand who was working with Eurostat who reported this to me.

Let's close the issue anyway, thanks @bduye !

Fixed: I can see this dataset in production: https://db.nomics.world/Eurostat/tqoe1c2

closed

changed the description

added 7h of time spent at 2019-10-02