Eurostat: missing datasets

Some Eurostat datasets are "not found".

Under Database by themes > International trade > International trade in goods - detailed data, for example: DS-016890

A problem with "Eurostat employment data for Ireland" was also mentioned by a user on the forum.

added 1 deleted label

assigned to @cbenz

added Fetcher label

added Doing label

changed the description

removed Doing label

unassigned @cbenz

added Doing label

assigned to @cbenz

assigned to @bduye and unassigned @cbenz

mentioned in issue #476 (closed)

Problem description

After a first look, it appears that for this dataset (and very probably many others) there's no download link given by provider: in table_of_contents.xml file Eurostat's API returns for this dataset:

<nt:leaf type="dataset">
    <nt:title language="en">EU trade since 1988 by CN8</nt:title>
    <nt:title language="fr">Commerce UE depuis 1988 par NC8</nt:title>
    <nt:title language="de">EU Handel nach CN8 seit 1988</nt:title>
    <nt:code>DS-016890</nt:code>
    <nt:lastUpdate />
    <nt:lastModified />
    <nt:dataStart />
    <nt:dataEnd />
    <nt:values />
    <nt:unit language="en" />
    <nt:unit language="fr" />
    <nt:unit language="de" />
    <nt:shortDescription language="en" />
    <nt:shortDescription language="fr" />
    <nt:shortDescription language="de" />
    <nt:metadata format="html" />
    <nt:metadata format="sdmx" />
</nt:leaf>

Problems that prevents from downloading data:

• <nt:metadata format="sdmx" /> is empty
• <nt:downloadLink format="sdmx" /> is not present

Here's are "corrects" nodes from another dataset in the same file for comparison:

<nt:metadata format="html">https://ec.europa.eu/eurostat/cache/metadata/en/ext_tec_esms.htm</nt:metadata>
<nt:metadata format="sdmx">https://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?file=metadata/ext_tec_esms.sdmx.zip</nt:metadata>
<nt:downloadLink format="tsv">http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?file=data/ext_tec10.tsv.gz</nt:downloadLink>
<nt:downloadLink format="sdmx">http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?file=data/ext_tec10.sdmx.zip</nt:downloadLink>

Proposed action

@thomasbrand I've no idea if this is a "normal" behavior, and maybe you could send them an email to ask it.

I think that we could change the creation of category tree to don't include datasets that have no downloadLink provided by Eurostat. @thomasbrand are you ok with this proposal ?

added 2h of time spent at 2019-07-30

Dataset DS-016890 seems to be missing from the bulkdownload facility at https://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?dir=data&start=d&sort=1 (or stored with a different name)

It would be interesting to check in git if we were ever able to download it.

@MichelJuillard I checked and no; this dataset has never been downloaded by download script

I sent a request to technical support:

Waiting for answer... In the meantime, I'll change the behavior of downloader to remove those datasets from category tree.

added 3h of time spent at 2019-07-30

Answer from Eurostat:

Dear Bruno,

Thank you for your message and your interest in Eurostat’s data.

This dataset is an International Trade dataset coming from EasyComext.

Even if there are visible through Data Explorer, they follow another workflow and they cannot be used by the JSON Webservice.

This information about limitation is presented on the page https://ec.europa.eu/eurostat/web/json-and-unicode-web-services/data-scope-and-query-size

Please see the table of contents for a full list of available datasets and tables.

These Web Services do not support following data scope:

    The International trade detailed data.
    The Census Hub data.

We remain at your disposal should you require further guidance for this request.

Should you have any further requests, please do not hesitate to open a new ticket.

Best Regards,            

They say "Please see the table of contents for a full list of available datasets and tables." but in the linked table of contents dataset DS-016890 is actually listed

I asked them where to find data, here's the answer:

You can find all our detailed data on international trade here:

https://ec.europa.eu/eurostat/web/international-trade-in-goods/data/database

and directly in our Comext database - please see attached guide.

http://epp.eurostat.ec.europa.eu/newxtweb/

We remain at your disposal should you require further guidance for this request.

So data for this category is not available for direct download, but only via web interface. Adding those data to DBnomics would need new development.

So I'll remove those datasets from category tree as -for now- we do not know any way of getting those data.

I removed those datasets from category tree, so for me the problem is solved.

added 4h of time spent at 2019-07-31