Fetcher METI doesn't update DBnomics

added Fetcher Priority 1 Must + 1 deleted label

changed the description

The zip filename for IIP has changed from b2010_g1e.zip to b2015_g1e.zip. The strange things is that the data for Indices of Tertiary Industry Activity by industry for which the zip filename hasn't changed aren't updated either.

Note that the filenames are hard coded in download.py and in convert.py. In convert.py the name of the dataset, hard coded, has to be changed

mentioned in merge request meti-fetcher!1 (merged)

The immediate corrections are done in https://git.nomics.world/MichelJuillard/meti-fetcher/tree/b2015
One needs to check what happens in production for Indices of Tertiary Industry Activity by industry
We need to parse the website rather than hard code URLs

@MichelJuillard I merged meti-fetcher!1 (merged)

There is still an error with a ZIP file: https://git.nomics.world/dbnomics-fetchers/meti-fetcher/-/jobs/155430 Did you already see this?

@cbenz I just tried on my machine and it works. Maybe a transient network problem.

@cbenz the log on the server indicates a problem with the zip files downloaded in directory zip_file but I can't replicate it locally. It must be something linked to the production environment.

From eros server we have the following HTML when downloading http://www.meti.go.jp/english/statistics/tyo/iip/csv/b2015_g1e.zip

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;meti&#46;go&#46;jp&#47;english&#47;statistics&#47;tyo&#47;iip&#47;csv&#47;b2015&#95;g1e&#46;zip" on this server.<P>
Reference&#32;&#35;18&#46;6f2b1102&#46;1577985886&#46;1b6a346b
</BODY>
</HTML>

From my machine, or dolos server, we have a valid ZIP file.

We seem to have been blacklisted, but why would that be?

I also tried to route the jobs for Meti to the nomos runner via meti-fetcher@879ce87f but the ZIP URL also returns the same "access denied" HTML content.

Also I made the fetcher fail on any error, but download the next ZIP file, by using the newly published fetcher toolbox (to be documented more...) so that the Dashboard shows the problem.

I think that we should get in touch with Meti: @MichelJuillard could you do that? Otherwise I can setup a small new server for Meti to take advantage of its new IP address.

I sent the following message via Contact Us form on METI web site

DBnomics is a non-profit project making available international macroeconomic data in a common format. Our web site is https://db.nomics.world and redistribute data form 32 statistical agencies world wide.

We used to redistribute Indices of Industrial Production by Industry and Indices of Tertiary Industry Activity by industry but our sever: eros.nomics.world (Indices of Tertiary Industry Activity by industry) seems to be blacklisted on your site. We try to visit your web site daily to check whether there are new data available.

Many economists are interested in the Japanese economy and we would like very much to be able to redistribute your data as well.

added 1 deleted label

assigned to @cbenz

added 45m of time spent at 2020-01-02

added 30m of time spent at 2020-01-02

@cbenz I will try to get in touch with METI

removed 1 deleted label

METI never answered my message

mentioned in issue #920 (closed)

assigned to @MichelJuillard and unassigned @cbenz

added Stalled label

Is METI pipeline deactivated for this issue ?

In that case I'll add a TODO section in issue description and add a checkbox to remember to re-activate it

mentioned in issue #985 (closed)

Fetcher METI doesn't update DBnomics

Designs

Child items ...

Activity