DeStatis fetcher
Website
To get a preview of a datacube (ex: 51000BJ001), follow:
- click on "Themes" link (in "Available data" menu)
- Select given theme (here 51 "Foreign Trade")
- Click on "Time series icon"
- Click on chosen datacube (here 51000BJ001)
- Click on "Value Retrieval" button
Warning : After a few minutes, the user is automatically redirected to the home page :-(
Glossary
- Wert: value
- Fall: default value
- Ganz: whole
- Jahr: Year
Categories
Chosen by Thomas
- 51 Foreign Trades
- 61 Prices
- 62 Earnings, Labour costs
- 63 Income and expediture of households
- 71 Public finances
- 81 National accounts of the federation
Download
Download operation needs an account (username, password).
For each category, a datacubes file is downloaded from URL
curl "https://www-genesis.destatis.de/genesisWS/web/RechercheService_2010?method=DatenKatalog&kennung={USER_NAME}&passwort={PASSWORD}&filter={CAT}*&bereich=Alle&listenLaenge=&sprache=en" -o {CAT}/{CAT}.datacubes.xml
Parsing the datacubes file, we extract datacube ids for this category and retrieve all datacube structure file:
curl "https://www-genesis.destatis.de/genesisWS/web/ExportService_2010?method=DatenAufbau&kennung={USER_NAME}&passwort={PASSWORD}&namen={DATACUBE_CODE}&bereich=Alle&sprache=en" -o {CAT}/{DATACUBE_CODE}.structure.xml
datacube value file:
curl "https://www-genesis.destatis.de/genesisWS/web/ExportService_2010?method=DatenExport&kennung={USER_NAME}&passwort={PASSWORD}&namen={DATACUBE_CODE}&bereich=Alle&format=csv&werte=true&metadaten=false&zusatz=false&startjahr=&endjahr=&zeitscheiben=&inhalte=®ionalmerkmal=®ionalschluessel=&sachmerkmal=&sachschluessel=&sachmerkmal2=&sachschluessel2=&sachmerkmal3=&sachschluessel3=&stand=&sprache=en" -o {CAT}/{DATACUBE_CODE}.xml
Some datacube value file contain no value. Checks if contains data.
attribute info file (attribute code are extracted from datacube value file)
curl https://www-genesis.destatis.de/genesisWS/web/ExportService_2010?method=AuspraegungInformation&kennung={USER_NAME}&passwort={PASSWORD}&name={ATT_CODE}&bereich=Alle&sprache=en" -o att.{ATT_CODE}.xml
Convert
Gets all unique unit values from structure xml files:
xmlstarlet sel -t -m "//merkmale[art='I']" -v "concat('=',masseinheit,'=')" */*.structure.xml | sed -e $'s/==*/\\\n/g' | sort -u
Gets all unique types of period values:
xmlstarlet sel -t -m "//zeitraum" -v "concat('=',.,'=')" */*.structure.xml | sed -e $'s/==*/\\\n/g' | sort -u
Gets all unique frequency values:
xmlstarlet sel -t -m "//merkmale[typ='ZI']/code" -v "concat('=',.,'=')" */*.structure.xml | sed -e $'s/==*/\\\n/g' | sort -u