BUBA - duplicates in category tree

changed the description

And there are only codes for these datasets (instead of names).

added Fetcher label

added 1 deleted label

assigned to @bduye

added 1 deleted label and removed 1 deleted label

changed the description

@thomasbrand in ticket description you wrote "BBMMB also" but I don't see only one instance of BBMMB dataset in BUBA's datasets list

(but BBMMB haven't got any dataset name neither)

changed the description

Datasets names

In convert script there's a special case for BBK01 dataset; provider doesn't provide any SDMX "struct" file for this dataset. Indeed, in source data we can see the error returned by API for this dataset

This is the same for BBMMB: https://git.nomics.world/dbnomics-source-data/buba-source-data/blob/ff478c12236515dc6905216d2d1c3e34aac2efde/structures/BBMMB.xml

Maybe we could imagine to add to datamodel some kind of "remarks", or "extra information" that could be added at dataset/series level and displayed in UI to user. In that case, the remark should be "BUBA doesn't provide any dataset names for those series".

Of course @thomasbrand if you know [the/a] name for those datasets we could hard-code them

Duplicates

It appears that this is also "normal" for this provider to have BBK01 dataset in more than one topics:

cepremap@eros:~/fetchers-envs/buba/buba-source-data/topics$ find -iname BBK01
./BANKEN/time series/BBK01
./IWF/time series/BBK01
./EURORAUM/time series/BBK01
./GELD/time series/BBK01
./GESAMT/time series/BBK01
./AUSSENWIRTSCHAFT/time series/BBK01
./FINANZEN/time series/BBK01

This directory on eros server reflects exactly what API returns

To resume: nothing different from provider's side here.

@thomasbrand we can:

let it like it is and close this issue
add "remarks"; this implies:
- brainstorm with the team to find a proper implementation
- add it to datamodel
- add it to website
hard-code BBK01 and BBMMB datasets names, but where to find this information ?

I'm waiting for decision

added 1 deleted label

Maybe we can find an answer through this bulk-download process : https://www.bundesbank.de/en/statistics/time-series-databases/-/help-on-the-time-series-databases-750894#tar-10 . Can you check @bduye please ?

I think the problem is larger than "dataset name" because the series names in those datasets are just series codes apparently.

I downloaded some data from the link you provided:

https://www.bundesbank.de/statistic-rmi/DownloadServlet?its_fileFormat=Archive&mode=its

This bulk only contains series information, no dataset information
There's apparently an API entry point for downloading metadata, but this one seems broken. Example: https://www.bundesbank.de/statistic-rmi/StatisticDownload?metaDSI=GESAMT returns an error

I think the problem is larger than "dataset name" because the series names in those datasets are just series codes apparently.

I'm looking to that. This is maybe linked to #501 (closed)

In fact those problems are directly linked to the fact that there's no "structure" SDMX returned by API for those two datasets, so there's no datasets dimensions nor dimensions labels, so we can't generate a name from dimensions

I found that we can get series names using BBK_TITLE attribute for BBMMB dataset; without duplicates. The purpose of #501 (closed) was to stop using this information because it creates many duplicates; but for this special case (where we do not have dimensions labels) we can use it:

cepremap@eros:~/fetchers-envs/buba/buba-source-data$ grep BBK_TITLE   ./topics/GELD/time\ series/BBMMB/*.xml
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.cm.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.CM" DECIMALS="0" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Calculation method">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.nb.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.NB" DECIMALS="0" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Number of banks">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.nt.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.NT" DECIMALS="0" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Number of transactions">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.r25.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.R25" DECIMALS="2" UNIT="Prozent" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Rate at 25th percentile of volume">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.r75.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.R75" DECIMALS="2" UNIT="Prozent" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Rate at 75th percentile of volume">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.rp.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.RP" DECIMALS="0" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Publication type">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.tt.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.TT" DECIMALS="0" UNIT_MULT="6" BBK_TITLE="Euro Short-Term Rate - Total volume">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.vl.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.VL" DECIMALS="0" UNIT="Prozent" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Share 5 largest banks">
./topics/GELD/time series/BBMMB/bbmmb.d.eu000a2x2a25.wt.xml:        <bbk:Series FREQ="D" BBK_ID="BBMMB.D.EU000A2X2A25.WT" DECIMALS="3" UNIT="Prozent" UNIT_MULT="0" BBK_TITLE="Euro Short-Term Rate - Volume-weighted trimmed mean rate">

I didn't found a way to get uniq series names for BBK01 dataset. In all cases using BBK_TITLE is excluded as it would create many duplicates. Example:

./topics/GELD/time series/BBK01/bbk01.wt0913.xml:        <bbk:Series FREQ="D" BBK_ID="BBK01.WT0913" BBK_TITLE_ENG="Yields on debt securities outstanding issued by residents /Public debt securities / Mean residual maturity of morethan 2 and up to 3 years / Daily data"
./topics/GELD/time series/BBK01/bbk01.wt0914.xml:        <bbk:Series FREQ="D" BBK_ID="BBK01.WT0914" BBK_TITLE_ENG="Yields on debt securities outstanding issued by residents /Public debt securities / Mean residual maturity of morethan 3 and up to 4 years / Daily data"

If the problem is a lack of information from the API (no structure SDMX), maybe we can write an email to BUBA ?

Yes we can write to them. Do you want me to do that ?

@bduye can you prepare a message explaining technically what does not work and post it as a comment on this issue? Then @thomasbrand will validate

Here is the mail I suggest to send:

Hi

[introduction: I'm ... from DBnmics team, a platform that ...]

We're using BUBA API to get data but some topics endpoints are apparently missing: we are not able to download SDMX metadata for BBMMB and BBK01 datasets:

(all other datasets works)

Is is a bad usage of the API from our side ?

Thanks

Thanks @bduye , it sounds good. One detail : should we precise the type of metadata we need (dataset name, ...) ?

I don't think so: metadata design a precise SDMX XML file for each provider. API documentation you pointed out also use this term for this: the title of the section about this is "Downloading the SDMX metadata", so I think they'll understand what's about.

Sent email to BUBA using contact form:

Hello, 

We are using BUBA API to feed our macroeconomic database https://db.nomics.word.

We noticed that some topics endpoints are apparently missing; we are not able to download SDMX metadata for BBMMB and BBK01 datasets:

https://www.bundesbank.de/statistic-rmi/StatisticDownload?metaDSI=BBMMB
https://www.bundesbank.de/statistic-rmi/StatisticDownload?metaDSI=BBK01

(it seems that this call works with all other datasets)
Is is a bad usage of the API from our side ?

Thanks

They answered today:

Dear Mr Buye,

Thank you for your question.

It doesnt't exist a description for the dataset BBK01, because BBK01 is no real SDMX-Format.
For the dataset BBMMB doesn't exist a description at the moment.
We are trying to offer this format in the nearer future.

You can find a list with all offered datasets in the drop-down-menu on this site:
https://www.bundesbank.de/dynamic/action/en/homepage/search/statistics/749202/macro-economic-time-series


Yours sincerely,

Gerhard Juracka
DEUTSCHE BUNDESBANK

It doesn't exists because ... it doesn't exists.

@thomasbrand I suggest to close this issue

We can't close this issue. These datasets can't be used. We need to find another way. We can put this issue in the backlog.

OK @thomasbrand I'll do that

removed 1 deleted label

added 7h of time spent at 2019-10-17

changed the description

added 1 deleted label

added Doing label and removed 1 deleted label

added 35m of time spent at 2019-10-31

added 1 deleted label

removed 1 deleted label

removed Doing label

BUBA - duplicates in category tree

Designs

Child items 0

Activity

Datasets names

Duplicates