SCB Download - some datasets are not downloaded: TypeError: list indices must be integers or slices, not str
download.py fails with
$ time python download.py "$WORKSPACE_SOURCE_DATA_DIR"
download.py:86: DeprecationWarning: Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0. Use 'allowed_methods' instead
requests_retry = Retry(total=12, backoff_factor=2, status_forcelist=[
INFO: * category AA
INFO: * category AM
INFO: * category BE
INFO: * category BO
INFO: * category EN
INFO: * category FM
INFO: * category HA
INFO: Download of dataset 'OImpExpSITC4Ar' took: 47 minutes 14 seconds
WARNING: This dataset is ignored to avoid too long download times
INFO: Download of dataset 'ImpExpSPIN2007M' took: 30 minutes 49 seconds
WARNING: This dataset is ignored to avoid too long download times
WARNING: This dataset is ignored to avoid too long download times
WARNING: This dataset is ignored to avoid too long download times
WARNING: This dataset is ignored to avoid too long download times
INFO: * category HE
ERROR: Unexpected exception occurred while downloading "HushallT31" dataset
Traceback (most recent call last):
File "download.py", line 193, in download_dataset
response = download_dataset_data(url, dataset_code, metadata)
File "download.py", line 168, in download_dataset_data
for dimension in metadata["variables"]
TypeError: list indices must be integers or slices, not str
ERROR: => Dataset HushallT31 aborted
INFO: Download of dataset 'SamForvInk1c' took: 46 minutes 59 seconds
INFO: Download of dataset 'SamForvInk2' took: 25 minutes 54 seconds
INFO: Download of dataset 'InkAvTjanst' took: 2 hours 30 minutes 7 seconds
INFO: Download of dataset 'BeskForvInk' took: 22 minutes 48 seconds
INFO: Download of dataset 'Skatter' took: 6 hours 29 minutes 8 seconds
INFO: Download of dataset 'Skattereduktioner' took: 43 minutes 55 seconds
INFO: * category JO
INFO: * category LE
INFO: * category ME
INFO: * category MI
ERROR: Unexpected exception occurred while downloading "MI0305T003" dataset
Traceback (most recent call last):
File "download.py", line 193, in download_dataset
response = download_dataset_data(url, dataset_code, metadata)
File "download.py", line 168, in download_dataset_data
for dimension in metadata["variables"]
TypeError: list indices must be integers or slices, not str
ERROR: => Dataset MI0305T003 aborted
INFO: * category NR
INFO: * category NV
INFO: * category OE
ERROR: Unexpected exception occurred while downloading "OffTillgSektor" dataset
Traceback (most recent call last):
File "download.py", line 193, in download_dataset
response = download_dataset_data(url, dataset_code, metadata)
File "download.py", line 168, in download_dataset_data
for dimension in metadata["variables"]
TypeError: list indices must be integers or slices, not str
ERROR: => Dataset OffTillgSektor aborted
INFO: * category PR
INFO: * category TK
INFO: * category UF
INFO: Download of dataset 'YHStudT1dN' took: 55 minutes
INFO: Download of dataset 'UF0542T2B' took: 20 minutes 15 seconds
INFO: Download of dataset 'UF05031cHgsk' took: 2 hours 51 minutes 7 seconds
INFO: Download of dataset 'UF05032c' took: 1 hour 56 minutes 43 seconds
INFO: Download of dataset 'UF05035a' took: 21 minutes 6 seconds
INFO: Download of dataset 'UF05035c' took: 15 hours 25 minutes 33 seconds
ERROR: Job failed: execution took longer than 48h0m0s seconds