DARES: Download files

• As a fetcher conversion script
• I want to read the source files of the provider in a Git repository
• in order to convert them to JSON.

Acceptance criteria

• The source files of the provider MUST be committed to the source repository.
• Some HTML files MUST be committed too to memorize the names of the files and the categories.
• The download script MUST push to the remote repository only if used with the argument --push.

As system, I need to download files with data from DARES in order to convert them later

Depends from issue #36 (closed)

Fetcher description: https://git.nomics.world/dbnomics/widukind-projects/blob/master/docs/fetcher_description/dares.rst

Parse each category HTML page in order the get the following links (don't hardcode the links in case the web sites change in the future)

Normally, we take all xls files from a category, but watch for possible exceptions.

List of files by category

L’activité partielle

['IMG/xls/tdb_ap_t12017.xls']

Les demandeurs d’emploi inscrits à Pôle emploi : données nationales

[ 'IMG/xls/ser_nat_cvs-lala39bim59_.xls',

'IMG/xls/ser_nat_brut-lala39bim59_.xls', 'IMG/xls/defm_age_detaille-lala39bim59_.xls']

### Les demandeurs d’emploi inscrits à Pôle emploi : données régionales, départementales et par zone d’emploi

[ 'IMG/xlsx/ser_reg_dep-lala39bim59_.xlsx',

'IMG/xlsx/ser_reg_dep_cvs-lala39bim59_.xlsx', 'IMG/xlsx/ze_internet-lala39bim59_.xlsx']

### La durée collective hebdomadaire

[ 'IMG/xlsx/series_detailles_duree_20172_diffuse.xlsx',

'IMG/xls/Series_detaillees_duree_diffusion_081223__v2-2.xls']

### La durée individuelle du travail ['IMG/xlsx/duree_individuelle_de_travail_062017.xlsx'] ### L’emploi salarié ['IMG/xls/donnees_trimestrielles_emploi_salarie_2017t1.xls'] ### Les emplois vacants ['IMG/xls/emplois_vacants_2017t2_prov-2.xls'] ### L’emploi intérimaire

[ 'IMG/xls/donnees_mensuelles_juin_2017_sur_l_interim.xls',

'IMG/xls/donnees_trimestrielles_au_4_trimestre_2016_sur_l_interim.xls', 'IMG/xls/donnees_annuelles_sur_l_interim_2015.xls']

### Les journées individuelles non travaillées (JINT) ['IMG/xls/donnees_annuelles_jint_jusqu_en_2014.xls'] ### Les heures supplémentaires ['IMG/xls/les_heures_supplementaires_au_1er_trimestre_2017.xls'] ### Les heures supplémentaires rémunérées ['IMG/xls/les_donnees_annuelles_sur_les_heures_supplementaires_remunerees_maj2014-2.xls'] ### Les offres collectées et satisfaites par Pôle emploi ['IMG/xlsx/ser_offres-lala39bim59_.xlsx'] ### La participation, l’intéressement et l’épargne salariale ['IMG/xls/participation_interessement_et_epargne_salariale_de_2006_a_2015.xls'] ### Les dispositifs publics d’accompagnement des restructurations

[ 'IMG/xls/series_lic_eco_cvs_2017.t1.xls',

'IMG/xls/series_lic_eco_brut_2017.t1.xls']

### Les ruptures conventionnelles

[ 'IMG/xls/donnees_mensuelles_ruptures_conventionnelles_juillet2017.xls',

'IMG/xls/les_donnees_mensuelles_a_mars_2017_inscriptions_des_de_a_la_suite_de_rupture_conv.xls']

### Les indices de salaire de base

[ 'IMG/xlsm/series_salaires_2017t2_diffuse.xlsm',

'IMG/xls/Series_detaillees_salaires_diffusion_081223__v2.xls', 'IMG/xls/Series_longues_1945_2008-4.xls']

### Les sortants des listes de demandeurs d’emploi inscrits à Pôle emploi

[ 'IMG/xls/series_enquetesortants_cvscjo_afm_decembre16.xls',

'IMG/xls/series_enquete_sortants_brut_afm_decembre16.xls']

### Le temps partiel ['IMG/xlsx/le_temps_partiel_1982-2016_072017.xlsx'] ### Les tensions sur le marché du travail par métier

[ 'IMG/xls/donnees_cvs_tensions_n17t2.xls',

'IMG/xls/donnees_brutes_tensions_n17t2.xls']

Criteria of acceptance

• 32 XLS files must have been stored

added Fetcher label

Criteria of acceptance

• 32 XLS files must have been stored

I counted 31 XLS in this listing and correctly downloaded 31 files

@MichelJuillard Did I miss one that would have not be mentioned?

@c24b could you please check which one you are missing in the following list

1. 'IMG/xls/tdb_ap_t12017.xls'
2. 'IMG/xls/[ser_nat_cvs-lala39bim59_.xls'
3. 'IMG/xls/[ser_nat_brut-lala39bim59_.xls'
4. 'IMG/xls/[defm_age_detaille-lala39bim59_.xls'
5. 'IMG/xlsx/[ser_reg_dep-lala39bim59_.xlsx'
6. 'IMG/xlsx/[ser_reg_dep_cvs-lala39bim59_.xlsx',
7. 'IMG/xlsx/[ze_internet-lala39bim59_.xlsx'
8. 'IMG/xlsx/series_detailles_duree_20172_diffuse.xlsx'
9. 'IMG/xls/Series_detaillees_duree_diffusion_081223__v2-2.xls
10. 'IMG/xlsx/duree_individuelle_de_travail_062017.xlsx'
11. 'IMG/xls/donnees_trimestrielles_emploi_salarie_2017t1.xls'
12. 'IMG/xls/emplois_vacants_2017t2_prov-2.xls'
13. 'IMG/xls/donnees_mensuelles_juin_2017_sur_l_interim.xls'
14. 'IMG/xls/donnees_trimestrielles_au_4_trimestre_2016_sur_l_interim.xls',
15. 'IMG/xls/donnees_annuelles_sur_l_interim_2015.xls'
16. 'IMG/xls/donnees_annuelles_jint_jusqu_en_2014.xls'
17. 'IMG/xls/les_heures_supplementaires_au_1er_trimestre_2017.xls'
18. 'IMG/xls/les_donnees_annuelles_sur_les_heures_supplementaires_remunerees_maj2014-2.xls'
19. 'IMG/xlsx/[ser_offres-lala39bim59_.xlsx'
20. 'IMG/xls/participation_interessement_et_epargne_salariale_de_2006_a_2015.xls'
21. 'IMG/xls/series_lic_eco_cvs_2017.t1.xls'
22. 'IMG/xls/series_lic_eco_brut_2017.t1.xls'
23. 'IMG/xls/donnees_mensuelles_ruptures_conventionnelles_juillet2017.xls'
24. 'IMG/xls/les_donnees_mensuelles_a_mars_2017_inscriptions_des_de_a_la_suite_de_rupture_conv.xls'
25. 'IMG/xlsm/series_salaires_2017t2_diffuse.xlsm'
26. 'IMG/xls/Series_detaillees_salaires_diffusion_081223__v2.xls',
27. 'IMG/xls/Series_longues_1945_2008-4.xls']
28. 'IMG/xls/series_enquetesortants_cvscjo_afm_decembre16.xls'
29. 'IMG/xls/series_enquete_sortants_brut_afm_decembre16.xls'
30. 'IMG/xlsx/le_temps_partiel_1982-2016_072017.xlsx'
31. 'IMG/xls/donnees_cvs_tensions_n17t2.xls'
32. 'IMG/xls/donnees_brutes_tensions_n17t2.xls'

OK found!

My mistake: because it is the only XLSM to download ... 25. IMG/xlsm/tableaux_web_series_salaires_2017t3-.xlsm

Fixing my major omission right now...

To avoid any further mistakes: a dares_settings.py as been integrated that loads the CONSTANTS settings as dependency for the two steps of the fetcher dares_to_source_data and dares_to_dbnomics

It makes a first serie of basic Assertion tests:

• test1 top_categories_nb must be equal to 19
• test2 source_files_nb must be equal to 32 if not the dares-fetcher doesn't start downloading or the dares-fecther doesn't convert

changed title from DARES - download files to DARES: Download files

changed the description

added Priority 1 Must label

assigned to @c24b

added Doing label

Pending review @cbenz @bduye on commit dares-fetcher@8cbdd251

closed

reopened

mentioned in merge request dares-fetcher!1 (closed)

added ~121 and removed Doing labels