|
|
|
# 17-09-06 DB.nomics Technical Committee
|
|
|
|
|
|
|
|
## Attendees
|
|
|
|
Michel Juillard
|
|
|
|
Julien Lasselot
|
|
|
|
Thomas Brand
|
|
|
|
Bruno Duyé
|
|
|
|
Emmanuel Raviart
|
|
|
|
Christophe Benz
|
|
|
|
Johan Richer
|
|
|
|
|
|
|
|
## Agenda
|
|
|
|
- Full text search infrastructure
|
|
|
|
- Data model for Mongo
|
|
|
|
- Scheduler
|
|
|
|
- Outstanding bugs in existing fetchers
|
|
|
|
- Outstanding other issues
|
|
|
|
- Workflow
|
|
|
|
- durée et objectifs de la première itération/Milestone
|
|
|
|
- Taiga/Jira (project management software)
|
|
|
|
- snipets réalisés durant l'analyse d'un provider
|
|
|
|
- fetcher DESTATIS
|
|
|
|
|
|
|
|
## Update on deliverables and projected timetable
|
|
|
|
- The first batch of fetchers is expected to be delivered by January 31, 2018.
|
|
|
|
- 19 weeks, 14 fetchers = average of ~7 work days by fetcher
|
|
|
|
- Estimation of development time for one fetcher: ~8 man-days.
|
|
|
|
- 4 persons assigned to work on fetchers: Emmanuel, Bruno (arrived August 28), Constance (arriving September 18) and Christophe (starting in October).
|
|
|
|
- New UI is expected to be delivered at the end of September.
|
|
|
|
|
|
|
|
## Main action points
|
|
|
|
|
|
|
|
### Organisational
|
|
|
|
|
|
|
|
- [A Kanban board](https://git.nomics.world/dbnomics-fetchers/management/boards) has been set up to manage the fetchers development process, which is straighforward:
|
|
|
|
- 1. **Backlog**: fetchers to-do list. A request should be added as an issue on Banque de France's side. The pipeline order can be prioritised as wanted.
|
|
|
|
- 2. **Analysis**: Banque de France selects a desired fetcher and assigns a person to edit the issue item with any insights, notes, links, snippets... that will facilitate the development of the fetcher. Issue item moved to the next column to signal this step is finished.
|
|
|
|
- 3. **Development**: An issue item is assigned to a developer to complete scripts (downloader & converter), storage, indexation and finally display in the UI. Issue item moved to the next column to signal this step is finished.
|
|
|
|
- 4. **Validation**: Banque de France assign a person to an issue item for validation. The proper functioning of a fetcher should then be verified from end to end. This step can be discussed and decided during a Technical Committee Meeting. Issue item moved to the next column to signal the fetcher is officially finished.
|
|
|
|
- 5. **Closed**: List of finished fetchers.
|
|
|
|
- **Iterations** will be conducted weekly to pace the development of DB.nomics (UI, fetchers and all aspects of the product). An unformal meeting will take place every Monday at 2pm to conclude an iteration and briefly demonstrate the progress. A new iteration will be defined and published on the forum shortly after. Iteration duration may increase to 2 weeks depending on the velocity of the team.
|
|
|
|
- **Technical Committee meetings**: They will take place every two weeks at Banque de France. Their purpose can be to share the progress made on fetchers during the previous iterations (validate them), as well as giving a preview of the current/next one.
|
|
|
|
- **Calendar**: Members of the Technical Committee will share their calendar with each other.
|
|
|
|
|
|
|
|
In addition, following today's discussion by email, we propose that communication between members of the Technical Committee should take place here on the DB.nomics forum, within the [Technical Committee category](https://forum.db.nomics.world/c/tech-com). Which, in the near future, should effectively function as a mailing list (working on it!).
|
|
|
|
|
|
|
|
### Technical
|
|
|
|
- **Missing values**: Fetchers never complete missing values in series. DB.nomics always store data "as is". This matter should be resolved at the API level.
|
|
|
|
- **Scheduler**: Publication dates should be parsed by the fetcher and stored in the metadata, when applicable (series, dataset, provider).
|
|
|
|
- **Storage**: Source data and JSON-transformed data are stored in Git repositories to keep history with one commit by fetcher pass.
|
|
|
|
|
|
|
|
## Next steps
|
|
|
|
- [Ongoing iteration](https://forum.db.nomics.world/t/20170911-iteration/87) to be concluded on Monday September
|
|
|
|
11.
|
|
|
|
- Next Technical Committee meetings: Sept 15 and Sept 29. |
|
|
|
\ No newline at end of file |