Remove deleted series from Solr index
- As a user of the website
- I want to receive search results with only series that exist in the JSON repository
- in order to avoid manipulating obsolete data.
Acceptance criteria
-
After the execution of the conversion job, the index job MUST delete from the index the series which were deleted in the JSON repository.
Technical steps
This would be trivial to delete all the series and before importing them, but big repositories like Eurostat take a few hours to index.
Detection of deleted series can be done either:
- by inspecting the latest Git commit
- by retrieving all Solr ids and comparing with the IDs of the latest commit
- by setting a token or a datetime for all the newly imported series, and delete those who don't have this datetime (idea of @pdi)