Allow stopping iteration of resources
Currently process_resources processes all resources sequentially. If there is an exception during processing a resource, an event with status FAILURE is created and the next resource is processed.
Fetchers would need to stop the iteration of resources given a particular response of a server. For example, if an XML file is expected, and we get an HTML snippet with "Service unavailable", on a particular server it would be problematic to continue processing all other resources, because it would mean that the server is overloaded.
So process_resource of a fetcher would need to raise a custom exception exposed by dbnomics_fetcher_toolbox.resources.AbortIteration, and process_resources would catch it and break the for loop. The current resource would have its event with status=FAILURE, and every following resource would have an event created with status=SKIPPED (cf #637 (closed)) and message="Processing of resources aborted while processing resource 'FOO'".
Tasks
-
define a AbortIterationexception with its docstring explaining its usage indbnomics_fetcher_toolbox.resources -
update _process_resource_wrapperand_process_resources_in_sequenceto react properly toAbortIteration