Allow stopping iteration of resources
Currently process_resources
processes all resources sequentially. If there is an exception during processing a resource, an event with status FAILURE
is created and the next resource is processed.
Fetchers would need to stop the iteration of resources given a particular response of a server. For example, if an XML file is expected, and we get an HTML snippet with "Service unavailable", on a particular server it would be problematic to continue processing all other resources, because it would mean that the server is overloaded.
So process_resource
of a fetcher would need to raise a custom exception exposed by dbnomics_fetcher_toolbox.resources.AbortIteration
, and process_resources
would catch it and break
the for
loop. The current resource would have its event with status=FAILURE
, and every following resource would have an event created with status=SKIPPED
(cf #637 (closed)) and message="Processing of resources aborted while processing resource 'FOO'"
.
Tasks
-
define a AbortIteration
exception with its docstring explaining its usage indbnomics_fetcher_toolbox.resources
-
update _process_resource_wrapper
and_process_resources_in_sequence
to react properly toAbortIteration