Toolbox: add event for skipped resources
The function process_resources
processes resources and creates events with status={SUCCESS,FAILURE}
. It skips resources by applying arguments like --only
, --limit
(maybe other...), without creating any event for those.
It would be useful to have a ResourceEvent
with status=SKIPPED
to keep explicitly what was done in status.jsonl
. Moreover, process_resources
returns resource_event_by_id
to the fetcher which can reuse this status to enrich the category tree by reporting it in the leaves (the datasets).
Tasks
-
in process_resources
afterresources_to_process = get_resources_to_process(resources, args)
, add each skipped resource toresource_event_by_id
-
write a test in test_resources.py
that callsprocess_resources
with some resources skipped, and checks thatresource_event_by_id
contains exactly the same number of items than the number of resources.
Edited by Pierre Dittgen