Toolbox: add event for skipped resources
The function process_resources processes resources and creates events with status={SUCCESS,FAILURE}. It skips resources by applying arguments like --only, --limit (maybe other...), without creating any event for those.
It would be useful to have a ResourceEvent with status=SKIPPED to keep explicitly what was done in status.jsonl. Moreover, process_resources returns resource_event_by_id to the fetcher which can reuse this status to enrich the category tree by reporting it in the leaves (the datasets).
Tasks
-
in process_resourcesafterresources_to_process = get_resources_to_process(resources, args), add each skipped resource toresource_event_by_id -
write a test in test_resources.pythat callsprocess_resourceswith some resources skipped, and checks thatresource_event_by_idcontains exactly the same number of items than the number of resources.
Edited by Pierre Dittgen