- Implement adding requests in the background using
asyncio.Task and wait for them in the drop() method.
- Implement
wait_for_all_requests_to_be_added parameter for:
BasicCrawler.add_requests()
RequestProvider.add_requests_batched()
BaseStorageClient.batch_add_requests()
wait_for_all_requests_to_be_added: bool = False
asyncio.Taskand wait for them in thedrop()method.wait_for_all_requests_to_be_addedparameter for:BasicCrawler.add_requests()RequestProvider.add_requests_batched()BaseStorageClient.batch_add_requests()