Skip to content

Implement wait_for_all_requests_to_be_added for adding baches of requests #187

Description

@vdusek
  • Implement adding requests in the background using asyncio.Task and wait for them in the drop() method.
  • Implement wait_for_all_requests_to_be_added parameter for:
    • BasicCrawler.add_requests()
    • RequestProvider.add_requests_batched()
    • BaseStorageClient.batch_add_requests()
wait_for_all_requests_to_be_added: bool = False

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions