Skip to content

feat: batched request addition in RequestQueue#186

Merged
vdusek merged 11 commits into
masterfrom
improve-add-requests
Jun 17, 2024
Merged

feat: batched request addition in RequestQueue#186
vdusek merged 11 commits into
masterfrom
improve-add-requests

Conversation

@vdusek

@vdusek vdusek commented Jun 14, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

  • RequestQueue supports adding requests in batches.
  • The RequestQueueClient in StorageClient has the same interface as in the apify-python-client.
  • MemoryStorageClient implements this interface as well.

Related issues

Testing

  • A new unit test was implemented.

Checklist

  • Changes are described in the CHANGELOG.md
  • CI passed
@vdusek vdusek requested a review from janbuchar June 14, 2024 09:16
@vdusek vdusek self-assigned this Jun 14, 2024
@github-actions github-actions Bot added this to the 91st sprint - Tooling team milestone Jun 14, 2024
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Jun 14, 2024
@vdusek vdusek marked this pull request as ready for review June 14, 2024 09:20
janbuchar
janbuchar reviewed Jun 14, 2024
View reviewed changes
Comment thread src/crawlee/models.py
@vdusek vdusek requested a review from janbuchar June 14, 2024 09:35
janbuchar
janbuchar reviewed Jun 14, 2024
View reviewed changes
Comment thread src/crawlee/basic_crawler/basic_crawler.py
Comment thread src/crawlee/storages/request_queue.py Outdated
Comment thread src/crawlee/storages/request_queue.py Outdated
@vdusek vdusek requested a review from janbuchar June 14, 2024 12:31
janbuchar
janbuchar reviewed Jun 17, 2024
View reviewed changes
Comment thread src/crawlee/_utils/wait.py
Comment thread tests/unit/storages/test_request_queue.py
Comment thread src/crawlee/storages/request_queue.py Outdated
@vdusek vdusek mentioned this pull request Jun 17, 2024
Closed
@vdusek vdusek requested a review from janbuchar June 17, 2024 09:57
janbuchar
janbuchar reviewed Jun 17, 2024
View reviewed changes
Comment thread src/crawlee/storages/request_queue.py
@vdusek vdusek requested a review from janbuchar June 17, 2024 11:51
@vdusek vdusek merged commit f48c806 into master Jun 17, 2024
@vdusek vdusek deleted the improve-add-requests branch June 17, 2024 11:58
@vdusek vdusek mentioned this pull request Jun 17, 2024
Closed
@vdusek vdusek mentioned this pull request Jun 17, 2024
Closed
@vdusek vdusek mentioned this pull request Jun 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

2 participants

@vdusek @janbuchar