refactor!: сhange default `incognito context` to `persistent context` for `Playwright` by Mantisus · Pull Request #985 · apify/crawlee-python

Mantisus · 2025-02-15T03:07:39Z

Description

Changes the PlaywrightCrawler from using the standard browser context to using a persistent browser context.
Allows passing a user_data_dir with the path to the directory for the context. If user_data_dir is not provided, a temporary directory will be created.

Issues

vdusek

Since it contains a breaking change, could you please describe the breaking changes in the PR's description and also summarize it in the Upgrading guide?

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

)

### Description - fix public imports in `__init__` files - Add `rich` to direct dependencies. It is one of `cookiecutter`'s dependencies, but we use it directly in `statistics._models.py` --------- Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

### Description Add adaptive context helpers and documentation for AdaptivePlaywrightCrawler. ### Issues - Closes: apify#249 --------- Co-authored-by: Jan Buchar <Teyras@gmail.com> Co-authored-by: Jan Buchar <jan.buchar@apify.com>

…#988) ### Description - change custom `LRUCache` to `cachetools.LRUCache`. In my opinion, `functools.lru_cache's` logic isn't well-suited for this use case. Therefore, if we want to modify our caching approach, using `cachetools` appears to be a better option. ### Issues - Closes: apify#86

### Description - update curl-cffi version requirement to >=0.9.0. - update default `impersonate` from `chrome124` to `chrome131`

- Migrate from `poetry` to `uv`. - Relates: apify#628 - The update of templates to use `uv` will be implemented separately.

- `project.urls` - python 3.13 in ci - unify name "Set up uv package manager" - fix contributing guide - add all extra, remove dev extra (move to dev deps) - relates: apify#628

- Relates: apify#628

…pify#959) Add `additional_http_error_status_codes` and `ignore_http_error_status_codes` to PlaywrightCrawler. Since they exist now on all crawlers, move them to `BasicCrawler` level. Do not use `_http_client` attributes for getting additional status codes related variables. **Breaking:** Remove `HttpCrawlerOptions` -> No unique options compared to `BasicCrawlerOptions` anymore. - Closes: apify#953

vdusek

There are failing tests for Windows Python 3.12.

Mantisus · 2025-02-21T01:30:11Z

There are failing tests for Windows Python 3.12.

The problem is not related to this PR.

It's solved in the PR #1007

Pijukatel

Thanks for the tests, just two minor comments.

vdusek

Looks good, just a few comments.

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

vdusek

LGTM

add support persistent context

ff43115

Mantisus requested review from Pijukatel and vdusek and removed request for vdusek February 15, 2025 03:07

Mantisus self-assigned this Feb 15, 2025

Mantisus added 3 commits February 15, 2025 03:21

update docstring

92d2741

update pre_navigation_hook test

ffecf3f

rollback to AsyncNewContext for browserforge

02bfae8

Mantisus changed the title ~~refactor!: сhange default 'incognito context' to 'persistent context' for Playwright~~ Feb 15, 2025

vdusek added this to the 108th sprint - Tooling team milestone Feb 19, 2025

vdusek requested changes Feb 19, 2025

View reviewed changes

Comment thread src/crawlee/crawlers/_playwright/_playwright_crawler.py Outdated

Comment thread src/crawlee/browsers/_browser_pool.py Outdated

Comment thread src/crawlee/browsers/_playwright_browser.py Outdated

Pijukatel requested changes Feb 19, 2025

View reviewed changes

Comment thread src/crawlee/browsers/_playwright_browser.py

Pijukatel reviewed Feb 19, 2025

View reviewed changes

Comment thread src/crawlee/browsers/_browser_pool.py

Mantisus and others added 16 commits February 19, 2025 13:26

Update src/crawlee/browsers/_playwright_browser.py

3441fb1

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

Update src/crawlee/browsers/_browser_pool.py

013166e

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

Update src/crawlee/crawlers/_playwright/_playwright_crawler.py

2fbd9f9

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

chore(deps): update react monorepo (apify#879)

6a55da8

chore(deps): update typescript-eslint monorepo to v8.24.0 (apify#902)

8ea4a5e

chore(deps): update dependency eslint-plugin-react to v7.37.4 (apify#898

04aa9e4

)

chore: fix public imports (apify#987)

27fdcc0

### Description - fix public imports in `__init__` files - Add `rich` to direct dependencies. It is one of `cookiecutter`'s dependencies, but we use it directly in `statistics._models.py` --------- Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

chore(deps): update typescript-eslint monorepo to v8.24.1 (apify#992)

32f222c

feat: Add adaptive context helpers (apify#964)

6394389

### Description Add adaptive context helpers and documentation for AdaptivePlaywrightCrawler. ### Issues - Closes: apify#249 --------- Co-authored-by: Jan Buchar <Teyras@gmail.com> Co-authored-by: Jan Buchar <jan.buchar@apify.com>

chore(release): Update changelog and package version [skip ci]

b133d4f

chore(deps): update curl-impersonate to 0.9 (apify#986)

fb5e8ee

### Description - update curl-cffi version requirement to >=0.9.0. - update default `impersonate` from `chrome124` to `chrome131`

chore: migrate to uv package manager (apify#994)

4c436ef

- Migrate from `poetry` to `uv`. - Relates: apify#628 - The update of templates to use `uv` will be implemented separately.

chore(deps): lock file maintenance (apify#990)

1a7972b

chore: fix uv setup (apify#996)

db17359

- `project.urls` - python 3.13 in ci - unify name "Set up uv package manager" - fix contributing guide - add all extra, remove dev extra (move to dev deps) - relates: apify#628

ci: switch apify workflow branch back to main (apify#997)

90848cc

- Relates: apify#628

Mantisus and others added 6 commits February 19, 2025 17:14

add test for persist context

296ec6c

chore(release): Update changelog and package version [skip ci]

1ffd98d

add test persist context with fingerprint

957060a

update upgrading guide

339cd1c

resolve

bf63e32

Mantisus requested review from Pijukatel and vdusek February 19, 2025 17:22

vdusek reviewed Feb 20, 2025

View reviewed changes

Merge branch 'master' into pw-persist-browser-context

5e97e31

Mantisus force-pushed the pw-persist-browser-context branch from 6b6236d to 5e97e31 Compare February 20, 2025 23:25

rollback unrelated test

52d8d4b

Pijukatel reviewed Feb 21, 2025

View reviewed changes

Comment thread tests/unit/crawlers/_playwright/test_playwright_crawler.py Outdated

Comment thread tests/unit/crawlers/_playwright/test_playwright_crawler.py

Mantisus added 2 commits February 21, 2025 12:17

user_folder to pytest.tmp_path

a4d1a79

resolve

4bb848b

Mantisus requested review from Pijukatel and vdusek February 21, 2025 13:41

Pijukatel approved these changes Feb 24, 2025

View reviewed changes

vdusek requested changes Feb 24, 2025

View reviewed changes

Mantisus and others added 3 commits February 24, 2025 19:10

Update docs/upgrading/upgrading_to_v0x.md

c79029f

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

async for clean temp dir

7a49385

drop cast

6ad7725

Mantisus requested a review from vdusek February 25, 2025 02:45

Update _playwright_browser.py

4e79253

vdusek approved these changes Feb 25, 2025

View reviewed changes

vdusek merged commit f01520d into apify:master Feb 25, 2025

vdusek modified the milestones: 108th sprint - Tooling team, 109th sprint - Tooling team Feb 25, 2025

vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Feb 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor!: сhange default `incognito context` to `persistent context` for `Playwright`#985

refactor!: сhange default `incognito context` to `persistent context` for `Playwright`#985
vdusek merged 34 commits into
apify:masterfrom
Mantisus:pw-persist-browser-context

Mantisus commented Feb 15, 2025 •

edited

Loading

vdusek left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek left a comment

Mantisus commented Feb 21, 2025

Pijukatel left a comment •

edited

Loading

Uh oh!

Uh oh!

vdusek left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek left a comment

Labels

3 participants

Uh oh!

Conversation

Mantisus commented Feb 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Mantisus commented Feb 21, 2025

Pijukatel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Labels

3 participants

Mantisus commented Feb 15, 2025 •

edited

Loading

Pijukatel left a comment •

edited

Loading