Skip to content

feat: Replace HeaderGenerator implementation by browserforge implementation#960

Merged
vdusek merged 41 commits into
masterfrom
remove-header-generator
Feb 25, 2025
Merged

feat: Replace HeaderGenerator implementation by browserforge implementation#960
vdusek merged 41 commits into
masterfrom
remove-header-generator

Conversation

@Pijukatel

Copy link
Copy Markdown
Collaborator

Description

Replaces internal HeaderGenerator implementation by BrowserForge implementation.

Issues

Pijukatel and others added 30 commits December 16, 2024 14:50
TODO: Tests, JS page function for injecting fingerprint json.
Todo: Make it work with page.add_init_script
Added dev test
TODO:
Add proper tests
Sync header generation with fingerprint generation (use gen from browserforge)
Added more tests.
Use browser_pool_options from crawler to pass fingerprint related stuff to be similar to JS
Draft adapter and basic test.
TODO: Solve circular imports
Remove types unused in this change.
Format, lint, type check.
(Hint about browserforge being just implementation detail and not core functionality.)
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Update docstrings.
Add example code + doc page.
(To avoid some import race conditions downloads when running pytest with multiple processes)
@github-actions github-actions Bot added this to the 107th sprint - Tooling team milestone Feb 6, 2025
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Feb 6, 2025
@Pijukatel Pijukatel changed the title Replace internal header generator by browserforge implementation Feb 6, 2025
@Pijukatel Pijukatel marked this pull request as ready for review February 6, 2025 09:06
@Pijukatel Pijukatel requested review from janbuchar and vdusek February 6, 2025 09:06
@Pijukatel Pijukatel added the enhancement New feature or request. label Feb 6, 2025
@vdusek vdusek requested a review from Mantisus February 19, 2025 07:54

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py Outdated
Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py Outdated
Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py Outdated
Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py Outdated
Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py Outdated
Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py Outdated
Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py
Pijukatel and others added 2 commits February 25, 2025 13:03
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
@Pijukatel Pijukatel requested a review from vdusek February 25, 2025 12:11

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/crawlee/fingerprint_suite/_browserforge_adapter.py
@vdusek vdusek merged commit c2f8c93 into master Feb 25, 2025
@vdusek vdusek deleted the remove-header-generator branch February 25, 2025 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

4 participants