-
Notifications
You must be signed in to change notification settings - Fork 7.2k
[DNR] [deps] Use uv for requirements compiled #60622
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request migrates the Python dependency compilation process from pip-tools to uv, which should provide a significant performance improvement. The changes in ci/ci.sh correctly fetch the uv binary from the Bazel workspace and use it to compile the requirements. The pyproject.toml and the generated requirements_compiled.txt are updated accordingly, with the latter now having more specific and reproducible dependency markers. I've added a couple of suggestions to clean up the CI script by removing redundant code and installations.
|
|
||
| echo "Target file: $TARGET" | ||
| pip install "pip-tools==7.4.1" "wheel==0.45.1" | ||
| "$UV_BIN" pip install "pip-tools==7.4.1" "wheel==0.45.1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| # pip-compile --verbose --resolver=backtracking \ | ||
| # --pip-args --no-deps --strip-extras --no-header \ | ||
| # --unsafe-package ray \ | ||
| # --unsafe-package pip \ | ||
| # --unsafe-package setuptools \ | ||
| # -o "python/$TARGET" \ | ||
| # python/requirements.txt \ | ||
| # python/requirements/lint-requirements.txt \ | ||
| # python/requirements/test-requirements.txt \ | ||
| # python/requirements/cloud-requirements.txt \ | ||
| # python/requirements/docker/ray-docker-requirements.txt \ | ||
| # python/requirements/ml/core-requirements.txt \ | ||
| # python/requirements/ml/data-requirements.txt \ | ||
| # python/requirements/ml/data-test-requirements.txt \ | ||
| # python/requirements/ml/dl-cpu-requirements.txt \ | ||
| # python/requirements/ml/rllib-requirements.txt \ | ||
| # python/requirements/ml/rllib-test-requirements.txt \ | ||
| # python/requirements/ml/train-requirements.txt \ | ||
| # python/requirements/ml/train-test-requirements.txt \ | ||
| # python/requirements/ml/tune-requirements.txt \ | ||
| # python/requirements/ml/tune-test-requirements.txt \ | ||
| # python/requirements/security-requirements.txt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cursor Bugbot has reviewed your changes and found 4 potential issues.
| python/requirements/ml/train-test-requirements.txt \ | ||
| python/requirements/ml/tune-requirements.txt \ | ||
| python/requirements/ml/tune-test-requirements.txt \ | ||
| python/requirements/security-requirements.txt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing index URL emission in uv compile output
High Severity
The uv pip compile command specifies --index and --find-links for resolution, but unlike pip-compile, uv does not emit these to the output file by default. The --emit-index-url flag is required but missing. As a result, requirements_compiled.txt no longer contains the --extra-index-url https://download.pytorch.org/whl/cpu and --find-links lines that were previously present. This could cause downstream pip installations to fail when packages are only available from the PyTorch CPU index.
Additional Locations (1)
| else | ||
| UV_TARGET="uv_x86_64-linux" | ||
| UV_SUBDIR="uv-x86_64-unknown-linux-gnu" | ||
| fi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wrong uv binary selected for Intel Mac architecture
Medium Severity
The darwin case selects uv_aarch64-darwin for all macOS systems. However, the earlier arm64/aarch64 check (lines 20-26) returns early before this code is reached on Apple Silicon Macs. This means only Intel Macs (x86_64-darwin) would execute this branch, but they would receive the ARM binary which cannot run on Intel architecture.
|
|
||
| echo "Target file: $TARGET" | ||
| pip install "pip-tools==7.4.1" "wheel==0.45.1" | ||
| "$UV_BIN" pip install "pip-tools==7.4.1" "wheel==0.45.1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unused pip-tools package installed but never called
Low Severity
The code installs pip-tools==7.4.1 via uv pip install, but then uses uv pip compile (uv's built-in compile command) instead of the pip-compile command from pip-tools. The pip-tools package is installed but never used since the old pip-compile invocation is now commented out. This wastes CI time installing an unnecessary dependency.
| # python/requirements/ml/train-test-requirements.txt \ | ||
| # python/requirements/ml/tune-requirements.txt \ | ||
| # python/requirements/ml/tune-test-requirements.txt \ | ||
| # python/requirements/security-requirements.txt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.


using uv for requirements compiled
using uv binary defined in bazel workspace to compile deps