fs cp: share one multipart transfer budget across a recursive copy#5757
Open
renaudhartert-db wants to merge 1 commit into
Open
fs cp: share one multipart transfer budget across a recursive copy#5757renaudhartert-db wants to merge 1 commit into
renaudhartert-db wants to merge 1 commit into
Conversation
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Contributor
Waiting for approvalBased on git history, these people are best suited to review:
Eligible reviewers: Suggestions based on git history. See OWNERS for ownership rules. |
Collaborator
Integration test reportCommit: 99fd543
21 interesting tests: 13 SKIP, 7 KNOWN, 1 RECOVERED
Top 8 slowest tests (at least 2 minutes):
|
renaudhartert-db
force-pushed
the
multipart/05-fs-cp
branch
from
21024d1 to
74a11f3
Compare
renaudhartert-db
force-pushed
the
multipart/04-filer
branch
from
f38b27d to
6aec672
Compare
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This was referenced Jun 28, 2026
Routes the fs cp Volumes target filer through the large-file upload engine when DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD is set. The file-level copy concurrency (--concurrency) stays at the Files-API-safe default; the multipart cloud-transfer budget is sized independently, shared across all files in a recursive copy, and applied only to the Volumes target filer. Co-authored-by: Isaac
renaudhartert-db
force-pushed
the
multipart/05-fs-cp
branch
from
74a11f3 to
99fd543
Compare
renaudhartert-db
force-pushed
the
multipart/04-filer
branch
from
6aec672 to
42be751
Compare
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
databricks fs cpand bundle library uploads to Unity Catalog Volumes go through a singlePUT /api/2.0/fs/files, which caps a file at the single-request size limit and pushes it over one connection. This stack adds chunked upload (multipart on AWS/Azure, resumable on GCP) so large files upload reliably and in parallel. The whole feature is gated behind theDATABRICKS_EXPERIMENTAL_MULTIPART_UPLOADenvironment variable and is off by default, so merging the stack changes no behavior until the flag is set.Stack
fs cpshared transfer budget (this PR)fs cpprogress barThis PR
Decouples the two concurrency knobs in
fs cp.--concurrencykeeps governing how many files copy at once and stays at the Files-API-safe default; the multipart cloud-transfer budget is sized independently and, when the flag is on, set wider for part uploads, which go to cloud storage rather than the rate-limited Files API. Keeping them separate means a recursive copy of many small files does not burst the Files API, while a single large file can still fan its parts out wide. The budget is applied only to the Volumes target filer.Testing
Covered by the existing
cmd/fstests; the decoupling and Volumes-only budget are validated with a live recursivefs cpto a Volume.This pull request and its description were written by Isaac.