filer: route large Volumes writes through the multipart engine (off by default)#5756
Open
renaudhartert-db wants to merge 1 commit into
Open
filer: route large Volumes writes through the multipart engine (off by default)#5756renaudhartert-db wants to merge 1 commit into
renaudhartert-db wants to merge 1 commit into
Conversation
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Contributor
Waiting for approvalBased on git history, these people are best suited to review:
Eligible reviewers: Suggestions based on git history. See OWNERS for ownership rules. |
Collaborator
Integration test reportCommit: 42be751
21 interesting tests: 13 SKIP, 7 KNOWN, 1 RECOVERED
Top 8 slowest tests (at least 2 minutes):
|
renaudhartert-db
force-pushed
the
multipart/04-filer
branch
from
f38b27d to
6aec672
Compare
renaudhartert-db
force-pushed
the
multipart/03-engine
branch
from
4e22a8a to
3c814bb
Compare
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This was referenced Jun 28, 2026
…y default) Wires the large-file upload engine into FilesClient.Write behind the DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD environment variable, disabled by default so behavior is unchanged unless the flag is set. When enabled, seekable writes to a UC Volume go through the engine (single-shot for small files, multipart for large ones); non-seekable streams keep the existing single-shot PUT. NewFilesClient gains options for a shared concurrency limiter and transfer client, and an already-exists conflict from the engine maps to fs.ErrExist so skip-if-exists keeps working. Co-authored-by: Isaac
renaudhartert-db
force-pushed
the
multipart/04-filer
branch
from
6aec672 to
42be751
Compare
renaudhartert-db
force-pushed
the
multipart/03-engine
branch
from
3c814bb to
19f0a27
Compare
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
databricks fs cpand bundle library uploads to Unity Catalog Volumes go through a singlePUT /api/2.0/fs/files, which caps a file at the single-request size limit and pushes it over one connection. This stack adds chunked upload (multipart on AWS/Azure, resumable on GCP) so large files upload reliably and in parallel. The whole feature is gated behind theDATABRICKS_EXPERIMENTAL_MULTIPART_UPLOADenvironment variable and is off by default, so merging the stack changes no behavior until the flag is set.Stack
fs cpshared transfer budgetfs cpprogress barThis PR
Routes large Unity Catalog Volumes writes in
libs/filer.FilesClient.Writethrough the upload engine, behindDATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD(off by default). When enabled, a seekable write to a Volume goes through the engine, which sends small files in a single PUT and splits large ones into parts; non-seekable streams keep the existing single-shot PUT, and everything is unchanged when the flag is off. The filer builds one shared concurrency limiter and transfer client (via newNewFilesClientoptions) so concurrent writes draw from a single bounded budget and connection pool. A 409 from the engine is mapped tofs.ErrExist, preserving skip-if-exists. Note the blast radius: the bundle library-upload path uses this same filer, so with the flag on its large libraries also go multipart.Testing
Unit tests cover the seekability check, the already-exists to
fs.ErrExistmapping, the env-var gate, and the upload-concurrency option.This pull request and its description were written by Isaac.