fs cp: show an upload progress bar for a single large-file copy#5758
Open
renaudhartert-db wants to merge 1 commit into
Open
fs cp: show an upload progress bar for a single large-file copy#5758renaudhartert-db wants to merge 1 commit into
renaudhartert-db wants to merge 1 commit into
Conversation
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: 0314fdb
21 interesting tests: 13 SKIP, 7 KNOWN, 1 RECOVERED
Top 8 slowest tests (at least 2 minutes):
|
This was referenced Jun 28, 2026
Renders a live progress bar (the same one the experimental files-upload command used) when copying a single file to a UC Volume with DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD enabled. The upload engine's progress callback is threaded to the command through the context (filer.WithUploadProgress), so the Filer.Write interface is unchanged; recursive copies, non-Volumes targets, and other writers (e.g. bundle) are unaffected. A non-interactive terminal logs coarse progress instead of drawing the bar. Co-authored-by: Isaac
renaudhartert-db
force-pushed
the
multipart/05-fs-cp
branch
from
74a11f3 to
99fd543
Compare
renaudhartert-db
force-pushed
the
multipart/06-fs-cp-progress
branch
from
36e2d3d to
0314fdb
Compare
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
renaudhartert-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
databricks fs cpand bundle library uploads to Unity Catalog Volumes go through a singlePUT /api/2.0/fs/files, which caps a file at the single-request size limit and pushes it over one connection. This stack adds chunked upload (multipart on AWS/Azure, resumable on GCP) so large files upload reliably and in parallel. The whole feature is gated behind theDATABRICKS_EXPERIMENTAL_MULTIPART_UPLOADenvironment variable and is off by default, so merging the stack changes no behavior until the flag is set.Stack
fs cpshared transfer budgetfs cpprogress bar (this PR)This PR
Adds an upload progress bar to
fs cpfor a single large file copied to a Volume (when multipart is enabled). It reuses the bar from the experimental files-upload command: a spinner that animates a "Preparing upload" label during session setup, then a single-line bar with percentage, transfer rate, and ETA. The engine reports progress through a callback that the command threads down via the context (filer.WithUploadProgress), so theFiler.Writeinterface is unchanged. Recursive copies, non-Volumes targets, and other writers (bundle) are unaffected; a non-interactive terminal logs coarse progress instead of drawing the bar.Testing
Unit tests cover the fixed-width bar invariant, the rolling-window rate meter (including burst smoothing), and the speed and ETA formatters. The interactive bar is exercised with a live
fs cpof a multi-GB file.This pull request and its description were written by Isaac.