Skip to content

Switch to yauzl for unzipping files without loading everything into memory - #835

Merged
ashfame merged 4 commits into
trunkfrom
import_memory_crash_fix
Jan 27, 2025
Merged

Switch to yauzl for unzipping files without loading everything into memory#835
ashfame merged 4 commits into
trunkfrom
import_memory_crash_fix

Conversation

@ashfame

@ashfame ashfame commented Jan 23, 2025

Copy link
Copy Markdown
Member

Related issues

Proposed Changes

  • Remove adm-zip lib in favor of yauzl which is more mature & adheres to spec tightly
  • Read zip file without loading it all into memory

Testing Instructions

  • Import a 2GB+ back up file. I tested with a 8.93GB file.

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors?
@ashfame
ashfame requested a review from a team January 23, 2025 09:39
@ashfame ashfame self-assigned this Jan 23, 2025

@fredrikekelund fredrikekelund left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great and works as expected 👍

For completeness sake (I'm currently working on https://github.com/Automattic/dotcom-forge/issues/7611), I can also confirm that the new tests also pass on Windows.

Comment thread src/lib/import-export/import/handlers/backup-handler-zip.ts Outdated
Comment thread src/lib/import-export/import/handlers/backup-handler-zip.ts
Comment thread src/lib/import-export/import/handlers/backup-handler-zip.ts Outdated

@wojtekn wojtekn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Make sense to implement improvements proposed by @fredrikekelund , though.

ashfame and others added 3 commits January 27, 2025 21:34
Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>
Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>
Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>
@ashfame

ashfame commented Jan 27, 2025

Copy link
Copy Markdown
Member Author

@fredrikekelund Thanks for the detailed code review! ❤️

@ashfame
ashfame merged commit 44e1fe0 into trunk Jan 27, 2025
@ashfame
ashfame deleted the import_memory_crash_fix branch January 27, 2025 19:00
bgrgicak pushed a commit that referenced this pull request Jan 29, 2025
…emory (#835)

* switch to yauzl for unzipping files without loading everything into memory

* change import syntax that's used

Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>

* promisifying openReadStream

Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>

* using await with openReadStream

Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>

---------

Co-authored-by: Fredrik Rombach Ekelund <fredrik@f26d.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants