feat: automatic ZipFS memory management #3691
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's the problem this PR addresses?
ZipFS
memory management was entirely manual - we always had to call one ofZipFS.prototype.{save,discard}AndClose
to avoid memory leaks. This was very unfortunate because we had to design other parts of the architecture with this in mind. For example, thefetchResult
s had areleaseFs
function that would release theZipFS
memory and that had to be called immediately after using the fetch results. This meant that it was hard for us to reusefetchResult
s between different functions and the performance was taking a hit because of this (e.g. the link step had to decompress all archives again).How did you fix it?
Made ZipFS memory management entirely automatic using
FinalizationRegistry
to automatically free the memory allocated by libzip when theZipFS
instances are garbage-collected.Now, unfortunatelyFinalizationRegistry
only got introduced in Node 14.6.0 and we still have to support Node 12.x over Yarn 3.x's lifetime. Because of this, I don't see how this PR could be merged earlier than Yarn 4.x.We're introducing this in Yarn 4.
Breaking Changes:
api.holdFetchResult
insideInstaller['installPackage']
got removed since it doesn't make sense anymore. It got introduced in 3.1 so most likely nobody's using it.releaseFs
fromFetchResult
. It shouldn't affect people that returned it or called it since it was optional anyways. It's a type-only breaking change.fetchPackageFromCache
return an object with the results rather than a tuple so that it isn't order dependent. Still kept the tuple viaObject.assign
magic so that it isn't a runtime breaking change since all third-party plugins that add fetchers that use the cache would have to be updated. Yet again, a type-only breaking change for now. I don't intend to remove the fallback until at least Yarn 5.x.releaseFs
won't be called anymore.FinalizationRegistry
has to be defined forZipFS
to work.Checklist