Reworks the worker pool to be more configurable #5764
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's the problem this PR addresses?
This PR extends @anuragkalia's work in #5636 (it starts from the same commits and builds upon them).
The original implementation was keeping a singleton worker pool - I felt this was problematic since we may enter situations (even if only in tests) where multiple calls to
convertToZip
would want to use different pools.Converting an archive without using a worker was only possible when the concurrency was completely disabled. This was my idea, but retrospectively it felt better to me to have two settings: one for the concurrency, and another to define whether the concurrency is enabled or not.
Passing all those settings from the various fetchers would have been unwieldly.
How did you fix it?
I created a
TaskPool
interface;WorkerPool
implements it (using workers, same implementation as before), but so doesAsyncPool
(a new implementation that simply forwards the call to thepLimit
limiter).taskPoolConcurrency
, which doesn't directly refers to workers.A
getConfigurationWorker
function returns the proper task pool for the given configuration object. To make that possible, WeakMap instances can now be used as storage argument for theget*WithDefault
family of functions.Checklist