Benchmark PnP runtime overhead

[remorses]

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin

Describe the user story

I saw that the yarn pnp.js file extends some nodejs native functionality like require, path and fs functions

To prevent performance degradation we should make benchmark and stress tests that measure the runtime overhead, for example creating a bundle with webpack of a big project with at least 50 packages

Other performance degradations could be in glob searches

I am also scared that some of these functions reimplementations have time complexity that grows with the workspace packages you have, some of us have workspaces with 200 packages

Describe the solution you'd like

Implement a benchmark to measure the runtime overhead

Describe the drawbacks of your solution

Describe alternatives you've considered

Additional context

It is still not very clear to me why yarn reimplements fs, it would be cool to read more about it

[crubier]

I just did on a very small scale example, and there is already a significant impact of yarn berry that I did not expect.

https://github.com/crubier/yarn-benchmarks

For example, on a simple create-react-app + storybook, build storybook takes 23s with yarn berry, VS 12s with npm.

I think we suffer from this internally, on our big monorepo with many storybooks, the storybooks take a total of 90 min to build.... reducing to 45min would already help a lot...

[larixer]

@crubier Have you tried to check if the impact in Yarn berry due to zip compression, i.e. compessionLevel Yarn berry option

[crubier]

@larixer no I didn't know about this option. Let me try it now..

[crubier]

BTW What is the reason for compressing the cache? Is it only for saving space on the hard drive?

[larixer]

@crubier Yep, to save space on the hard drive or in the Git repo

[crubier]

Putting compessionLevel: 0 improves performance by around 0-10%, not much.

For example storybook goes from 21s-25s to 19s-24s

[merceyz]

build storybook takes 23s with yarn berry, VS 12s with npm.

That's not a good comparison, what you actually want to compare is PnP vs node_modules in V2

[crubier]

@crubier Yep, to save space on the hard drive or in the Git repo

I think the issue raised by @remorses here is a very good idea. I think the runtime performance impact of yarn on real-life benchmark has been very overlooked, and now might be the time to start thinking about it a bit more. On my part for example, I dont care about saving a few GB on my disk if this meant getting faster performance.

An example of the overlooking of runtime performance was #973 , which was mostly resolved thankfully (on our monorepo there is now "only" a delay of 7s before each yarn command, instead of 5minutes...)

What surprised me in this little benchmark is that yarn berry seems to have a 150-200% performance impact on many real life scripts.

[crubier]

That's not a good comparison, what you actually want to compare is PnP vs node_modules in V2

And yes, I include PnP when I say Yarn v2. The point of this issue is mostly about PnP I think.

I could add yarn v1 with pnp and yarn v2 without pnp in the benchmark.

[paul-soporan]

Your install benchmark isn't useful the way it's currently done, since it doesn't take into account if the archives are already in the cache or if the lockfile is already generated. If you need an example of a good install benchmark, here is ours; the results are here.

I could add yarn v1 with pnp

That's wouldn't be useful, as Yarn 1's PnP was just an experiment and not production ready.

yarn v2 without pnp in the benchmark.

Yes, Yarn 2 with the node-modules linker would be useful.

[crubier]

@paul-soporan

Your install benchmark isn't useful the way it's currently done, since it doesn't take into account if the archives are already in the cache or if the lockfile is already generated.

I know about that of course, I ran the benchmark only after installing in each folder (And the benchmark is ran several times in a row without cleanup). So it's a "reinstall" benchmark, not a "first install" benchmark.

[merceyz]

npm and Yarn v1 are irrelevant imo, what we're interested in is the performance overhead of Yarn V2's PnP mode vs node_modules. npm and Yarn v1 has hoisting bugs which can give them unfair advantages (v2 as far as we know, doesn't)

I think the runtime performance impact of yarn on real-life benchmark has been very overlooked

That's a false statement, i've done optimizations on real world usecases, for example:

• perf(pnp): reduce work done at runtime #1743 reduced the build time for the snowpack example in [Bug] Infinite loop / OOM when using yarn 2 pnp on a larger snowpack CRA codebase #1669 by ~54 seconds
• perf(pnp): reduce work done to find zip files and the pnp hook #1474 reduced the lint time of this repo by ~60 seconds

You can see all the performance related PRs i've done here https://github.com/yarnpkg/berry/pulls?q=is%3Apr+author%3Amerceyz+perf+is%3Aclosed

I know about that of course, I ran the benchmark only after installing in each folder. So it's a "reinstall" benchmark, not a "first install" benchmark.

The only case our benchmarks doesn't cover is runtime overhead of PnP vs node_modules, the rest we cover already https://p.datadoghq.eu/sb/d2wdprp9uki7gfks-c562c42f4dfd0ade4885690fa719c818

[crubier]

@merceyz "overlooked" does not mean "completely forgotten":

In the past 3 months you made at least 2 PRs that each improved by 100% the performance on certain use cases, this is very good and going in the right direction, but it also shows that there are still wide margins for improvement.

As I mentioned I also made a PR 7 months ago ( #998 ) which improved by 2500% the runtime performance on large monorepos. There again, there was large margin for improvement of the runtime performance.

My only point is that we should indeed start measuring this runtime performance improvement because this is now becoming the limiting factor for us internally.