[Feature] Allow opting-out of vendoring berry's files

[aduh95]

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin

Describe the user story

I am a Yarn user, I have run yarn set version berry on my repo. Now I would like not to commit .yarn/releases; it's not my code, I am not confortable having it on my CVS; also, upgrading to a new version produces huge diffs.
One of the reason I am using Yarn, it is to avoid storing vendor files; I feel forcing users to vendor is not the way to go.

Describe the solution you'd like

We could imagine yarn set version xx keeps the actual version number on the .yarnrc.yml; when running yarn, it will first check if the yarnPath file exists, and if not, checks if a yarnVersion is defined and download it from the network. Note that it doesn't change anything for projects that vendor the file, while providing an easy opt-out (adding .yarn/ to .gitignore).

Describe the drawbacks of your solution

Besides the debat vendoring vs using a package manager, I can't think of any.

Describe alternatives you've considered

Put the .yarn folder and the .yarnrc.yml to my .gitignore and put a note on the Getting Started section of the README to run yarn set version berry...

[arcanis]

Although doable, I would prefer to avoid downloading binaries from remote sources on commands that aren't supposed to have anything to do with the network (such as yarn run, for example). Otherwise, projects using Yarn may end up being unusable or vulnerable should it die someday (I don't find that likely to happen anytime soon, but I always try to take the bus factor into account).

I know that a few people from the Node project have toyed with the idea "what if Node didn't ship with any package manager", or "what if Node shipped with a package manager installer" rather than just npm. Maybe it would be the right time to start a discussion to see how we could make it easier for everyone to manage multiple package manager versions?

[aduh95]

projects using Yarn may end up being unusable or vulnerable should it die someday

One day, the npm registry will be gone forever, and then most of my projects using Yarn will become unusable. I guess that's a risk I willing to take, as I don't vendor my npm dependencies and use Yarn to download them for me. But I see your point regarding the security aspect.

Maybe limit the proposed behaviour to yarn install? That command already downloads dependencies from the network, one could argue yarn-berry.js is also a project dependency.

Maybe it would be the right time to start a discussion to see how we could make it easier for everyone to manage multiple package manager versions?

I couldn't agree more, that would be awesome! But Berry would probably keep its "per-project" install policy though, right?

[arcanis]

I couldn't agree more, that would be awesome! But Berry would probably keep its "per-project" install policy though, right?

Yes. But depending on how this package manager installer would work, it could be the right place to store this "install on execution" logic. For example, let's say that (very basic proposal I just came up with, refine it at will):

• Node ships binaries named yarn and pnpm (it could apply to npm as well, but let's ignore that for a minute as there's political issues as well)

• Those binaries would just contain bootstrappers similar to the following

  find closest .pm-version from cwd
  if there is none
      abort and print an error requesting to select one with "node --pm-init <name>"
  
  read the package manager and version from the .pm-version
  if the .pm-version references another package manager
      abort and print an error requesting to use the other package manager
  
  check if the specified version is installed inside $HOME/.node/pms/<name>/<version>.js
  if it isn't
      abort and print an error requesting that the user runs "node --pm-install"
  otherwise
      run the binary we found
  

• And finally, it would require Node to ship a node --pm-init <name> mode which would download and store locally a given version of the package manager and would write the .pm-version file.

By doing something like this, we would solve many problems at once:

• People using zero-install wouldn't have to run npm install -g yarn just to have a Yarn binary into their global PATH (binary that they wouldn't use anyway)

• Runtime performances for people using zero-install would be significantly improved, because we wouldn't have to parse a 6MB file just to shell out to another project.

• We would be able to prevent mistakes when one project is used instead of another, leading to different installs and subpar developer experience (especially when one project uses features unique to the package manager it's been created for).

[andersk]

While we’re waiting for such a .pm-version mechanism to become widely available (which could take years), is it worth trying to see how close we can come with only Yarn changes? This would help users who are interested in adopting Yarn 2 now, and would also provide experience to help build a case for .pm-version and inform its design.

Although doable, I would prefer to avoid downloading binaries from remote sources on commands that aren't supposed to have anything to do with the network (such as yarn run, for example).

Similarly to your above proposal, when there’s a configured Yarn version that hasn’t been downloaded yet, yarn run could print an error telling you to first yarn download-yarn (or yarn install, if we’re comfortable with that downloading Yarn).

Otherwise, projects using Yarn may end up being unusable or vulnerable should it die someday (I don't find that likely to happen anytime soon, but I always try to take the bus factor into account).

It could download Yarn from the registry—and if the registry is hit by a bus, even a vendored Yarn probably isn’t going to be too useful. 🙂 In any case, this would of course be an opt-in mechanism for projects willing to accept the risk of relying on external dependencies.

[arcanis]

Right, but I'm not sure Yarn core is the right layer to implement this (similar to how Node doesn't implement nvm / fnm / nodenv / etc). The system I highlighted can be implemented on top of it.

and if the registry is hit by a bus, even a vendored Yarn probably isn’t going to be too useful

It does. For example, none of my companies will bat an eye when the registry will die (or even regularly goes down, like it did twice these past weeks), because we don't rely on it for our deployments. Something something offline mirrors / check-in your cache 🙂

[andersk]

It might not be technically ideal, but the reason I think this belongs in Yarn core is that, from the perspective of a project considering adopting Yarn 2, it’d be a lot more palatable to ask developers to upgrade their global yarn to 1.23 than to

• wait years for a yarn wrapper to make its way into Node, if that ever even happens, or
• ask developers to replace their global yarn with a weird third-party yarn wrapper, or
• ask developers to upgrade their global yarn to 2.x (breaking compatibility with Yarn 1 projects), or
• commit a vendored yarn into the repository (even though it is likely bigger than the rest of the repository, ruins git grep, and can’t realistically be audited).

I understand that some projects will be comfortable vendoring yarn, and they can keep doing that. But for others, it’s not a good option.

Node users can get by without nvm and friends in the typical case because there’s a reasonably smooth migration path between Node versions—if you install an arbitrary version as the global node and upgrade it occasionally, you’ll probably be happy. The Yarn 1 → 2 upgrade isn’t like that and isn’t intended to be, so another way to bridge that gap is needed.

[arcanis]

I conceptually agree but I personally don't have time to invest into developing this tool at the moment, so that's something that someone else will have to take ownership of.

I'd encourage you to go ahead and build a prototype of the specs I outlined outside of the Yarn repository (something that can be quickly installed with npm i -g @myusername/pm, for example). Then once there's a better idea of the problems and solutions we can decide whether we want to publish it as an official package or not.

[merceyz]

For the CLI you can use corepack https://github.com/nodejs/corepack. Uninstall yarn, run npm install corepack -g, and add "packageManager": "yarn@2.3.3" to your package.json and corepack will handle downloading and running the correct version

[larixer]

I have written the tool that works almost the way the topic starter @aduh95 described, it computes and stores URL of the Yarn 2 bundle and URLs of official plugins in a JSON file .pinyarn.json, it also writes a small (less than 4Kb) wrapper .pinyarn.js which you can easily review. The wrapper is responsible for downloading Yarn bundle and Yarn plugin if they have not been downloaded yet, because the wrapper is very small it does not affect Yarn 2 boot time:
https://github.com/sysgears/pinyarn

[Californian]

I'm baffled and would really like to understand what I'm missing here, because there has to be some extenuating circumstance to explain why a team of such obviously-competent developers would make a decision that would cause me to reject a software engineering candidate if they offered it as a solution in an interview. I would legitimately like to understand, I'm not just going on a rant for ranting's sake.

How is checking in an entire release of a package manager into your VCS a good idea? pip isn't included in Python repos; that would cause insane bloat, as it does here.

Also, the whole point of versioning is to maintain consistency without having to copy individual files. Add a hash if you want more certainty.

As for installing binaries/executables from remote, I think this was implied in the response, but just to be clear, that's the whole point of what yarn itself does. And the solution here is the same: host a mirror of the releases yourself. Then you only have one copy of each version you use in your projects, which saves resources for every set of packages that share a common version.

I apologize if this isn't worded as productively as it could be, it was just a bit of a shock to see this when switching to using berry and it's really making me revisit that decision.

[merceyz]

Now that Node.js ships with Corepack and Yarn 4 will prefer it over yarnPath if possible, do you consider this issue resolved @aduh95?

[aduh95]

Yep, I’ve been using Corepack for some time now, and it does fulfill my use case. Hopefully rest of the ecosystem will adopt it swiftly too!