RFC: Combine multiple RSpec repositories into a single one.

[xaviershay]

Summary

Combine rspec-core, rspec-mocks, rspec-expectations, rspec-support, rspec-dev (which would be mostly deprecated with this proposal), the rspec meta-gem, and possible rspec-rails into a single repository.

Motivation

Making cross-cutting changes to RSpec involves a complicated dance of PRs. Selected examples:

• Upgrading rubocop involved multiple PRs, that all needed to be somewhat kept in sync as a result of changes/feedback, and are still pending a final rspec-dev sync. As well as been a pain at the time, this has caused confusion for a subsequent contribution.
• Adding to rspec-support to fix a bug in (say) rspec-mocks requires a PR to support, a PR to mocks pointing at that branch to demonstrate that the support change works, merging the support PR, then redoing the mocks PR to point back at master. (e.g. Add specs demonstrating problem with verified doubles and keyword args. rspec-mocks#726)
• Any of the rspec-dev tasks to "sync" configuration to all repos requires a PR for each repo.
• complex and partially overlapping CI builds

It's also not always clear to issue reporters which project is appropriate for an issue. We shouldn't expect newcomers to know which sub-project an issue could be related to!

Approach

Issues

Given a motivation is to simplify issue reporting, we should migrate them all to the main rspec project (which current has issues disabled) using something like github-issues-import. We can use tags (mocks, etc...) to track what came from where, but for new issues they would be optional.

Inflight PRs

We should explicitly close or merge every open PR. This will include closing many stale PRs, that will need to be re-opened against the new monorepo. We can mitigate this with a notice period (i.e. "one month from now we're merging repos").

CI

I see two ways to set up CI:

1. Run all the specs for each project on every CI. Conceptually the simplest, potential downside is runtime. Given a lot of build time is build setup (installing ruby, gems, etc...) I suspect this will work out fine.
2. Add conditional build logic to identify which projects are likely affected based on file paths changed, and only run those. Speeds things up, but more complicated and not technically as thorough (though probably good enough).

This is where rspec-rails would be an interesting addition, because it has a test matrix with rails versions that isn't applicable to the other projects and would probably force us to option 2. I'm leaning towards keeping it out, at least to begin with. We can always incorporate it later!

For now we should benchmark option 1.

rspec-dev

We need to audit rspec-dev to see what things it does are still appropriate and how to best incorporate them into the monorepo. Given it's primarily used to "manage" multiple repos, my hope is it can be mostly deprecated. This will be investigated in the prototype suggested below.

Drawbacks and mitigations

• Git history is harder to work with. We should keep around the old git projects for spelunking purposes.
• Intermingled git logs. I'm not worried about this. Possibly consider it an upside, being able to see what was happening across the entire project.
• Easy to accidentally add unwanted dependencies. Since the code is "there" this potentially makes it easier to accidentally introduce coupling between the various components. I figure we have enough history and discipline keeping them separate that this shouldn't be an issue in practice.
• wrap up or manually transfer all hanging pull requests in Expectations/Mocks/Support/Core
• align git version tags - they overlap, and we need to keep them in sync
• CI limit of simultaneous jobs constraints our a bit tighter

Next steps

• Feedback from other members of core team.
• Create a new project to test this idea (rspec-monorepo), including setting up CI and incorporating rspec-dev.

[xaviershay]

@rspec/rspec feedback please :)

[xaviershay]

Background reading on monorepos:

• https://danluu.com/monorepo/
• https://github.com/babel/babel/blob/master/doc/design/monorepo.md
• http://blog.shippable.com/our-journey-to-microservices-and-a-mono-repository

We don't have enough code (nor should we ever) to start hitting some of the bigger downsides of a monorepo.

And also my own relevant background: I was responsible for the CI teams at Square for many years, dealing with a large Java monorepo and many disparate ruby repositories.

[soulcutter]

I support this idea.

[myronmarston]

I very much support this idea. The amount of overhead we deal with due to having separate repos is demoralizing. Elixir (the main language I dealt with the pass couple years) has first-class support for monorepos with many sub-applications via its umbrella feature and I'm a big fan of the approach when you're tooling supports it.

There's also another potential benefit we get out of a monorepo: it can greatly lessen the effort involved in extracting a feature out of the core gems and into an extension gem. For example, if we ever want to extract any_instance into a separate gem so it doesn't ship with rspec-mocks (something I'd advocate we consider for RSpec 4), with our current approach, extraction would be difficult. We'd have to refactor rspec-mocks to promote some new things to first-class extension APIs so the any_instance code could use it. That would be a lot of effort. With a monorepo, we could potentially move the any_instance code into a sub-directory of the monorepo, w/o needing to define new extension APIs. Instead, we can just include the any_instance spec suite in our travis build, so we can keep ourselves from breaking the any_instance logic.

possibly rspec-rails into a single repository.

I'm in favor of not merging rspec-rails. It's pretty different, and not part of core RSpec. The CI build is way different with all the rails versions it supports. And we've talked about potentially de-coupling it's versioning from the versioning of the rest of RSpec.

Given a motivation is to simplify issue reporting, we should migrate them all to the main rspec project (which current has issues disabled) using something like github-issues-import. We can use tags (mocks, etc...) to track what came from where, but for new issues they would be optional.

Great idea. A few additional thoughts:

• It would be good to cleanup our issues before doing this so we don't import issues that should really be closed.
• The merging of issues could potentially happen before (or after) the code is merged. The fewer changes in flight at once, the better.
• When importing an issue into the monorepo, can we include a link back to the original issue?
• It would be great if we could have the import script close the original issue, with a link to the issue in the monorepo.
• I think there's value in tagging issues with which gem the issue applies to on an ongoing basis, not just for imported issues.
• We probably want to prevent the creation of new issues once we've imported, but not turn off the issues feature of the old repos entirely, so folks can still browse/search/etc old issues. Not sure what github supports here.

This will include closing many stale PRs, that will need to be re-opened against the new monorepo.

Honestly, I think we can just close stale PRs, with a note saying the person can re-open a new PR on the monorepo if they want to see it get merged. I don't think we need to go through the effort of manually importing PRs (and automating it sounds hard). Cleaning up open PRs would be really helpful as a preliminary step.

Run all the specs for each project on every CI. Conceptually the simplest, potential downside is runtime. Given a lot of build time is build setup (installing ruby, gems, etc...) I suspect this will work out fine.

I would favor this approach over the more complicated analysis approach you mentioned. I think we can optimize this using a well-designed build matrix. The cukes are generally the slow part of our build. I'd favor doing something like:

• One matrix entry for all the spec suites (and other quick little checks like yard coverage, etc)
• One matrix entry for each cuke suite (core, expectations, and mocks)

So that would be 4x matrix entires per supported Ruby version...which sounds bad, but isn't actually any worse than our current situation, where we've got 4 builds (core, expectations, mocks, support) per supported version. And it should run simpler and faster.

We can consider not running the cukes on all supported ruby versions if there's a concern about the number of matrix entries.

This is where rspec-rails would be an interesting addition, because it has a test matrix with rails versions that isn't applicable to the other projects and would probably force us to option 2. I'm leaning towards keeping it out, at least to begin with. We can always incorporate it later!

I don't think I can even wrap my head around how we'd incorporate rspec-rails in a monorepo build. I very much advocate for keeping it separate. It's a special beast.

We need to audit rspec-dev to see what things it does are still appropriate and how to best incorporate them into the monorepo. Given it's primarily used to "manage" multiple repos, my hope is it can be mostly deprecated.

There's a couple different things rspec-dev provides:

• Canonical copies of some files that we then push out to all the other repos. Once we have a mono repo, we can just have one copy of each file, with no tooling to sync it up needed.
• A bunch of rake tasks for supporting RSpec development. Some of these are just there to deal with the pain of not having a monorepo (e.g. tasks to change branches in all repos, etc), and these can probably be dropped. Other tasks support doing things like bumping versions on all gems (which we'll still need), cutting releases, generating stats for blog posts, etc, and we'll still want these tasks--they'll just go in the root Rakefile of the monorepo.

I don't think we need to stringently audit rspec-dev and try to bring everything over--instead, we can just bring things over as we have the need for a task.

Create a new project to test this idea (rspec-monorepo), including setting up CI and incorporating rspec-dev.

This is a great idea. Would rspec-monorepo be a prototype, and then we'd re-do the merge on rspec/rspec for real? A couple thoughts/ideas:

• rspec-monorepo should probably start off as a fork of rspec/rspec so you can test out merging into an existing repo with history.
• Whoever works on this should keep copious notes on how they did it so we can easily reproduce on the rspec repo. Automating the process with some scripts is probably good, although as a one-time process we shouldn't try to go overboard.

One thing you didn't mention: how are we going to actually merge the source history of the repos? Is there a good tool for that? Obviously, all the SHAs are going to change. If possible, it would be really cool if whatever tool we used to import commits amended the message of each commit with a note like [Imported from rspec/rspec-core@abc1239237e8234bedf] so there's a clear link from each commit to the original. And we could potentially create a link the other way, too: since github lets you leave comments directly on commits, our import tool could leave an automated comment on each commit linking to its new location in the monorepo. Although, given the number of commits, I can imagine that would be super slow....so probably not worth it.

While I'm 100% on board with this idea, I can't say I'm excited (or particularly available) to do much of the work here. Is there someone who is willing to lead this effort? (Looking at you, @xaviershay :)...)

[JonRowe]

I'm on board with the mono repo idea, would certainly simplify keeping everything in sync!

On the CI side I'd recommend we not split out the various builds into matrices, as it will (ironically) slow the build down. Travis seems to limit the number of in progress builds at a time (one of the reasons why rspec-rails builds are so slow), which is fair given its a free service, so having one build per Ruby version will probably work out faster in practise.

[yujinakayama]

👍 for the monorepo idea.

Some ideas for the import of commits:

• Keep original author info, so that they will appear in the contribution graph on GitHub
• Keep original author/commit dates
• Add commits in chronological order for easier history investigation
  • Commits "Add foo" in rspec-support and "Use foo from rspec-support" in rspec-core should be close in the merged history

[xaviershay]

Some ideas for the import of commits:

Oh interesting, I was just planning to start a new repo and copy the code over, with initial commit being "see the old repos". I can play around with various ways of importing history ... guess it should be pretty straight forward since there's no way they could conflict with one another.

[myronmarston]

Oh interesting, I was just planning to start a new repo and copy the code over, with initial commit being "see the old repos". I can play around with various ways of importing history ... guess it should be pretty straight forward since there's no way they could conflict with one another.

I think for the prototype repo, copying in all the code is OK so you can work on getting CI setup, etc., but for the final merge I think the extra effort to preserve the source history as much as possible is valuable. I like @yujinakayama's ideas.

[xaviershay]

Sorry I have made basically no progress on this. I'm going to de-commit from it rather than let it go on in limbo.

I made this prototype repo to start playing around with ideas if anyone wants to pick it up: https://github.com/rspec/rspec-monorepo-prototype

[p8]

We could merge the other repos while keeping all the history.
I had some success with the steps described here (https://thoughts.t37.net/merging-2-different-git-repositories-without-losing-your-history-de7a06bba804):

git clone git@github.com:rspec/rspec-core.git
git clone git@github.com:rspec/rspec-expectations.git
cd rspec-core
git remote add rspec-expectations ../rspec-expectations
git fetch rspec-expectations
git co -b merge-rspec-expectations
git merge -S --allow-unrelated-histories rspec-expectations/master

#2665

[yujinakayama]

Recently I've been working on my project RepositoryMerger which satisfies the requirements for the repository migration, and finally it's almost complete.

Here's a prototype monorepo:

https://github.com/yujinakayama/rspec-monorepo

The largest benefit of the monorepo generated by RepositoryMerger is that we can handle the past codebase and commits in the same way as we do after the migration.

• We can checkout any commit like git checkout 3-10-maintenance and just run cd rspec-core; bundle exec rspec as always. This is equivalent to rake git:checkout[3-10-maintenance] and cd repos/rspec-core; bundle exec rspec in rspec-dev.
• All the commit history is available. Of course we can git bisect to find the commit that introduced a bug.

How it works

Though I think the monorepo generated by RepositoryMerger is basically intuitive and easy to handle, below is a brief explanation of the behavior for review from the core team members.

First we need to choose target branches, which we consider important as history and want to import into the monorepo. In other words, commits unreachable from the target branches in the original repos (e.g. stale pull requests) won't be imported. In our case, the target branches should be main and the maintenance branches (2-2-maintenance ... 3-10-maintenance).

Then we can import commit history for each branch one by one in the following way:

configuration = RepositoryMerger::Configuration.new(
  original_repo_paths: %w[rspec rspec-core rspec-expectations rspec-mocks rspec-support],
  monorepo_path: 'monorepo'
)

repo_merger = RepositoryMerger.new(configuration)

repo_merger.merge_commit_history_of_branches_named('main')
repo_merger.merge_commit_history_of_branches_named('2-2-maintenance')
repo_merger.merge_commit_history_of_branches_named('2-3-maintenance')
# ...
repo_merger.merge_commit_history_of_branches_named('3-10-maintenance')

The imported files are placed under subdirectories for each original repo in the same manner as rspec-dev.

Importing a single set of branches

For simplicity, let's consider a case of importing only a single set of branches from two repos: creating main branch in the monorepo by importing the two main branches from rspec-core and rspec-expectations:

repo_merger.merge_commit_history_of_branches_named('main')

RepositoryMerger imports all the commits from the original branches with the following rules to preserve the original commit graph structure as much as possible:

• Mainline commits (commits traceable from the target branch head, by following only first parents) are mixed in commit time order.
• Always import parent commits before children, even if children have earlier commit time than the parents.
• Non-mainline commits are branched from the same commits and merged into the same ones as the original.

Importing multiple sets of branches

Now let's consider a case of importing two sets of branches from two repos: creating main and 3-0-maintenance branches in the monorepo by importing the two sets of branches from rspec-core and rspec-expectations:

repo_merger.merge_commit_history_of_branches_named('main')
repo_merger.merge_commit_history_of_branches_named('3-0-maintenance')

The first import phase for main does exactly the same processing as the case above. In the second phase for 3-0-maintenance, it reuses already imported commits if possible instead of creating new ones so that main and 3-0-maintenance branches in the monorepo will share some of the commits until they diverge:

This result comes as no surprise.

Importing multiple sets of branches with complex commit graph

However, it may produce a surprising result if original repos have complex commit graph structures. For example, let's consider a case where the main branch in rspec-core has been merged into the 3-8-maintenance through a bug fix branch at a point after they diverged:

In this case, RepositoryMerger intentionally creates duplicate commits, instead of reusing commits that would cause commit contamination. As a result, the commit graph seems reassembled. At first glance this may look strange, but if you look at each branch this is reasonable to avoid commit contamination between target branches.

Commit message conversion

I've implemented the following commit message conversions for convenience and clarity:

• Prepend scope of the commit: e.g. Fix a bug in rspec-core -> [core] Fix a bug
• Convert repo-local GitHub issue references to absolute ones: Closes #123 in rspec-core -> Closes rspec/rspec-core#123
• Reference to original commit: add footer This commit was imported from https://github.com/rspec/REPO/commit/SHA1. to every commit.

Examples:

• yujinakayama/rspec-monorepo@a5fa9ba
• yujinakayama/rspec-monorepo@42a6b32

Tag name conversion

Tags are suffixed with their original component name: v3.10.0 in rspec-core is imported as v3.10.0-core.

Remaining issue: some old tags point orphan commits

Through trial and error in the development of RepositoryMerger, I've found a strange fact that some old tags in our current repos point orphan commits that have different root commits from the main branches.

These are the tags with orphan commits:

• rspec-core:
  • v2.11.2
  • v2.11.3
• rspec-mocks:
  • v2.2.1
  • v2.3.1
  • v2.5.1
  • v2.5.2
  • v2.6.1
  • v2.6.2.rc
  • v2.6.2
  • v2.6.3.beta1
  • v2.6.3
  • v2.6.4
  • v2.7.1

Visualized commit graphs generated with the following command (note that you cannot distinguish the orphan commits with --oneline format):

git log --graph --format='%ci %C(yellow)%h%Creset%Cgreen%C(bold)%d%Creset %s%n' --simplify-by-decoration '--remotes=*-maintenance' origin/main --tags

rspec-core:

rspec-mocks:

I'm not sure how we should handle these tags. Technically importing them into monorepo is possible but it should cause confusion since the imported branch will also be an orphan one and contain a lot of duplicate commits.

@myronmarston Do you know how these tags have been created?

---

@rspec/rspec What do you think?