Move Panopoly from 'manyrepos' to 'monorepo' with a subtree split

Thanks for your feedback, Matt! I'll definitely check out those articles when I have a chance.

• Prevents other distributions from including a component of Panopoly without also including all of the other code
• Makes the git commits a little harder and longer to parse for a particular component

Actually, with the subtree split, these wouldn't be the case!

The subtree split means that any time we commit something to the monorepo for, say, panopoly_widgets, a commit will automatically get added to the individual Git repo for panopoly_widgets with just the panopoly_widgets changes.

So, other distro's or just plain Drupal sites could still use the panopoly_widgets module directly, the only thing that changes is that we'd treat the individual Git repo for panopoly_widgets as read-only, and only ever commit anything to the monorepo.

As to the last point:

It is different than how we did it in D7 so documentation/user-learning will need to be re-tooled

This is certainly true for contributors to Panopoly, however, users of it wouldn't notice any difference -- they could keep downloading the individual modules and even checking out the individual Git repos.

Thanks, @populist! Yeah, this will absolutely take a some experimentation and testing before going live -- splitting the subtrees out is pretty well defined, however, merging previously "manyrepos" into a "monorepo" and splitting is less well defined. It'd be really cool if the subtree hash of the merged panopoly_* modules matched the last hash in the individual repos, so literally nothing would change in the individual repos, but that might not be possible and there'd need to be a commit to each of the individual repos to get the hashes aligned. Much experimentation ahead!

Thanks, @mrfelton! Since that's what Symfony uses (and I was largely inspired by Fabpot's evangelizing on this topic) that's exactly what I was considering. :-) But I haven't used it for a real project yet, so it's really great to hear that you've had a good experience using it in the past. If you had time to help set this up, that'd be really awesome! But if not, no worries, I'm excited to learn it

I've been looking at a couple tools to try and create the monorepo:

• https://github.com/unravelin/tomono
• https://github.com/shopsys/monorepo-tools

I tried nr 1, but after splitting again with splitsh-lite, I didn't end up with the same history and I don't want to break the history of the individual repos in this process. I'm going to try nr 2 next!

Also, this website may prove to be useful:

https://gomonorepo.org/

I was able to get monorepo-tools to work more-or-less, with the following caveats:

• I had to do a search and replace in monorepo-tools changing 'master' to '7.x-1.x' because those scripts have the 'master' branch hard-coded
• The history does come out the the same after splitsh-lite, all the way up until around Panopoly 1.57 (October 2018). The problem is, that around then, I started GPG signing my commits. This seemed like a good idea at the time (and is, generally!) but it means that the git commit hashs can't be maintained because the signature is part of the hash. However, I was able to make a single merge commit to the individual repos, which rejoins their history, but that means that all commits from October to now, will be doubled. That sucks, but I don't think it's enough to stop this effort.

The main thing that's left before I push this up to the repos on Drupal.org is that we need some CI process to run splitsh-lite and update the individual panopoly_* repos. But that does make for a kind of chicken-egg situation, because I don't want that to actually do anything until this is all in place. Maybe better would be to start with a script that maintainers could run locally?

Other things that will also need to be updated include:

• The release scripts: low priority - this just needs to happen before the next release
• The drupal-org.make and related .make files: high priority - this needs to happen right away so that we can still make builds on Drupal.org
• The tests: high priority - we depend on our tests and will need this to continue developing productively

Also, all my work above is with 7.x-1.x, but this'll all need to be done again for 8.x-2.x as well. Once I've got this working for one branch I'm sure it'll be easier the 2nd time around :-)

Updating title since this happening :-)

Ok! I've pushed this for 7.x-1.x, and also updated the drupal-org.make (works!) and apparently the tests didn't need any modification, which is awesome. :-) There is a script that maintainers can run to update the all the panopoly_* repos:

./scripts/subtree-split.sh --push

Without the --push it'll make a bunch of branches like panopoly_*-7.x-1.x (ex. panopoly_admin-7.x-1.x) that you can inspect and make sure everything is good for.

Still TODO:

1. Update scripts/create_test_branch.py so we can still test patches from the D.o queue like usual
2. Update the release scripts
3. Do the same conversion for 8.x-2.x

Alright! I updated the scripts/create_test_branch.py script so we can still easily test patches from the issue queue. I also updated the release scripts, and did a release to test (and it was a convenient time to get some non-security updates out).

However, there was an issue in the release scripts: the CHANGELOG.txt for the individual panopoly_* repos got committed AFTER the release. So, the changelog entries for 7.x-1.69 will actually show up in 7.x-1.70. I've made some changes to the release scripts to try and fix that for the next release, but I can't really fully test it now, so that'll have to wait until the next release comes up! In any case, I have some code and a plan, so I should be ready. :-)

The problem was basically that the order of the old release process needs to be reversed now. Before it worked like:

1. Make all the panopoly_* releases
2. Update the panopoly profile for those releases (because they are pulled in via the drupal-org.make by the version number)
3. Make the panopoly profile release

Now, the order needs to be:

1. Make the panopoly profile release (which includes the CHANGELOG.txt updates)
2. Run ./scripts/substree-split.sh --true which'll put the commit with the CHANGELOG.txt changes into the panopoly_* repos
3. Make all the panopoly_* releases

I'm not really ready to just automate running the subtree split, so I've broken the release into two stages, so I can run the automation for step nr 1, then run the subtree split script, then run the automation for step nr 3.

Next up: converting 8.x-2.x to a monorepo!

Alright! 8.x-2.x is on a monorepo now too. :-) I was able to move and just slightly modify most of the scripts, so that was a lot easier this time around.

Marking this is as finally closed!

A quick FYI, about the last commit...

While working on #3056745: Tests on Travis CI should use Composer to build Drupal site, I realized that we need to change how we were using the profile's composer.json for a monorepo. We needed to make all the panopoly_* modules be in 'replace' (so that a requirement for 'panopoly_core', for example, wouldn't install 'panopoly_core' again if the full profile was already there) and we needed to merge all the sub-modules requirements into the top-level profile's requirements. I added a script to do the latter.