Problem/Motivation

As described in #2372815: [meta] Make the <root>/composer.json file a working example. There is a need for a subtree split of Drupal core to exist on Drupal.org. This will allow site administrators to install Drupal core via git, a git submodule, or via Composer.

This issue is currently blocking #2372815: [meta] Make the <root>/composer.json file a working example.

Proposed resolution

Create and maintain subtree split of Drupal core.

This should be a "Read-only" repository and will live at:
http://drupal.org/project/corehttp://github.com/drupal/core

This is similar to Symfony's subtree split of Symfony's components:
https://github.com/symfony

Some work has already been done to create a subtree split of Drupal core (via a script) here:
https://github.com/drupal-composer/drupal-core

Ideally, the split should happen every time a commit is pushed to the primary drupal repository via a git hook. However if that is not possible, it should at least occur every release (i.e. beta1, beta2, etc.). I believe Symony does it nightly, the more often, the better.

Remaining tasks

  1. create the drupal/core repo on github
  2. Move existing subtree commits from drupal-composer/drupal-core to drupal/core repo
  3. Add code to drupalorg to create subtree splits utilizing splitsh, and push those changes to github, and have it push 8.2.x-dev branch as a whole new branch to github.drupal.org
  4. Configure the git hooks on github
  5. Repoint (or recreate) drupal/core on Packagist to the official subtree split

Comments

lizzjoy’s picture

Project: Drupal.org webmasters » Drupal core
Version: » 8.0.x-dev
Component: Other » base system
tstoeckler’s picture

Status: Active » Postponed

Note that until #1975220: Allow a Composer user to manage Drupal, modules, and PHP dependencies with a custom root composer.json lands it is very tedious to pull in the subtree split as is discussed at https://github.com/tstoeckler/drupal-core

Therefore marking this as postponed.

davidwbarratt’s picture

Created a parent issue so there's a list of all of the things that need to be done for the <root>/composer.json file to be a working example.

davidwbarratt’s picture

Mile23’s picture

davidwbarratt’s picture

Project: Drupal core » Drupal.org infrastructure
Version: 8.0.x-dev »
Component: base system » Continuous Integration

I think this belongs in Drupal.org infrastructure since there's not a patch that can be written for this.

davidwbarratt’s picture

Since people could use a subtree split to install Drupal core as a git submodule (or just a straight Git repo) as described in the issue summary, then this issues is not really dependent on #1975220: Allow a Composer user to manage Drupal, modules, and PHP dependencies with a custom root composer.json .

However, this issue is blocking #2372815: [meta] Make the <root>/composer.json file a working example

webchick’s picture

Status: Active » Postponed

AFAIK this and all other D.o/Composer issues atm are blocked right now on DrupalCI (Testbot 2.0), per catch's comment at #1475510-64: Remove external dependencies from the core repo and let Composer manage the dependencies instead. The old testbot platform is on life support atm and we don't want to make any changes to it if at all possible, since it only makes moving to the new one more difficult.

Additionally, before asking the D.o tech team to do work on this, there needs to be a workable plan with general consensus and sign-off from at least a core maintainer (possibly Dries, if it will dramatically impact workflow/tools). AFAIK that has not happened yet for anything apart from that issue above about moving vendor stuff out of the core code base.

catch’s picture

Title: Create (and maintain) a subtree split of Drupal core » [policy, no patch] Create (and maintain) a subtree split of Drupal core
Project: Drupal.org infrastructure » Drupal core
Version: » 8.0.x-dev
Component: Continuous Integration » base system
Assigned: Unassigned » Dries
Status: Postponed » Active

I don't think the subtree split would actually require us to remove the /vendor directory from the core repo yet (and hence wouldn't affect any workflow or qa.drupal.org). All it's doing is providing a split of the core repo so that people can install it properly with composer, which is a bit different. It's related but should have zero impact on the actual core repo. People are already maintaining at least one sub-tree split on github and that also works without the core repo worrying about it.

Either way, I think we should do this:

- move this issue back to core as [policy, no patch]
- assign it to Dries
- once it's agreed move it back to infra

Personally I think this would be great to have on d.o, as long as it's managed by git commit hooks and doesn't require manual intervention. The only concern I have is that as with #1475510-64: Remove external dependencies from the core repo and let Composer manage the dependencies instead until we properly pin composer.json and/or composer.lock in the core repo an install with the subtree and an install with /vendor bundled in could have quite different versions of external libraries, but that really blocks using the subtree split for anything as opposed to just having it.

Dries’s picture

Assigned: Dries » Unassigned

I agree with it in principle, as long this is automated and read-only. Per Angie's comment in #8, I don't think this should be the infrastructure's team top priority -- I'll leave that for them to decide though.

daffie’s picture

Does anybody have the link to infrastructure team issue for creating subtree split of Drupal core?

davidwbarratt’s picture

#11,

I think this one was it until #9. Should a new one be created?

daffie’s picture

I think that the infrastructure team has its own issue queue.

davidwbarratt’s picture

Project: Drupal core » Drupal.org infrastructure
Version: 8.0.x-dev »
Component: base system » Blocked IPs

I missed this part of #9:

once it's agreed move it back to infra

Moving back to Infra. :)

davidwbarratt’s picture

Title: [policy, no patch] Create (and maintain) a subtree split of Drupal core » Create (and maintain) a subtree split of Drupal core
Component: Blocked IPs » Continuous Integration
davidwbarratt’s picture

Issue summary: View changes
davidwbarratt’s picture

Issue summary: View changes
tstoeckler’s picture

I've created a little bash script for maintaining the subtree split at https://github.com/tstoeckler/drupal-core/blob/master/subtree-split

It's not rocket science but it could either be used as is or as inspiration for something that should be run in a post-receive (or some other) hook here on Drupal.org.

The basic flow is
1. git fetch Update to latest changes (perhaps unnecessary when working directly from the canonical repository on Drupal.org)
2. git subtree split Perform the actual split. Git stores the subtree split in a commit or branch in the same repository. This takes ~2 min on my machine, so it might make sense to introduce some sort of buffer here. Also I've found that using the -b option of git subtree split (as my bash script does) is a good idea (Although the latter is possible, Git likes it better to push from branch to branch instead of from commit to branch)
3. git push Push the split somewhere.

When splitting tags some additional care needs to be taken because - as noted above - the subtree split is stuck into the same repository, but a single repository can only have a single tag with the same name. Thus, the tag deletion dance in my bash script.

webchick’s picture

Status: Active » Postponed

Raised this on the DSWG meeting today, and drumm pointed out that it wasn't clear whether this one had the official stamp of approval or was just in "wild and crazy idea" phase, so just confirming that per #10, Dries has signed-off on this, so while it may be a "wild and crazy idea" it is nevertheless an official one. ;)

Also, just to set expectations, this is not expected to get looked at until around DrupalCon LA, in favour of other higher-priority things on the Drupal.org roadmap, since the community has a workaround for now and D8 is not in danger of shipping anytime soon. :) If you disagree with this assessment, let's talk.

dawehner’s picture

Well, I think its a good time now to establish the good practises to build sites with Drupal 8.
Allowing early adopters to use the tools they way in a as straightforward as possible way would be great, as it helps the adoption rate of Drupal 8.

webchick’s picture

Right, but either way they can still use composer/packagist now, it's an implementation detail that it happens to be pulling from tstoeckler's repo atm, correct?

davidwbarratt’s picture

That's correct.

See https://www.drupal.org/node/2404989 for instructions on how to use Composer with Drupal.

tvn’s picture

Mile23’s picture

davidwbarratt’s picture

#24,

Please follow the instructions:
https://www.drupal.org/node/2404989
on how to use Drupal with Composer.

If you do not follow all of the instructions, you will not have a successful result.

MKorostoff’s picture

Issue tags: +composer
Mile23’s picture

Added a few target behaviors this will support in #2002304: [META] Improve Drupal's use of Composer.

One of these duplicates the test in https://travis-ci.org/paul-m/d8-create-project

davidwbarratt’s picture

FYI:

I mentioned this issue in my session at DrupalCon Los Angeles: Using Subtree Splits to spread Drupal into everything

davidwbarratt’s picture

Status: Postponed » Active

I'm marking this one as Active since I *think* the DrupalCI is ready(?) if it's not this can be moved back to Postponed.

davidwbarratt’s picture

Also, I've created #2513388: Create (and maintain) a subtree split of each Drupal Component as a follow-up to this one. I think it's important, but this issue is more important.

webflo’s picture

Issue summary: View changes
hestenet’s picture

kenorb’s picture

drumm’s picture

To keep this flexible enough for more subtree splits in the future, we’ll want to store what gets split to where in www.Drupal.org’s DB, rather than setting up one-off Jenkins jobs or commit hooks. An implementation of hook_versioncontrol_code_arrival() can react on commits.

https://speakerdeck.com/fabpot/a-monorepo-vs-manyrepos hints at a potentially nice helper.

Crell’s picture

fabpot has already offered to help us set that tool up for Drupal's use. I connected him with Mixologic a few weeks ago, but I'm not sure what came of it.

Mixologic’s picture

Issue summary: View changes

So, an update to this:

  1. I added a hook to packaging that we can use to manipulate an already checked out git repo, so we can have the subtree splits happen during packaging (http://cgit.drupalcode.org/drupalorg/commit/?id=637ff6b6b8daeb095eb096e9...)
  2. Fabpot released https://github.com/splitsh/lite , so the tool is open source, and we can use it, we'll just have to get a build process in place for Go.
  3. Had a quick meeting with drumm and we determined that there could be a lot of places in the drupal.org database and code that does not expect there to be copies of commits or subtree splits of things that already exist in the database, so probably the best thing for us to do is to create the splits, and push them to the drupal/ organization on github.com.
  4. Talked with webflo about the existing drupal/core namespace on packagist.org, and we want to ensure that the drupal/core subtree split does not break any existing sites that are using the existing subtree split for 8.0.x/8.1.x (https://github.com/drupal-composer/drupal-core), so we'll start with an empty repo, push the commits from the existing subtree split (that uses webflo's script), and then from 8.2.x onwards we'll utilize splitsh/lite to generate the subtree split. For the components, we'll just use splish/lite to create those splits.
Mixologic’s picture

Issue summary: View changes

The main questions I have is how broken would it be for folks to get different git hashes for a subtree split? do we need to move the drupal-composer/drupal-core commits over, or can we assume that an old repo would get updated to have all new hashes and things might still work?

tstoeckler’s picture

Well, the hashes are referenced in the composer.lock file, so assuming you have that in version you cannot run composer install anymore once we change the hashes. Assuming you're on the latest version it's not too much of a hassle, you just run composer update drupal/core and you're good to go. If you're on an older version, you're going to have to put specific constraints in your composer.json in order to make Composer fetch the old versions.

So it is going to break pretty hard for people, especially because probably not everyone using drupal/core reads Drupal Planet or wherever this might be announced. It gets worse for people running composer install as part of their deployment. That is of course not best practice, but I still think there are a lot of people doing it, and in the worst case, you are in a situation where you can't deploy anymore if we make this change.

On the other hand, I don't know how much of a hassle it would be to keep the old hashes.

We could also build the subtree split and not yet push it to Packagist, so that not everything breaks. Then we could put the drupal/core metadata on the Drupal.org Packagist/Composer endpoint. Then everyone opting in to the Drupal.org Packagist would get the new drupal/core. Then we can track when the usage of the drupal/core package on Packagist.org drops below some threshold and only then actually push the new hashes there. Does that make sense?

webflo’s picture

Well, the hashes are referenced in the composer.lock file, so assuming you have that in version you cannot run composer install anymore once we change the hashes.

We would keep the old repostiory (drupal-composer/drupal-core) as is. The composer.lock contains a full path to the repository not just the hash. Composer uses the full path during package installation. It should continue to work for existing installs even if we update metadata on Packagist.org. Running "composer update" loads the new metadata from Packagist and changes the repository path in composer.lock. The user is now on the new repository.

tstoeckler’s picture

Oops, yes, you are of course correct. I had that part confused in my head. Sorry!!

In that case it will only be a problem if people pin a specific commit in the composer.json, correct? (If so, I think that is something we can get away with breaking IMO.)

Mixologic’s picture

Great. Im going to assume that we can break pinned commits of core in composer.jsons, because that's *reallly* edge casey IMO, so I'll proceed with building a complete subtree split for drupal/core and get it to github. Then, when we're all ready to switch, we can switch the packagist urls to point to https://github.com/drupal/core.

Mile23’s picture

Just pinging here in the aftermath of https://www.drupal.org/SA-2017-001

In IRC we discovered that since the drupal-composer project is maintained by community members, it can be hard to perform a security update if your site was created with drupal-composer due to the lag time between security update release and updates to the packagist records.

RobLoach’s picture

@Mile23 We could mirror the repository over at /project/core and have http://packages.drupal.org/8 handle the repo. That would just be another thing to maintain, however.

Alternatively, feel free to create an issue in the queue and I'm sure we could add a few more people as maintainers to help out.

If you need up to the second updates, you can always fork the repository, update it yourself, and reference it through Composer's "repositories".

webflo’s picture

FYI, the process for new releases (tags) and commits is completely automated. The subtree split for 8.2.x runs every hours, and for 8.3.x and beyond every 10 minutes. The subtree split for 8.3.x and newer runs much faster and efficient because the split is build with split.sh.

I would be happy to move the build process to drupal.org infrastructure in the future.