Add composer-ready project templates to Drupal core

Multiple templates is easy, we could do it by providing multiple templates and subtree splits.

• drupal-project introduces a web and separate vendor folder, its the continuation of drupal-composer/drupal-project
• drupal-project-legacy keeps the tarball structure as is

This is a core requirement of the Composer initiative, so escalating to Major.

Also there's a patch! :D

Tagging for manual testing.

1. +++ b/core/scripts/composer/drupal-project-legacy/composer.json
   @@ -0,0 +1,34 @@
   +    "name": "drupal/drupal-project-legacy",
   +    "description": "Project template for Drupal 8",
   

   Can the description explain the difference between this project and drupal/drupal-project? A _readme section in the extra block might also be useful here.

2. +++ b/core/scripts/composer/drupal-project-legacy/composer.json
   @@ -0,0 +1,34 @@
   +            "drush/contrib/{$name}": ["type:drupal-drush"],
   

   I like this line, but I thought core wasn't supposed to be Drush-aware? I'm asking merely for completeness' sake.

Related to #6 - 2

The current version in on the repo root is already Drush-aware :)

Another question: will the subtree split begin at core/scripts/composer/drupal-project? If so, maybe we want to add a .gitignore which ignores the vendor directory and any directories in the webroot which are likely to be heavily customized (modules/custom, for example). I realize this is out of scope for this issue, but maybe a follow-up is called for?

Yes the subtree split should start at core/scripts/composer/drupal-project. All other required files like .htaccess, .gitignore should be shipped via drupal-scaffold.

I just want to ensure that composer create project drupal/drupal-project-legacy creates an identical copy to what we get today with a git clone (except /composer.json and /composer.lock)

If I understand this correctly, that will require the scaffold plugin to be ready first. So maybe we should block on that issue.

That sounds like a pretty good plan to me. As far as I know, Drupal CI has no way to test this kind of packaging functionality, but we could easily do it on Travis CI.

Just an idea, feel free to disagree or suggest improvements, but what do you think of renaming drupal-project-legacy to drupal-project-everything-in-document-root or drupal-project-root-is-document-root?

Also, the word "kickstart" doesn't appear in the patch at all. Should we remove it from the issue title as well?

I don't know if this is feasible, but something like this might be cool if it were:

A single project with installer paths like:

"{docroot}/core": ["type:drupal-core"],

And the ability to pass the value of docroot through when calling composer create-project, maybe something like:

composer create-project drupal/drupal-project --docroot=web 

or

composer create-project drupal/drupal-project --docroot=.

I think this issue needs some scope love.

If this is the issue where we make drupal/kickstart, then we should spec it out and only do that.

So we have:

• What's it named?
• Where should it live?
• It replicates the filesystem of the current non-Composer tarball that we have.

(Two bikesheds and a specification walk into a bar...)

I suggest outside of core/ for the location since core/ is also its own Composer package. We could have a projects/ or packages/ directory, if that won't conflict with something or be too confusing. Also should have a README.txt per project explaining the whole thing.

We can do packaging type behaviors on DrupalCI, and eventually we should move to doing that, but we don't have a way to trigger different types of builds. In some ways that would be good because we can always make sure a patch doesn't break a build. But that's a bit wasteful.

Some experimentation tells me that we can make our own packages.json file for the local file system and specify that as composer create-project --repository path/to/packages.json. This would help us write in-place patch tests of builds. I'm not sure where the canonical subtree split file is located; maybe we could repurpose that in some way.

I agree that it probably should not be nested underneath /core since we're eventually wanting to make a subtree split of core that does not contain nested packages. Granted we may be able to do that anyhow, but since we're no longer treating the core repository as our product, then yes, we can make an arbitrary top level directory that is "starter-templates" or something.

Agreed. Could be just "composer", or "composer_templates" (we tend to prefer underscore over dashes, no?).

By way of an update:

So I had been doing some work on https://github.com/drupal/drupal-project-legacy with my own fork at https://github.com/paul-m/drupal-project-legacy

The goal was to make a file-diff perfect replica of the 'tarball' package using Composer. We currently don't have a way to test this with DrupalCI, but we're using travis-ci.

I succeeded, but I had to import the Composer event scripts from core to accomplish this, since using diff means we can’t just compare the packages that were installed, but we also have to compare all the test code removal that our scripts perform when we run composer install.

I talked it over with @mixologic and the consensus was that we needed to solve the following problem: We must use those scripts to clean out test code from dependencies, but the scripts are only present in one of the dependencies, not in our template project.

That is, we must use the event scripts, but we must also not put the scripts in the drupal/drupal-project-legacy codebase.

That’s because once you use the project template, it never goes away, and more importantly it’s never modified or updated.

This is reflected in the concerns in #2990257: Determine how to handle composer scripts in drupal/drupal

The diffs in https://travis-ci.org/drupal/drupal-project-legacy/builds/443380886#L841... are there because the autoloading setups are slightly different.

I just removed the whole vendor/composer directory in my version of that travis test: https://github.com/paul-m/drupal-project-legacy/blob/passing/.travis.yml... because as your diff points out: We'll have different sets of classes loading, especially with the scaffolding plugin, and potentially different order of adding the dependencies.

So if we can run and pass tests then it's clear that the autoloading is occurring, but it might not be a carbon copy of the autoloader files. Since we don't install with --dev, we have to figure out some other suitable way of proving that autoloading works. Maybe try to do the quickstart install before we delete all the different stuff: php core/scripts/drupal quick-start demo_umami

All the differences are either components or packages that aren't added to the tarball.

You're correct that it's the merge that does it. drupal-project-legacy doesn't do the merge that drupal/drupal does.

In drupal/drupal we merge core/composer.json recursively, and core/composer.json merges all the components. We do this so we at least verify that the components' composer.json files are usable, and that they reconcile with the other dependencies.

I just spent 20 minutes waiting for various phases of composer create-project to timeout due to network issues, so congratulate me on verifying that drupal-project-legacy does in fact add a PSR-4 classmap for Drupal\Component, which catches all the components:

    'Drupal\\Component\\' => array($baseDir . '/core/lib/Drupal/Component'),

Whereas drupal/drupal sets up PSR4 for all the individual components like Drupal\Component\[YourComponentHere], which is arguably better but not by much. See especially the last line:

    'Drupal\\Component\\Uuid\\' => array($baseDir . '/core/lib/Drupal/Component/Uuid'),
    'Drupal\\Component\\Utility\\' => array($baseDir . '/core/lib/Drupal/Component/Utility'),
    'Drupal\\Component\\Transliteration\\' => array($baseDir . '/core/lib/Drupal/Component/Transliteration'),
    'Drupal\\Component\\Serialization\\' => array($baseDir . '/core/lib/Drupal/Component/Serialization'),
    'Drupal\\Component\\Render\\' => array($baseDir . '/core/lib/Drupal/Component/Render'),
    'Drupal\\Component\\ProxyBuilder\\' => array($baseDir . '/core/lib/Drupal/Component/ProxyBuilder'),
    'Drupal\\Component\\Plugin\\' => array($baseDir . '/core/lib/Drupal/Component/Plugin'),
    'Drupal\\Component\\PhpStorage\\' => array($baseDir . '/core/lib/Drupal/Component/PhpStorage'),
    'Drupal\\Component\\HttpFoundation\\' => array($baseDir . '/core/lib/Drupal/Component/HttpFoundation'),
    'Drupal\\Component\\Graph\\' => array($baseDir . '/core/lib/Drupal/Component/Graph'),
    'Drupal\\Component\\Gettext\\' => array($baseDir . '/core/lib/Drupal/Component/Gettext'),
    'Drupal\\Component\\FileSystem\\' => array($baseDir . '/core/lib/Drupal/Component/FileSystem'),
    'Drupal\\Component\\FileCache\\' => array($baseDir . '/core/lib/Drupal/Component/FileCache'),
    'Drupal\\Component\\EventDispatcher\\' => array($baseDir . '/core/lib/Drupal/Component/EventDispatcher'),
    'Drupal\\Component\\Discovery\\' => array($baseDir . '/core/lib/Drupal/Component/Discovery'),
    'Drupal\\Component\\Diff\\' => array($baseDir . '/core/lib/Drupal/Component/Diff'),
    'Drupal\\Component\\DependencyInjection\\' => array($baseDir . '/core/lib/Drupal/Component/DependencyInjection'),
    'Drupal\\Component\\Datetime\\' => array($baseDir . '/core/lib/Drupal/Component/Datetime'),
    'Drupal\\Component\\ClassFinder\\' => array($baseDir . '/core/lib/Drupal/Component/ClassFinder'),
    'Drupal\\Component\\Bridge\\' => array($baseDir . '/core/lib/Drupal/Component/Bridge'),
    'Drupal\\Component\\Assertion\\' => array($baseDir . '/core/lib/Drupal/Component/Assertion'),
    'Drupal\\Component\\Annotation\\' => array($baseDir . '/core/lib/Drupal/Component/Annotation'),
    'Drupal\\Component\\' => array($baseDir . '/core/lib/Drupal/Component'),

The scope here is to be able to build a file-perfect replica of drupal/drupal. So we can do that easily minus the autoloading files, if we prove that the autoloading works by running something in the travis test.

We have a few options:

1) Not care about this discrepancy as long as we run something to prove that autoloading works.

2) Add the merge plugin and have it recursively merge core/composer.json. This gets us a similar set of PSR4 arrays. The reason we have the merge is to exercise the composer.json files. We don't need the merge here because we exercise it whenever we use drupal/drupal. But it would maybe give us similar PSR4 arrays.

3) Make core/composer.json require the components. That's part of the goal in this issue: #2943842: Allow component namespaces to be autoloaded independently and others. So we shouldn't do that here.

Is there a PR or github branch related to this work?

Updating IS to show that the only real blocker here is #2982684: Add a composer scaffolding plugin to core

Over in the Composer Initiative meeting we decided that the scaffolding plugin and the vendor cleanup plugin should be different. #3053800: Agenda for 5-15-2019

So now we're also blocked by #3057094: Add Composer vendor/ hardening plugin to core

We'll also be able to reliably test after #3031379: Add a new test type to do real update testing but that might not be a hard blocker.

I forked https://github.com/greg-1-anderson/drupal-drupal-composer to demonstrate the legacy project using the drupal/scaffold plugin #2982684: Add a composer scaffolding plugin to core.

I'd like to document some thoughts coming out of the the most recent Composer in Core initiative meeting.

For the initiative to be successful, we will need to modify and create multiple composer packages. I was having some trouble wrapping my head around all the packages, their purposes, their file structures, and their dependencies, so I've drafted a quick summary of the plan:

drupal/core

8.8.x

This will contain Drupal core files and files that were formerly "scaffold" files within the drupal/drupal package. I.e., the structure will look like:

* assets
** scaffold
*** .htaccess
*** robots.txt
*** ...
** ...
* config
* includes
* ...

drupal/drupal

8.7.x

This branch would be deprecated. It (still) cannot be used to create new Drupal projects that can be managed with Composer.

I.e., the structure will (still) look like:

* core
* .htaccess
* vendor
* ...

8.8.x

This branch will be used to create new Drupal projects that can be managed with Composer.

I.e., the structure will look like:

* docroot
** core (provided by drupal/core package)
*** assets
**** scaffold
***** .htaccess
** .htaccess (copied from docroot/core/assets/scaffold/.htaccess)
** ...
* vendor
* ...

And it’s composer.json will include:

{
  "name": "my/project",
  "require": {
    "drupal/core": "^8.8",
    "drupal/core-strict": "^8.8",
  },
}

drupal/drupal-legacy

This package will support the legacy layout for future versions of core. I.e, it will support the layout of files used in drupal/drupal:8.7.x and prior, but will do so for drupal/core 8.8.x and forward.

The purpose of this package is to provide an upgrade path for users that downloaded Drupal using a tarball. This package will be used to continue to generate tarballs on Drupal.org using an (unchanged) legacy layout of files.

drupal/core-strict

This package will force projects to use exactly the versions of pinned dependencies that Drupal core specifies in its own composer.lock file, much like https://github.com/webflo/drupal-core-strict.

New child issue, working on the scaffolding package: #3067645: Add core scaffold assets to drupal/core's composer.json extra field

New child issue. Begone, foul merge! #2912387: Stop using wikimedia/composer-merge-plugin

#3067645: Add core scaffold assets to drupal/core's composer.json extra field is in I think we can un-postpone this.

Yes, we can start working here. We probably want to do #3076600: Create drupal/core-recommended metapackage before we call this issue done, though.

The tests for my prototype project are green again using the scaffold files from drupal/core:

https://travis-ci.org/greg-1-anderson/drupal-drupal-composer/builds/5768...

This project is a good starting point for making the templates here.

Here are some templates for the "recommended" and "traditional" project layouts. As noted in the README, the Vendor Hardening plugin is not included yet pending #3077455: Move Drupal Components out of 'core' directory.

I included a couple of extremely rudimentary tests for the provided template projects: composer install must work, and both the core directory and the autoload.php file must be placed in the correct location. We can improve the tests with #3031379: Add a new test type to do real update testing, once that is committed.

Fix typo / quote escaping error.

My naive tests did not play well with the test bot; rather than fiddle with it, I am just going to take out the simple assertions and rely on only the exit code for `composer install`, and see if we can't do something better after #3031379: Add a new test type to do real update testing.

Marking as postponed on #3077455: Move Drupal Components out of 'core' directory, which is preventing us from adding the Vendor Hardening plugin here.

+++ b/core/drupalci.yml
@@ -51,9 +51,5 @@ build:
-          - "[ -f ${SOURCE_DIR}/composer/Template/RecommendedProject/web/autoload.php ] || echo 'No autoload.php file' && exit 1"
-          - "[ -f ${SOURCE_DIR}/composer/Template/RecommendedProject/web/core/authorize.php ] || echo 'No authorize.php file' && exit 1"
...
-          - "[ -f ${SOURCE_DIR}/composer/Template/TraditionalProject/autoload.php ] || echo 'No autoload.php file' && exit 1"
-          - "[ -f ${SOURCE_DIR}/composer/Template/TraditionalProject/core/authorize.php ] || echo 'No authorize.php file' && exit 1"

Let's create a script file for these and then we can run that here.

@jibran: That's not a bad idea. However, even if we do not wait on #3031379: Add a new test type to do real update testing, we are still blocked on #3077455: Move Drupal Components out of 'core' directory. I'd recommend waiting to see how both of those issues progress before deciding how to move forward here; however, I wouldn't be opposed if someone wanted to try making a patch for #47 as a contingency.

RE #48: How about something like this?

#50 looks very similar to the script I wrote last night. :)

I'm +1 on it, except that I would put the test script somewhere in 'core/tests' rather than in 'core/scripts'.

We also need to add the vendor hardening plugin to the traditional template (but not the recommended template).

I'm going to wait until the tests in #50 complete before rolling a new patch, but anyone else who feels like pushing this forward is welcome to do so.

Postponed on https://www.drupal.org/project/drupal/issues/3080649

Added vendor hardening plugin, moved the script, and fixed up the .gitignore file for the recommended project template.

Setting the status to 'needs review' so that the tests will run, but this should go back to 'postponed' once the job is complete.

+++ b/core/scripts/drupal_project_templates.sh
@@ -0,0 +1,44 @@
+set -euo pipefail
+IFS=$'\n\t'
+
+#/ Usage:       ./drupal_project_templates.sh
+#/ Description: Container command to check default composer templates.
+#/ Options:
+#/   --help: Display this help message
+usage() { grep '^#/' "$0" | cut -c4- ; exit 0 ; }
+expr "$*" : ".*--help" > /dev/null && usage
+
+info()    { echo "[INFO]    $*" ; }
+fatal()   { echo "[FATAL]   $*" ; exit 1 ; }

I copied these utlity methods from @nicksanta's blog post https://www.nicksantamaria.net/post/boilerplate-bash-script/.

See #54

The issue that this one is postponed on is RTBC, so I expect we should be un-postponed again here soon. In the meantime, if anyone has any review comments on things that need to be changed, feedback would be appreciated. Of particular note is whether the README says what it needs to say. We could also use a change record here (the two already listed are actually for subtasks of this issue). I'll probably have time to write up the change record later this evening if someone else doesn't do it first.

Here is a suggestion.

+++ b/core/drupalci.yml
rename from core/scripts/drupal_project_templates.sh
rename to core/tests/test_composer_project_templates.sh

Let's move it under core/tests/scripts/.

+++ b/core/tests/test_composer_project_templates.sh
@@ -0,0 +1,44 @@
+#/ Usage:       ./drupal_project_templates.sh

+1 on core/tests/scripts/.

Also, can we add a @todo to this script so that we'll convert this to a build test after #3031379: Add a new test type to do real update testing?

That way you won't need bash to run this test, and it can be easily performed locally.

@todo-ing the tests is a good idea.

> When core is released, should these be 8.8.0?

Actually, we can just start using '^8.8'. With minimum-stability: dev, that will select 8.8-x.dev. Once we have a stable release of Drupal 8.8.0, and stable release of our behat/mink dev dependencies, we can switch minimum-stability to "stable".

self.version would not be right, because folks usually do not tag their projects with the version of Drupal they are using, and these are templates for copying -- they will not be used directly.

> I'm somewhat in favor of 'legacy' over traditional

We should definitely delve into the naming conventions here so that we can hone in on the best name. First off, I want to be clear that I'm okay with 'legacy', and think it's fine to go with this option if it is the consensus.The reason I prefer 'traditional' over 'legacy' is that the later implies 'deprecated'. The flat layout is definitely the less-esteemed option, but I also feel like it's probably here to stay. Bare-bones providers commonly have no-config web server setups where there is no option but to serve files from the root of the directory that you can write files to. Drupal will probably still be supporting this layout in Drupal 9 and Drupal 10. I would prefer to use the word 'legacy' only when you're actually getting legacy code - something that is not main-line Drupal. If you use the 'traditional' layout, what you get is the real Drupal.

Other names we might consider:

OK, most of those are not serious. But I care 3/10, let's just take a straw poll or something and pick one. (Last option, for the ballot stuffers: boaty-mc-boat-project).

> Are these artifacts of including cweagans/composer_patches

I copied this configuration from drupal-composer/drupal-project. In general, I think that we should strip as much optional stuff out of the template as possible, and strictly speaking, cweagans/composer-patches and its configuration thereof is optional. However, it is very common for folks to need to patch their Drupal sites, so it seemed very desirable to me to have common configuration like this available in the standard template.

This configuration is also the answer to the question I had in slack the other day: does patching drupal/core even work (since the patch was rolled against drupal/drupal), and if so, how? This config sets -p2 for drupal/core only, which adjusts for the difference in nesting levels in the patch. Patches rolled against modules don't need this setting, so they won't get it.

@Mixologic, I thought I'd test doing it with self.version, but to do that I'd like to reproduce "the existing subtree splitter" locally. Where can I learn more about that / see exactly what is being run?

Argh, I was one refresh behind on this issue, never mind.

self.version would not be right, because folks usually do not tag their projects with the version of Drupal they are using, and these are templates for copying -- they will not be used directly.

#3080649: Drupal core scaffolding : path to vendor directory in generated autoload.php file is incorrect is in; back to "needs review" (I think).

Woot! We are unblocked! Still a pile of things to do, mostly minor, in #58, #59, #60 and #61.

Pile of minor things patch

Re:

We should definitely delve into the naming conventions here so that we can hone in on the best name. ... let's just take a straw poll or something and pick one

I'll suggest that I agree with @greg.1.anderson. "traditional" by itself isn't strong language, but contrasted with "recommended" I think communicates here-there-be-dragons at the appropriate volume. I'll then request that my vote count proportionally less to bigger contributors on this initiative.

Oh, another thing on the naming conventions front: if we go with project templates such as `drupal/recommended`, `drupal/traditional` and/or `drupal/legacy`, we are folding into the same namespace as all of the Drupal projects. Today, there are no projects named "recommended" or "traditional", but there IS a project named "legacy". Applesauce.

If we go with "legacy", we'd have to add a modifier, like "legacy-project". I kind of prefer the shorter names, although the issue summary still says "drupal-project-legacy" (by which I think "drupal/project-legacy" was meant).

+1 on #66, but I can't RTBC here. I'll write up a change record, and then I think we just need to ro-sham-bo on the project names to be done (so not quite RTBC yet anyway).

Regarding #60 / #61, I was just reading the issue summary again, and it definitely says that patches is a non-goal for the template projects. So, as very useful and common as cweagans/patches is, perhaps we should take that out.

I'll do that & fix the test failure in #66.

Adding NCR and ND tags cuz we need those things. Documentation can be just a copy of the readme.

For names: +1 for drupal/legacy-project and drupal/recommended-project.

We're going to recommend the recommended project. :-) But since it's recommended and not required, it should be clear that people can roll their own if they want to.

Also, that's how you say it: Drupal legacy project, Drupal recommended project.

+++ b/core/tests/scripts/test_composer_project_templates.sh
@@ -0,0 +1,46 @@
+# @todo: convert to a build test after #3031379 is in.

Sorry I referenced the wrong issue. We should link to the more-specific #2984031: Create Build Tests For Composer and Drupal instead.

Remove cweagans/patches configuration, and adjust base pass in test script. Renamed 'drupal/traditional-project' to 'drupal/legacy-project'. Updated issue reference in test script TODO.

Update issue summary.

Provide link to follow-on task to update tarball download generation.

Update documentation to change references to "traditional" to "legacy".

Wrote a change record.

Added a documentation page (copy of the README):

https://www.drupal.org/docs/develop/using-composer/use-drupal-composer-p...

Add another follow-on task: update documentation once Drupal 8.8.0 is sufficiently ready.

I think we should not postpone on #78.1 or #78.2. Let's get this committed, as that will unblock other important work, like #3082470: Update drupal/drupal tarball generator. We can come back and make follow-on commits for those item. #78.2 already has a follow-on issue. I don't know if we need a separate follow-on for #78.1, as this issue and #3080205: Implicitly allow drupal/core and legacy scaffold projects to scaffold files. are both really close. Should the later happen to be committed first, we can adjust this issue. If this one gets committed first, we can adjust that issue.

Simplified `composer-scaffold` config enabled by just-fixed #3080205: Implicitly allow drupal/core and legacy scaffold projects to scaffold files.

Woot, #80 addresses #78.1.

On everyone's favorite topic of naming things, it occurs to me that as we take a pronounced "minimum file to get you a Drupal" approach in these templates (ex: recent patches like the removal of cweagans/patches configuration in #71 and #80), we're also moving away from what I believe was an initial goal of the initiative to stop having to tell people to use community drupal-composer/drupal-project to avoid pain.

I don't mean to or want to re-examine the minimalist philosophy of the templates, official minimal project templates definitely have a place. But what about including the minimalist philosophy in the names? This might just lead to a clearer set of project template names should a future arise where we also decide to ship templates that include more of the niceties of drupal-composer/drupal-project.

drupal/recommended-minimal-project
drupal/recommended-project

Instead of

drupal/recommended-project
drupal/but-you-really-want-this-project

And if that future doesn't happen, no biggie to call it what it is anyway.

I like the idea of #82 (and in fact was thinking similar things recently); however, today there is no difference between `drupal/recommended-project` and `drupal/recommended-minimal-project`, as we are recommending the minimum as an MVP for our first step here.

If we eventually decide that `drupal/recommended-project` should be more full-featured, then I would advise making the minimal project `drupal/minimal-project`. It would be confusing to have two recommended projects, and we'd rather recommend the one with extras, not the minimal one.

Speaking of additions, in #45, "drupal/core": "-p2" is correct for the Recommended project, but I think it should be "drupal/core": "-p1" in the legacy project (if we later expand the scope of these projects to add more 'nice' features.)

I guess it's true that since they're only templates and site builders immediately take ownership of them, it's not really a b/c break to change the contents or philosophy of a template and keep calling it the same thing.

Speaking of naming things, #37 says the consensus from a Composer Initiatives meeting was to make the relocated docroot location "docroot", but this patch still places it in "web", like drupal-composer/drupal-project does. I don't see a firm consensus on this name in the 10 July meeting minutes, and the variable that sets the name of this location is [web-root], so it is my opinion that "web" is fine.

#80 failed because #3080205: Implicitly allow drupal/core and legacy scaffold projects to scaffold files. did not get written out by the subtree splitter. Re-running #80 after that happens should make it go back to green. No additional work needed here, but blocked by that.

link to "the existing subtree splitter" in IS

With a very heavy heart and apologies but I need to reset this from RTBC.

"php": ">=5.6", this is incorrect, Drupal now requires 7.0.8. See my reaction at https://twitter.com/chx/status/1109966918813794304 -- speaking of which, won't core/composer.json enforce the php version anyways? Of course, my composer-fu is not that strong, so I have no idea really.

I also believe -- but less sure -- the descriptions could be improved. RecommendedProject/composer.json has "Project template for Drupal 8 projects with composer following drupal/drupal layout" and LegacyProject/composer.json has "Project template for Drupal 8 projects with composer following drupal/drupal layout" which really looks like the exact same.

@chx: It was very helpful for you to find and report these errors; it saves a lot of time when we fix errors before the core committer review, so thanks so much.

1) I agree with the fixes in #91.

2) It would not be correct to reduce .gitignore down to just example.gitignore as suggested in #92.

It is critical in a Composer-managaged project to .gitignore everything that is placed by Composer. /modules/contrib must be ignored, because that is where the Composer-managed contrib modules are placed. /modules/custom et. al. are not .gitignored, and are available if you want to commit a module or two that is custom to just this one site.

Perhaps we could enhance the documentation to explain how to manage .gitignore in a Composer-managed project?

We could remove Simpletest; that is gone / going away. We could remove the PHPStorm .idea directory. I think everything else needs to stay.

3) #93

Fixing the over-scaffolding should be a separate issue. Composer manages which files are installed so that it can avoid over-installing. Our scaffold tool started off simple, and therefore scaffolds every time. We could check mod dates or use other means to determine whether the scaffold files need to be refreshed. This might be a little complicated in terms of files that are appended, so we'd want to add this enhancement with caution. One thing that we could do is ensure that the scaffolding operation only runs once per Composer operation. Currently it is possible for it to run multiple times.

I don't think that we have done anything to increase the required memory for Drupal, but I did not try to reproduce or measure yet.

wikimedia/composer-merge-plugin is part of drupal/drupal, so I wouldn't expect to see it at all when using these templates, which should only include things from drupal/core et. al. And yet:

https://github.com/drupal/core-recommended/blob/8.8.x/composer.json#L54

So the thing is that core-recommended includes everything from the drupal/drupal composer.lock; this is always how the package generator has worked, but the intention really is that it should only include things from drupal/core. I think this would be hard to do in the general case, but we could certainly exclude wikimedia/composer-merge-plugin via special checking. It is the only thing in the root composer.json require section save for composer/installers and drupal/core at the moment.

Re #61

The reason I prefer 'traditional' over 'legacy' is that the later implies 'deprecated'.

Agree with that, I think in the context of a Composer project people will likely understand what "traditional" means.

I copied this configuration from drupal-composer/drupal-project. In general, I think that we should strip as much optional stuff out of the template as possible, and strictly speaking, cweagans/composer-patches and its configuration thereof is optional. However, it is very common for folks to need to patch their Drupal sites, so it seemed very desirable to me to have common configuration like this available in the standard template.

For me (and probably many others) I really appreciate the ability to just drop a patch into the composer.json, and I think to not have that available by default seems a little disappointing, especially since for every dev site I set up I'd then have to require the composer-patches package and make sure it's configured to work properly. Not including it by default could potential result in more support/feature queries because people using Drupal with Composer have come to expect the ability to easily apply patches, e.g. "I added patches to my composer.json but it's not find them" and "I included the dependency but patches aren't applying correctly and are failing with error messages", etc.

Perhaps we could have a no-dev/minimal/strict template which doesn't include this dependency and configuration, we already have a standard and minimal installation profile (although based on other comments it looks like that might already be the approach?)

Re #94

It is critical in a Composer-managaged project to .gitignore everything that is placed by Composer. /modules/contrib must be ignored, because that is where the Composer-managed contrib modules are placed. /modules/custom et. al. are not .gitignored, and are available if you want to commit a module or two that is custom to just this one site.

Does that cause problems though for setups where a mono-repo is used? For example in dev or test environments where the entire site (including contrib modules and Composer packages) is committed to one mono-repo then pulled down on the live site, this practice is used to ensure stability and consistency between dev, test and live.

#95: Yes, "must" was too strong of a word. Some workflows commit everything. I still think that ignoring composer artifacts by default is the best policy, as your diffs are minimal in PRs in that instance. We are discussing options in #composer (Drupal slack), e.g perhaps we could have two example gitignore files, and let the user choose which one to rename to .gitignore.

We are going to stick with minimal-feature templates for the initial commit, and can discuss adding features in follow-on issues.

Re: #95

Does that cause problems though for setups where a mono-repo is used?

Yes. But you're not even .gitignore-ing /vendor if you're in that boat. I'd accept as sufficiently established best practice in the php ecosystem not committing /vendor/* to git. The paths not literally being "vendor" here is just a product of `composer/installers` being nice enough to lay out composer-sourced things in a Drupal-specific way, but to my mind the principle is the same. I'm personally comfortable with having the "official"/"recommended" default adhering to this best practice.

Although re: #94:

It would not be correct to reduce .gitignore down to just example.gitignore as suggested in #92

We should think about for tarball-shipping purposes first removing this .gitignore that assumes Composer management. Or maybe renaming to example.composer.gitignore or something.

+1 to everything @mixologic said.

Okay, sounds like there's established but opposing schools of thought here. Maybe in that case making core things start manifesting a preference for one or the other isn't the best.

Maybe we don't have to. Limited to the context of this morning's .gitignore discussions, all that's at issue is what the `composer create-project` user initially should have in their new, owned-by-them tree, right? We could avoid becoming opinionated on that in a few ways, like 1) giving up and not having a file called .gitignore, perhaps offering a few 'example.thisorthat.gitignore' files in the project root, or 2) asking the user which school of thought they prefer through the the `core-composer-scaffold` plugin, and giving them the appropriate pre-prepared .gitignore.

1) is simple but litters dud (ineffectual, crufty) files about in what ought to be a clean starting state for a project
2) looks pretty easy to accomplish; there's an event unique to `create-project` dispatched by composer that we can just listen for. Super rough POC here. We'd want to use Composer's I/O, setup options so it could be specified non-interactively, move things out of handler directly, probably use Drupal\Composer\Plugin\Scaffold\ManageGitIgnore somehow etc if people like this idea. The lack of interaction presently makes me wonder if there's already been a deliberate decision not to do interaction as part of our `create-project`, so I just threw super quick thing together.

Prompting, if done, should be a follow-on issue.

I had a strong opinion about an hour ago that the legacy / traditional project should have a minimal .gitignore, and the recommended project should ignore everything placed by Composer. At this point, though, especially after #98, I think that we should commit here with no .gitignore, or perhaps with a couple of .gitignore examples, and leave the decision as to whether to do anything different as follow-on work.

I updated the Issue Summary with the things that still need to be done here, including follow-on issues that need to be created.

The level of collaboration and understanding of each other in this issue is just breathtaking. Thanks everyone.

+1 on prompting as a follow-up. I also recognized the docroot bit was a candidate for solving with prompting and was essentially previously discussed in #18, so was unsure how opposed to it folks were. I think it's fine for the docroot part to be worthy of separate templates though, that's a lot more foundational to what you're going to get overall. Prompting may be better for little details.

I started working on dealing with gitignores in the interim by offering two examples files (I think removing things is easier for people than locating the correct thing to include) but ran out of time. This is incomplete because for drupal/recommended-project there's an example.gitignore in web/, (it's always been there, not a new thing with this patch), but I think having that still there plus two in the project root is too much cruft. We should relocate it to the project root and fix up its contents for the relocated docroot. If that turns out to be too ambitious, let's remove all gitignoring from the project templates and accept the web/example.gitignore's presence.

I like #104, but I think that we should move it to a followup issue so that there is time to discuss it adequately without blocking this issue from being committed.

I like #104, but I think that we should move it to a followup issue

Okay so yank both the .gitignores?

No need to follow up on prompting and nuanced example files, is there?

I think we could have one issue to discuss what to do with .gitignore, including the possibility to prompt or not prompt, but yes, I think we should defer all decisions about what to do with .gitignore to a follow-up issue, and just leave it out here. This is consistent with the current strategy of drupal/drupal and the download archives to simply not have an active .gitignore to start.

I agree with @Mixologic in #98

I think the 'dont commit vendor' mantra isn't actually true in a lot of cases. Because the much much more important mantra of "dont run composer in production" takes precedence, and thus, somehow, you need to get a packaged vendor directory to production.

When you have a multi-environment setup with a mission critical web application you run composer commands in dev/test/staging, never in production, because you need to be absolutely sure that when it works in test/staging it'll work in production and that there's no risk of a nasty surprise.

However, this is where we could also leverage other features of composer - we could put a suggests in for cweagans/composer-patches that tells users 'if you need to patch your site, try cweagans/composer-patches' - same for drush and console.

Nice idea, but do we also want to find a way to ensure it's configured correctly, for example:

        "patchLevel": {
            "drupal/core": "-p2"
        },

Including these configuration by default or adding it programtically reduces the risk of people running into issues that result in support tickets for us. (Also makes my life as a developer that little bit easier)

Re #100

2) asking the user which school of thought they prefer through the the `core-composer-scaffold` plugin, and giving them the appropriate pre-prepared .gitignore.

That sounds like an interesting idea and worth exploring.

That being said, I'm currently wondering why we even need to provide a .gitignore in the first place, is it just to be helpful and support these different scenarios already described, or is there a more technical reason?

Re #101

At this point, though, especially after #98, I think that we should commit here with no .gitignore, or perhaps with a couple of .gitignore examples, and leave the decision as to whether to do anything different as follow-on work.

I think that partly hints at an answer to my question above, so if there's no technical reason to include a .gitignore, let's just discuss more in a follow-up.

Re #102

The level of collaboration and understanding of each other in this issue is just breathtaking. Thanks everyone.

😃

Re #103

Granted adding some interaction *does* add some complexity. As soon as this goes in somebody will want those questions in another language. Somebody ought to design a standard way that our command prompts interface with the user so that every command doesn't feel like its a different product (colors, formatting etc). Getting the wordsmithing correct so that people get what we're asking.

Another follow-up perhaps?

Re #104

Prompting may be better for little details.

👍

That's also the philosophy that Drupal Console seems to use when generating things, for example to generate an Entity Type there is two distinctly different commands, one for generating a Config Entity Type and the other for generating a Content Entity Type, the questions you get asked are mostly similar but are all relating to small details about the configuration of the Entity Type and what features it supports.

diff --git a/composer/Template/README.txt b/composer/Template/README.txt
index 1d0c793203..3818c709b8 100644
--- a/composer/Template/README.txt
+++ b/composer/Template/README.txt
+  example.gitignore:
+  Includes the files managed by Composer into your git repo.
+  Useful if you prefer a "monorepo" where site updates are deployed using only
+  git. This is the same example.gitignore shipped with traditional Drupal
+  archive releases.
+
+  example-composerdeployment.gitignore:
+  Excludes all files managed by Composer from your git repo.
+  Useful if you prefer keeping your repository limited to your project's code,
+  and you use a more complex deployment process to build your complete site.

I like these descriptions, concise yet clear, +1

Re #106

I like #104, but I think that we should move it to a followup issue so that there is time to discuss it adequately without blocking this issue from being committed.

+1 This issue is already very long, a follow-up would allow us to workshop a bit more as well.

Re #108

I think we could have one issue to discuss what to do with .gitignore, including the possibility to prompt or not prompt

+1

So all needs doing not in follow-on issues is destroying two files. I am good at destroying things. :)

This is against #91, not my last patch.

Cool, looks good. Let's make sure all the follow-on issues are created before setting RTBC. Haven't had much time this weekend, but I'll work on typing some up if that changes.

X off .gitignore and .gitignore follow-up issue tasks. That issue is #3082958: Add gitignore(s) to Composer-ready project templates

Update issue summary with links to three of the follow-on issues. Memory limit investigation remains.

Add cross-reference to final (for now) follow-on issue in summary.

All follow-on issues have been made, and all reviewer comments have been addressed, so back to RTBC.

Since we're talking about follow-ups, I'll just highlight part of my comment #110 which is quoting @Mixologic in #103,

Re #103

Granted adding some interaction *does* add some complexity. As soon as this goes in somebody will want those questions in another language. Somebody ought to design a standard way that our command prompts interface with the user so that every command doesn't feel like its a different product (colors, formatting etc). Getting the wordsmithing correct so that people get what we're asking.

Another follow-up perhaps?

I'm just highlighting this, since that concern was raised by @Mixologic I'm assuming it would be up to him if he feels that should be addressed in a follow-up.

Perhaps a well-documented .gitignore with many popular settings (like .idea) all commented out, like many config defaults?

Perhaps a well-documented .gitignore with many popular settings (like .idea) all commented out, like many config defaults?

@j. ayen green: we're continuing the .gitignore discussing on the issue #3082958: Add gitignore(s) to Composer-ready project templates, please do share your thoughts over there as well.

Update issue summary with result of memory investigation.

issue credits

Committed ef3544a and pushed to 8.8.x. Thanks!

The level of collaboration and understanding of each other in this issue is just breathtaking. Thanks everyone.

I concur

🎉

Composer intiative team: Now would be a good time to start thinking about a summary of all of the achievements in the 8.8 cycle that would go in the release notes, as I think just adding all the release note snippets together won't do it justice. Something that really fanfares all the big goals you've ticked off.

Congrats.

Not the release notes, but rather the frontpage post. (Release notes are just for disruptions). Tagging accordingly!

Now would be a good time to start thinking about a summary of all of the achievements in the 8.8 cycle that would go in the release notes

You say that as if the initiative is finished. :-)

Next: #3047468: Create a tool to convert a non-Composer-based Drupal to a Composer-based one

Should we add a guideline or documentation officially for how to switching from the 3rd party template drupal-composer/drupal-project to this one? I think there are a lot of drupal-composer/drupal-project users.

Fix a typo in the release-notes snippet.

@jungle: We can keep track of documentation needs in #3082601: Update Composer documentation to match Drupal 8.8.0+ capabilities

Re #127. Composer templates work as starter kits. The are only used for creating new projects. Once you've run composer create-project command the only way to switch to other template is removing everything and starting from scratch. So that existing sites created with drupal-composer/drupal-project don't have to update anything.

@chi: That is not quite true in this instance. Drupal 8.8.x introduces a new scaffold component. The new scaffold code has some new features that are useful for site builders, but beyond that, will also greatly reduce the load on drupal.org when a new Drupal version comes out. We therefore want to encourage users of the existing scaffold component to start using the new scaffold component. In addition, the files that drupal-composer/drupal-scaffold is downloading will be removed in Drupal 9, so we expect everyone to migrate before then.

Migrating should be a simple matter of renaming some components. There's a start (a composer.json for new projects) at https://github.com/drupal-composer/drupal-project/pull/522

Needs a follow-up to update documentation!

It's the related issue: #3082601: Update Composer documentation to match Drupal 8.8.0+ capabilities

What are the chances of #3086489: Exclude development libraries from templates' composer.json by default being committed before the release of Drupal 8.8? IMO it will fix a lot of issues for less experienced users and will also simplify documentation with having the same steps for all composer supported versions of drupal instead of different instructions before and after it lands.

It might help to mention that on the other issue rather than this closed one.

I can see why Drupal wants to maintain https://github.com/drupal/recommended-project and https://github.com/drupal/legacy-project.

What I can't understand is why they are officially recommended by Drupal. They don't even have a README.

https://github.com/drupal-composer/drupal-project is light years ahead and should be officially recommended in my opinion.

They don't even have a README.

They do
https://git.drupalcode.org/project/drupal/-/blob/8.8.x/composer/Template...