Problem/Motivation

The Drupal template projects (drupal/recommended-project and drupal/legacy-project) include dev dependencies in their composer.json file in the repository. Drupal infrastructure automatically removes these when running the subtree split for any tag (e.g. 8.9.0-alpha1), but allows them to remain for branches (e.g. 8.9.x-dev).

Proposed resolution

This behavior may be good enough. We could also consider removing the dev dependencies from the repository for 9.x.

Remaining tasks

We should update the documentation to clearly describe the situation.

User interface changes

None.

API changes

None.

Data model changes

None.

Old Summary

On the production server, and as a site builder, you shouldn't need the development libraries. So I suggest that development libraries are not included by default in Drupal core. Also, in that way the --no-dev and --update-no-dev parameters are no longer necessary for everyday-site-building.

How to create and manage a project without development libraries in the new version 8.8, where -n equals --no-interaction:

$ composer create-project drupal/recommended-project:^8.8@dev -n --no-dev mysite
$ composer update --with-dependencies --no-dev
$ composer require --update-no-dev drupal/admin_toolbar

On one hand, adding the developer tools project with for example a command like composer require drupal/devtools will be an extra step for Drupal developers, who are deep in the code. On the other hand, the benefits for those who mostly don't need these tools, like new Drupal users, site builders and dev ops people, there will be benefits. For example for dev ops people, the reduced complexity will mean less risk of unforeseen Composer events during a security update situation, like a lingering git commit blocking everything, which can grind the process down to a halt.

If the composer update --with-dependencies process breaks down, while applying late night security updates, having to debug why it happens, while under the the pressure of rolling out updates is not optimal, and might make dev ops people a little less fond of Drupal in the long run. Running composer update should be as boring a non-event as possible, and the chances of a smooth process should be maximized, by getting potential obstacles out of the way.

While there "shouldn't" be a problem, stuff happens, like with .git repository should not be included with composer package, where rfay comments:

This seems to break every composer update I do, have to remove the coder directory in vendor to fix.

Changing to a default installation, without dev dependencies

Pros

  • For new Drupal users there will be less moving parts and reduced complexity
  • No need to remember to use --no-dev and --update-no-dev
  • You don't get a screen load of updates, and changed composer.json if you forget to add --update-no-dev
  • Composer can calculate dependencies faster during composer update
  • Less risk of unforeseen Composer events, like a lingering git commit blocking composer update
  • Less changes in composer.lock, giving less results running git diff and better overview

Cons

  • The work involved in containing the developer tools in its own drupal/devtools project
  • Developers have to run composer require drupal/devtools

Statistics

Result with --no-dev, 53 packages

$ composer create-project drupal/recommended-project:^8.8@dev -n --no-dev mysite
Installing drupal/recommended-project (8.8.x-dev 995d694d3cf9c0863bb170e7c62c8f1aa6d35437)
  - Installing drupal/recommended-project (8.8.x-dev 995d694): Cloning 995d694d3c from cache
Created project in mysite
Loading composer repositories with package information
Updating dependencies
Package operations: 53 installs, 0 updates, 0 removals
  - Installing composer/installers (v1.6.0): Loading from cache
...

Stats: 24.534 items (and 91 hidden), totalling 391,2 MB

Result without --no-dev, 106 packages

$ composer create-project drupal/recommended-project:^8.8@dev -n mysiteDEV
Installing drupal/recommended-project (8.8.x-dev 995d694d3cf9c0863bb170e7c62c8f1aa6d35437)
  - Installing drupal/recommended-project (8.8.x-dev 995d694): Cloning 995d694d3c from cache
Created project in mysiteDEV
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 106 installs, 0 updates, 0 removals
  - Installing composer/installers (v1.6.0): Loading from cache
...

Stats: 29.914 items (and 208 hidden), totalling 414,8 MB

Release notes snippet

Development dependencies have been removed from the Composer Template Projects. See the change record for information on how to add development dependencies. It is recommended to only install development dependencies in development environments.

Comments

ressa created an issue. See original summary.

greg.1.anderson’s picture

Issue tags: +Composer initiative

There has been some talk of not including dev dependencies by default in the project templates, since running the Drupal tests is only supposed to be done from a git clone. However, we might need to wait until Drupal 9 to make this change, if we decide that we want to do it.

If dev dependencies are included by default, then those who do not wish to include them may run composer with the --no-dev flag. If dev dependencies are not included by default, then those who want them will need to know which project to require to get them. This is one reason why there has been some reluctance to make this change.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

ressa’s picture

Issue summary: View changes

Thanks for outlining the situation @greg.1.anderson. I understand there would probably be quite a bit of work involved in changing to a default installation without dev dependencies, so that user have to actively add developer tools, with a command like composer require drupal/devtools. I have updated the Issue Summary, with a bit more description, as well as a list of Pros and Cons.

greg.1.anderson’s picture

@mixologic and I were just talking about this issue. We are renaming drupal/dev-dependencies to drupal/core-dev for another purpose (so that composer update drupal/core* would get both core-recommended and core-dev at the same time; c.f. #3090684: Rename drupal/dev-dependencies and drupal/pinned-dev-dependencies).

It seems like it might not be too burdensome to ask a user to run composer require drupal/core-dev if the dev dependencies are needed. Perhaps we could even utilize #3087482: Add 'next steps' info to Composer installation output to socialize this.

We also have to decide if we're going to make this change to both drupal/recommended-project and drupal/legacy-project, or perhaps keep the dev dependencies in legacy (sacrificing consistency for maintaining expectations for legacy).

ressa’s picture

That sounds really great, thank you and @mixologic for bearing this issue in mind, I am glad to hear that it might not be as hard to do as initially feared.

Hearing Dries' talk in his Driesnote today, about lowering the barrier of entry as much as possible to make it easier for first time users to enter Drupal, excluding development libraries by default seems to me the logical choice, over having (mostly) developers who need the drupal/core-dev tools run a single command.

I agree that mentioning how to get the drupal/core-dev tools is an obvious candidate for inclusion on the #3087482: Add 'next steps' info to Composer installation output page.

Just to make sure, should development libraries eventually be excluded by default in Drupal core, the commands below will download only what the added parameter --no-dev downloads today (around 53 packages), and that afterwards, there will be no need to add --no-dev and --update-no-dev parameters to keep Drupal core lean, and free of drupal/core-dev tools, right?

$ composer create-project drupal/recommended-project:8.x-dev my_site -n
$ composer require drupal/admin_toolbar
$ composer update --with-dependencies

By the way, would it be possible to somehow set --with-dependencies as default?

I feel like the less clutter, the better. But of course, if it is a necessary Composer parameter, I understand if it has to stay.

It seems like the -n (--no-interaction) can be set in composer.json, with the COMPOSER_NO_INTERACTION parameter. Worth considering? If this parameter was added, it could be mentioned on the #3087482: Add 'next steps' info to Composer installation output page for expert users, that it can be disabled in composer.json, if desired.

Regarding drupal/recommended-project and drupal/legacy-project, I agree with you that excluding development libraries by default is most important for drupal/recommended-project, while keeping the dev dependencies in drupal/legacy-project would be a sane choice.

ressa’s picture

I created #3092140: Set Composer to --no-interaction by default to follow up on this issue in my previous comment.

mmjvb’s picture

@ressa

By the way, would it be possible to somehow set --with-dependencies as default?

No, composer doesn't have a way to set it as default. It does only apply for whitelisted packages though. So, it shouldn't make sense to use it on `composer update` because that isn't whitelisting packages. All packages are considered for update. Note that there is also an option --with-all-dependencies to include those that happen to be required in the root package, those are skipped with --with-dependencies unless whitelisted.
Not aware of a plugin that expands packages to include their dependencies. AFAIK it is not on the agenda for drupal composer initiative. At least not for phase 1, maybe an idea for later phases.

For removing dev requirements, suggest to use:
composer create-project --no-install ....
composer remove --dev ....
composer install

Put the package names on the ..., removing it prevents mistakes as there is no way to make --no-dev the default. Saving you a lot of trouble when omitting the --no-dev by mistake on one of the many commands of composer. Including the confusing option --update-no-dev of require and remove.

Mixologic’s picture

Status: Active » Needs review
StatusFileSize
new993 bytes

Right now, the only place there are --dev expectations are in the -dev versions of the legacy tarball. We can take the extra step to composer require drupal/core-dev-pinned when we build the legacy dev tarballs, and remove the require-dev sections from the templates.

Im sure that the 80/20 rule applies here and that the vast majority of folks are just going to run into friction in upgrading if they have both core-recommended and core-dev in their composer json because even if you use --no-dev when you composer create-project drupal/recommended-project --no-dev, when you go to upgrade drupal/core-recommended, it will *still* attempt to reconcile your packages with the require-dev in your composer.json -> which can be unexpected when trying to do a major version upgrade using composer require (even if you have the aforementioned inconsistently named --update-no-dev).

So Im pretty sure the best thing to do is to ship without the dev dependencies, and instruct people how to add them back if they need them.

As an aside "whitelist" in composer is one of those confusing terms that isn't very well documented - it just means "adding specific package names to the command line"

Let's hopefully get this in before 8.8.0-beta1

greg.1.anderson’s picture

Status: Needs review » Needs work

+1 on #9.

#3087482: Add 'next steps' info to Composer installation output has gone in, so we can update the post-create-project message to advise the user to add the dev dependencies iff they need them.

Mixologic’s picture

We should make sure to credit @ultimike on this issue too, he did some of this at dcon-ams here: https://www.drupal.org/project/drupal/issues/3091579

Mixologic’s picture

Should we add it to the post-create-project message or to the suggest (https://www.drupal.org/project/drupal/issues/3082983) section?

greg.1.anderson’s picture

Hm, yeah, suggests would work.

Mixologic’s picture

Status: Needs work » Needs review

I think we should just remove require-dev first, here, and add in the suggests in the suggests issue, and not block this on having instructions in the codebase, and assume that worst case scenario is that all of the documentation will instruct users how to add the development dependencies of core.

ressa’s picture

Thanks @Mixologic, I am very glad to see the patch. @ultimike should absolutely get some credit (Sidenote: He helped me get started with Migrate module at a sprint back in Barcelona 2015) and even though I am the creator of this issue, I have no experience in managing and giving credit, so I hope someone else can handle that.

I would like to try the patch, but am not sure how to test it, since it has to be implemented before composer create-project is run, if I understand it correctly ... Perhaps it's a complicated process?

Mixologic’s picture

@ressa there are a couple of ways you can test this.

The simplest/messy way is to apply the patch to core, cd to composer/Template/RecommendedProject and execute composer create-project

That will bootstrap the project in place.

You could copy that folder to somewhere else (after you patch it) to make it easier to test as well.

The other way is detailed here: https://www.drupal.org/project/drupal/issues/3082958#comment-13284131

But for this case I would go with the former.

ressa’s picture

Thanks for the help @Mixologic. That wasn't as difficult as I feared, once I figured out I should get the latest version via git, and that the resulting files end up in the composer/Template/RecommendedProject folder.

It works as expected, and the drupal/core-dev projects are excluded by default. I'll add the commands here, in case anyone else wants to try it out, but is not sure how to.

Get Drupal 8.8 (work-in-progress for next release):

git clone --branch 8.8.x https://git.drupalcode.org/project/drupal.git drupal88
cd drupal88

Get patch and patch Drupal core:

wget https://www.drupal.org/files/issues/2019-11-04/3086489_9.patch
patch -p1 < 3086489_9.patch

Check changes:

git status
git diff

Enter Composer template folder, and download Drupal files inside this folder in the new structure:

cd composer/Template/RecommendedProject
composer create-project

The result, where 56 installs signify that drupal/core-dev projects are excluded:

Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 56 installs, 0 updates, 0 removals
  - Installing drupal/core-composer-scaffold (8.8.0-alpha1): Loading from cache
...

To see the difference, run composer create-project from a default, un-patched Drupal 8.8. The developer projects ("109 installs") are included as expected:

Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 109 installs, 0 updates, 0 removals
  - Installing drupal/core-composer-scaffold (8.8.0-alpha1): Loading from cache

The new structure means adding a module can now be done without having to remember to add --update-no-dev, to prevent downloading all drupal/core-dev projects:

composer require drupal/admin_toolbar
Using version ^2.0 for drupal/admin_toolbar
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing drupal/admin_toolbar (2.0.0): Loading from cache

Updating can be done without having to add --no-dev:

composer update --with-dependencies
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
greg.1.anderson’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for testing, @ressa. It should be fine to remove these; it's easy enough to put them back if needed, and it will be very helpful for new users to have an easier update path.

ressa’s picture

Thanks @greg.1.anderson. I tried adding drupal/core-dev to a patched installation with composer require drupal/core-dev, which for some reason gave me a version for 8.7. Still, it adds the project in composer.json correctly, as well as downloads the projects into the vendor folder.

The difference between composer.json's in a current, default Drupal 8.8 and a "patched and afterwards added core-dev":

Current default

...
    "require": {
        "composer/installers": "^1.2",
        "drupal/core-composer-scaffold": "^8.8",
        "drupal/core-project-message": "^8.8",
        "drupal/core-recommended": "^8.8"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8"
    },
...

Patched version, developer tools added with composer require drupal/core-dev

...
    "require": {
        "composer/installers": "^1.2",
        "drupal/core-composer-scaffold": "^8.8",
        "drupal/core-dev": "^8.7",
        "drupal/core-project-message": "^8.8",
        "drupal/core-recommended": "^8.8"
    },
...
Mixologic’s picture

composer require drupal/core-dev, which for some reason gave me a version for 8.7.

Because you have prefer-stable: true, so it finds the most recent stable release, which is 8.7

ressa’s picture

That would explain it, thanks @Mixologic.

greg.1.anderson’s picture

@ressa: For the results you were expecting, try composer require --dev drupal/core-dev:^8.8@dev

ressa’s picture

Thanks @greg.1.anderson, the result of that command is virtually identical code bases:

composer.json

diff drupal88/composer/Template/RecommendedProject/composer.json drupal88_ORIG/composer/Template/RecommendedProject/composer.json
22a23,25
>     "require-dev": {
>         "drupal/core-dev": "^8.8"
>     },
66,68d68
<     },
<     "require-dev": {
<         "drupal/core-dev": "^8.8@dev"

composer.lock

diff drupal88/composer/Template/RecommendedProject/composer.lock drupal88_ORIG/composer/Template/RecommendedProject/composer.lock
7c7
<     "content-hash": "e22bdba131b0b09b997ec53410cf3247",
---
>     "content-hash": "cf2f2dd4f68f72c25ba3a360e5e79a21",
3936,3938d3935
<             "bin": [
<                 "bin/composer"
<             ],
6453,6455c6450
<     "stability-flags": {
<         "drupal/core-dev": 20
<     },
---
>     "stability-flags": [],
Mixologic’s picture

webchick 3 hours ago
This was the thing where the tarballs would not have e.g. phpunit but the git clones would? Off-hand I do think that’s ok, but we might want to survey or something to get wider input.

mixologic:drupalcon: 3 hours ago
@webchick its different than that

mixologic:drupalcon: 3 hours ago
So, this is the 'default instructions in composer'.

mixologic:drupalcon: 3 hours ago
Essentially, composer will install your "development dependencies" unless you specifcally remember to tell it not to

mixologic:drupalcon: 3 hours ago
Which is nbd when we create the tarballs, because it'd be scripted to not do that.

mixologic:drupalcon: 3 hours ago
But in any instructions that we put out there that say "In order to start a new drupal project using composer you do composer create-project drupal/recommended-project --no-dev and whenever you composer update a module or dependency, you'll have to remember to add --no-dev , and whenever you composer require a new module or dependency, you'll also have to remember to add --update-no-dev because composer is somewhat inconsistent.

mixologic:drupalcon: 3 hours ago
Thus, if we dont ship with those require-dev things in the template, nobody will have to do that unless they specifically want to have dev dependencies so that they can run tests and / or develop custom modules etc.

mixologic:drupalcon: 3 hours ago
And getting them back is a mere composer require drupal/core-dev

webchick 2 hours ago
That seems reasonable to me

Mixologic’s picture

Title: Exclude development libraries by default » Exclude development libraries from templates' composer.json by default
mglaman’s picture

That's a negative on my side. I think the default project should and must include the dev dependencies. I test heavily, which means now every project requires extra setup.

Most platforms always deploy with --no-dev. Running `composer install --no-dev` isn't hard. You should only run update/create-project/require locally, anyways. I don't see the benefit here, and it discourages people from writing tests.

---

EDIT: even if not phpunit, what about phpcs?

I would agree the "legacy" project, because dev dependencies have a higher risk of vulnerabilities.

eiriksm’s picture

I feel very strongly about removing it. AT LEAST 80% of the users probably do not have tests, or do not run core tests. And having dependencies you do not use can unnecessary pain when updating.

If you are such an advanced user that you have test you either:

- have your own project template with the dev depa you use
- know enough composer to run the extra command

Let's not burden the users that literally does not know what they have in their composer.json?

nikathone’s picture

nikathone’s picture

Mixologic’s picture

Cursory glance at the 'starting point' templates that other projects use. This is really just to illustrate that this concept is not a drupalism, and really just a choice between what the "out of the box experience" is supposed to be for the end users.

CMS's that do not have require-dev
https://packagist.org/packages/symfony/website-skeleton
https://packagist.org/packages/contao/managed-edition
https://packagist.org/packages/typo3/cms-base-distribution
https://packagist.org/packages/bolt/composer-install

Frameworks tend to come with require-dev's
https://packagist.org/packages/laravel/laravel
https://packagist.org/packages/cakephp/app

karolus’s picture

From the experience gained by talking Drupal and organizing events in the community, my sense would be that most site builds don't use dev tools.

As with any UX initiative, though, it would be good to have data be a guide to the decisions. Would a comparison of total Drupal core downloads to dev packages/tools (such as Devel, et.al) be starting point?

Mixologic’s picture

One strike against doing this that I thought of:

If an end user wants to add the dev deps back, there is now the possibility that they mistakenly `composer require drupal/core-dev` into the normal requirements.. which.. well, that might be bad. vs `composer require-dev drupal/core-dev`

rhovland’s picture

From the perspective of someone newer to Drupal (about a year):

I had a lot of problems using composer. The vast majority of them were things I didn't know about. The first time I went to upgrade core I got a cryptic error message which turned out to be because I needed to specify dev dependencies in the update command, even though my site was installed with --no-dev at the time. I saw numerous other people panicking in Slack because Drupalgeddon 2 was here and composer was refusing to update drupal/core.

While a lot of hosts automatically deploy your site with --no-dev, anyone using shared hosting or rolling their own would have to know to use it. I have encountered a dev shop that does not use --no-dev in deployment.

As the site builder I never once used any of the dev tools.

The developers who I contracted out for module development needed the dev tools. Making them run composer require --dev drupal/devtools in the project everytime they work on something means they need to ignore changes in composer.json and composer.lock at commit time. This would be especially troublesome if they needed to install the module they just wrote.

I think while removing the dev dependencies would help new users we should consider that the change is being made because of how composer functions out of the box.

Instead of making this change which would shift the pain from users to devs what if a composer plugin was created to change the default behavior?

It could change the default behavior of composer so running composer install would default to using --no-dev instead of --dev and if you wanted to install with dev dependencies you would use composer install --dev. We would make the same changes to composer update and maybe add --with-dependencies as a default (not sure if there's an instance where you don't want to run that). This also has the benefit that if someone doesn't like this change in behavior they can just remove the plugin from composer.json.

mglaman’s picture

#33: Interesting. So see if a plugin can layer in DX/UX on the flags passed to composer commands, to normalize what people expect in Drupal?

greg.1.anderson’s picture

I am reluctant to make Drupal's default Composer behavior contradict the expected default Composer behavior. I would be +1 on reversing the default for dev / no-dev in Composer itself, but it seems unlikely to me that this alteration would be accepted prior to Composer 2.

nikathone’s picture

StatusFileSize
new184.03 KB

That would be really great but not sure if composer can let a plugin do that. At the moment if you start with something like
composer create-project drupal-composer/drupal-project:8.x-dev my-project --no-interaction --no-dev && cd my-project
Notice the --no-dev flag. Then let say you want to downgrade drupal to 8.7.10 using composer require drupal/core:8.7.9 composer will return an error like

Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - webflo/drupal-core-require-dev 8.7.10 requires drupal/core 8.7.10 -> satisfiable by drupal/core[8.7.10] but these conflict with your requirements or minimum-stability.
    - webflo/drupal-core-require-dev 8.7.10 requires drupal/core 8.7.10 -> satisfiable by drupal/core[8.7.10] but these conflict with your requirements or minimum-stability.
    - Installation request for webflo/drupal-core-require-dev (locked at 8.7.10, required as ^8.7.0) -> satisfiable by webflo/drupal-core-require-dev[8.7.10].

even if you excluded the dev packages from being installed.

There is also the -update-no-dev flag you can add to composer require which actually provide a better explanation of why downgrading failed:

Running update with --no-dev does not mean require-dev is ignored, it just means the packages will not be installed. If dev requirements are blocking the update you have to resolve those problems.

Based on this I guess as long as dev dependencies are in the mix people will always have to worry about them...
composer require fail screenshot

mile23’s picture

Excluding dev dependencies is an OK solution.

The problem comes when people might accidentally add it to their require instead of require-dev.

The real solution lies in your workflow and not in what core provides. I can imagine the first thing a lead dev does is modify the composer.json file to have the proper dev stuff all sorted away in the proper place. Or make their own project template package for consistency's sake. Without the dev dependencies, there can't be testing, for instance. At least not using core's testing stuff.

This is definitely something we'd need to document thoroughly. Adding ND and NCR so there's a trail in commit history.

It would be great if there were a Composer way to specify that a package should only ever be dev. I don't see an upstream issue about that.

mile23’s picture

Status: Reviewed & tested by the community » Needs work
rhovland’s picture

So if we don't want to change default composer behavior what about allowing projects to add a section to their composer.json to hold the user's hand?

Create a plugin that adds an optional section to composer.json where a project can specify a list of packages that should always be whitelisted when updating it.

When a user runs composer update drupal/core or composer some-custom-script-name it also updates the dev dependencies because they are part of the default whitelist defined in core's composer.json.

This would also be very useful for Drupal Commerce as well since there's a huge number of packages you have to whitelist to get it to update without complaining.

I know, I'm not well versed in what the plugin system is capable of but I feel like the real problem lies in how composer operates and not in the contents of composer.json. Most PHP projects I've seen using composer only use it for dev work, and they create static packages for their end users. If drupal wants their regular users to use composer then I think it makes sense to add scripting or other affordances that simplifies some processes while still allowing developers to use it as designed.

mglaman’s picture

greg.1.anderson’s picture

Without the dev dependencies, there can't be testing, for instance. At least not using core's testing stuff.

A lot of folks have commented on this issue talking about the need for require-dev for testing; webflo/drupal-core-require-dev is part of drupal-composer/drupal-project for the same reason. However, I was looking around for documentation on how to test a Drupal site using the Drupal dev dependencies, and I couldn't find anything. The documentation on drupal.org focuses on Drupal core and modules. Blogs on testing with Behat or Drupal Test Traits do not seem to use Drupal's dev dependencies, but instead either provide their own (Drupal Test Traits) or expect the user to require-dev what is needed.

The documentation I could find on dev dependencies simply made reference to running the Drupal tests. However, the instructions for the Drupal tests themselves explicitly say that you should start with a `git clone` of drupal/drupal, not a Composer-managed site. The tests in drupal-composer/drupal-project that call through to the Drupal tests must exclude a large number of tests, and the drupal-composer/drupal-project tests themselves have been broken for a long time because Drupal tests that used to work stop working over time, because using them like this is an unsupported configuration.

I wanted to put some examples in the change record. If someone has any examples of documentation that recommends using these dependencies for testing a site, please provide them and I'll include it when I write it up. If, on the other hand, there really isn't a good use-case for when you would use these, then I don't see much point in documenting how to add them back into a project, or worrying about whether they might accidentally end up in `require` rather than `require-dev`.

mile23’s picture

Well that's what I'm saying: There's no documentation because we haven't made the change yet. :-) We also don't really have a plan for how to support BDD/DTT against existing sites that were built from our project templates. It seems like that's what we're making here.

mglaman’s picture

I'll add notes. I copy cores phpunit.xml and change paths, then our projects have unit and kernel tests. Even Functional JavaScript tests.

I don't use Behat anymore

greg.1.anderson’s picture

I wrote up a brief change record.

@mglaman @ultimike: If you could add any notes you may be able to provide to the placeholder documentation page, Using Drupal's test classes to write tests for a Composer-managed site, it would be appreciated.

mglaman’s picture

@greg.1.anderson yes! I can add the information there. I just want to say that I have "come to the light" and see how this simplifies life for most users. It just does add a mess once you add dev dependencies and then try to do things with --no-dev beyond composer install. That's where I was confused (never did composer require --no-dev, update --no-dev, etc.)

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new1.65 KB
new2.12 KB

Now we have a change record and enough documentation to instruct users how to add the dev dependencies back in. The documentation is linked from the change record, and, with the updated patch attached to this comment, also from the post-install message.

While the documentation still needs to be finished, I think we might have enough here to get started, so I'm putting this back to 'needs review'.

ressa’s picture

I just want to say that I have "come to the light" and see how this simplifies life for most users.

Thanks for sharing that @mglaman. Your comment in #26 ("That's a negative on my side.") gave me pause, so your change of mind, as well as the opinion from @eiriksm in #27 ("I feel very strongly about removing it.") is really valuable and weighty opinions, coming from seasoned developers.

The common site builder or devops person probably doesn't visit these Composer issues to suggest improvements, but are more likely to go to the forums, or just move on after having experienced yet another cryptic message from Composer.

Mixologic’s picture

Status: Needs review » Reviewed & tested by the community

The docs and CR both look good to me.

Should something like this go into a release notes snippet?

greg.1.anderson’s picture

Issue summary: View changes
Issue tags: -Needs documentation, -Needs change record

I don't know if we really need a release note snippet, since the template projects have never been stable, so in theory only a few people are using them. I added one here in case the release manager wants to include one anyway.

Mixologic’s picture

Yeah, I think its more of a 'tarball was always composer broken, so it didnt matter', but the 'defacto' community template _did_ come with them. So its sort of a change to standard procedure here.

naheemsays’s picture

Should the priority on this issue be moved to Major?

Getting this change in before 8.8.0 is IMO very important as then the workflow of less experienced users will not need to be changed again after this issue has landed. It will also let less experiences users to install drupal without needing to know about having to exclude dev dependencies.

alexpott’s picture

I’m +1 to this patch. I think on balance it helps people make the right choices first up. It is a hard choice. But I think I prefer learning that I can get development tools rather than learning I have to remove them.

mile23’s picture

Version: 8.9.x-dev » 8.8.x-dev

Any chance of getting it in 8.8.x?

Also, we'll need to update the Project Templates documentation to tell people how to add the dev stuff back in.

xjm’s picture

We are already in commit freeze for 8.8.0, so without having the reviewed the issue yet, I would say it's not eligible. I'll take a closer look in a few hours.

greg.1.anderson’s picture

There are no actual code changes here, so I hope we can make an eleventh-hour exception. Please chat with @webchick and @alexpott if possible.

If this does not go in 8.8.0, it probably should not go in 8.9.x either.

xjm’s picture

I might prefer to revert this from 8.9.x (edit: not commit to 8.9.x for now?) than cram it into 8.8.x at the last minute. Unfortunately, "this isn't disruptive" as a statement kind of has an error bar, historically speaking. :) I'll wait to see if @catch has thoughts though.

xjm’s picture

Like, also, we don't even commit documentation spelling fixes during a commit freeze. Much less things that change processes or best practices.

xjm’s picture

(Sorry for the multiple comments in a row; just sort of sorting through this in my head.)

I'm not against this issue in principle. In a way I actually see it as a security hardening. I'm just concerned about the timing of the commit here.

greg.1.anderson’s picture

Yeah it certainly would have been better if this issue had progressed a little faster. Having never been closely involved with the last moments of a Drupal release before, I am unfamiliar with how frozen "frozen" is. I certainly understand if it's too late. From a documentation perspective, it would be less confusing if the dev dependency handling was consistent across the Drupal versions, but we can adapt to whatever decision is made here.

daffie’s picture

Issue summary: View changes

-1 I appreciate the effort to make it easier for site builders to use composer, only we will now add a drupalism. Composer adds development stuff by default and Drupal will do the opposite. We want to attract other PHP developers to Drupal and this will not help. My suggestion will be to go to the composer project and make the change there.

alexpott’s picture

@daffie see #30 - this is not a Drupalism.

daffie’s picture

Issue summary: View changes

@alexpott: I guess that you are right. It just does not feel this way for me.

xjm’s picture

Version: 8.8.x-dev » 8.9.x-dev
Issue summary: View changes

Discussed with @larowlan and @catch. After discussing with them, I'm going to go ahead and say this issue is an issue for 8.9.x.

Based on @greg.1.anderson's earlier comment, I wanted to document here on the issue what the release phases are and the expectations for them, since we already did a lot of 11th-hour Composer Initiative work this minor, and we shouldn't have the same expectations in the future.

Issues are considered disruptive when they require changes to documentation, best practices, site owners' processes, core developers' processes, release managers' processes -- including from a previous beta or release candidate. Typically, we want all of these things to be the same for beta1 as they are for the minor release, and we would require them to be complete before the beta deadline. We made some exceptions around the Composer Initiative this quarter, because it was new in 8.8, because there were regressions and concerns that we were getting stuck with the wrong architecture for the minor release. For this issue and future issues, I think it's important to keep in mind that we're trying to complete such changes before alpha1, and at committer discretion, before beta1, if the issue is a strategic priority.

The purpose of the beta phase is to stabilize stuff we got badly wrong in beta1 that's causing regressions. You can see the list of allowed changes in the beta phase here. We might have considered this issue during the beta phase with a release note about it in the rc1 release notes, but we might have also required it to be 8.9.x even then.

The purpose of the release candidate phase is to allow widespread testing and fix any critical regressions that are uncovered. Release candidates have the same patch requirements as patch releases, which means we would not backport anything that was not (e.g.) a straightforward bugfix invisible to the site owner beyond that the bug went away. Patch releases can't include any disruptive changes, even internal API changes or bugfixes with disruptive changes, and we have the same expectation for release candidates. The things we commit at committer discretion here might typically be "a critical that makes a disruptive change but fixes a data loss or upgrade path issue". Not a DX improvement, even if it also has security hardening implications.

Finally, I'll describe how we run release windows and the expectations for the commit freeze. For most release windows, the commit freeze starts at 1200 UTC Tuesday and ends at 1200 UTC Thursday. This is to ensure nothing introduces a surprise before the release on Wednesday, and nothing introduces a change that might conflict with a hotfix after it goes out.

For alpha, beta, and occasionally RC windows, the scheduled release window is the whole week. The whole committer team discusses any issue being considered for backport during the week-long window. We flexibly start our 48 hours of commit freeze based on when strategic priorities are done -- like waiting to release beta1 until we'd deprecated SimpleTest properly, or waiting to release rc1 until a release manager had a chance to test another Comopser Initiative change that should ideally have been done before the beta but that had important implications for getting the new feature right. That was an exception based on evaluating impact vs. disruption, and it was still followed by a 48h commit freeze.

For minor releases, the commit freeze on the branch that's going to be released lasts all week. The scheduled release is on a single day. The only things we'd likely commit during this freeze would be fixes for data integrity, upgrade path, or security criticals, especially if they were regressions from the previous releases. Unlike alphas, betas, and RCs, where we can release them on any day of the week, minor releases need to be released on the scheduled date, to maintain confidence in the platform.

I've made two assumptions in evaluating this, and please correct me if I'm wrong about them:

  1. This does not change or affect what get packaged on Drupal.org. Dev dependencies will still not be included in tagged releases. They will still be included in dev tarballs (something which Mixologic and I agree merits revisiting, but is out of scope here).

  2. There is no regression for Composer-built sites already out there. As far as I know, all of our documentation instructs Composer site owners to always use --no-dev when they're deploying to production or any open environment. The docs have been that way since SA-CORE-2017-001 and the packaging change before that. If the new Composer initiative templates require --no-dev, that's parity with what we already have, and no regression from 8.7 in terms of whether dev dependencies are included when people follow our instructions.

If either of those statements are false, then there's a critical security regression with the 8.8 Composer Initiative feature-set that needs to be addressed. Otherwise, though, this is a DX improvement and a form of security hardening that is eligible for a minor release during its development phase prior to alpha1, which is 8.9.x right now. If the documentation has to explain that 8.9 or 9.1 improved this over 8.8, that's fine, especially if typing --no-dev either silently does the same thing as what's already happening or even provides a message that with that version of Drupal, it's the default behavior.

Depending on the disruptions (which we should summarize in the IS), this might need a release note to explain the differences over 8.8.x. (I've tweaked the release note we already have to be a little more clear; please add any other disruptions there.) But it would probably be an allowed change as a security hardening, unless I'm missing some deeply disruptive impact besides just documentation that has one "On 8.8.x" header and another "On 8.9.x or higher" header.

Hope this helps explain the context and set expectations for the future.

xjm’s picture

The relevant handbook documentation that covers all this policy:

xjm’s picture

xjm’s picture

I've also updated the section on "What is a disruptive change?" to cover the scenarios that have come up for this initiative:
https://www.drupal.org/core/d8-allowed-changes#disruption

mile23’s picture

Issue tags: +needs documentation updates

Based on #63 this issue is either 'closed (works as designed),' because the user can say composer create-project --no-dev, or still RTBC for 8.9 with a documentation update saying that if you want dev, you can add the appropriate package.

Adding 'needs documentation updates' because either way we have to tell the user how to, or how not to have dev packages: https://www.drupal.org/docs/develop/using-composer/starting-a-site-using...

greg.1.anderson’s picture

Version: 8.9.x-dev » 9.0.x-dev
Priority: Normal » Minor
Issue summary: View changes
Issue tags: -needs documentation updates +Needs documentation updates

So, we actually ended up making a change in Drupal Infra. When we split out the template project, we remove the dev dependencies iff the subtree split is a tag. If the split is being done for a dev branch, the dev dependencies are left in.

We could decide whether to change this for 9.x. We should still update the documentation. I updated the issue summary.

greg.1.anderson’s picture

Status: Reviewed & tested by the community » Needs review

Back to "Needs review" for discussion / decision on 9.x policy.

mglaman’s picture

So, did this end up getting committed? I did a fresh drupal/recommended-project build and there were no dev dependencies, I had to run composer require --dev drupal/core-dev-pinned. So I thought this was closed.. but it's not.

greg.1.anderson’s picture

@mglaman See #68. The Drupal Infrastructure scripts remove the dev dependencies as part of the subtree split operations.

If you want to add the dev dependencies back in, it's probably better to use the non-pinned variant, drupal/core-dev. Adding pinned dependencies will make your site harder to update.

mmjvb’s picture

You might want to extend it to settings 'minimum-stability' and 'prefer-stable'. These settings don't belong in non-dev projects.

greg.1.anderson’s picture

#72 is out of scope for this issue. If we want to discuss minimum-stability further, someone should open a follow-on issue.

Version: 9.0.x-dev » 9.1.x-dev

Drupal 9.0.10 was released on December 3, 2020 and is the final full bugfix release for the Drupal 9.0.x series. Drupal 9.0.x will not receive any further development aside from security fixes. Sites should update to Drupal 9.1.0 to continue receiving regular bugfixes.

Drupal-9-only bug reports should be targeted for the 9.1.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.2.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.1.x-dev » 9.3.x-dev

Drupal 9.1.10 (June 4, 2021) and Drupal 9.2.10 (November 24, 2021) were the last bugfix releases of those minor version series. Drupal 9 bug reports should be targeted for the 9.3.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new144 bytes

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

mile23’s picture

Version: 9.5.x-dev » 10.1.x-dev
Issue tags: +Needs reroll

The current behavior is that for dev branches, the project templates include drupal/core-dev as a require-dev. For tagged releases, drupal/core-dev is excluded.

Release:

% composer create-project drupal/recommended-project:@stable
Creating a "drupal/recommended-project:@stable" project at "./recommended-project"
Installing drupal/recommended-project (10.0.2)
  - Installing drupal/recommended-project (10.0.2): Extracting archive

[...]

% cd recommended-project 
% composer show "drupal/core-*"
drupal/core-composer-scaffold 10.0.2 A flexible Composer project scaffold b...
drupal/core-project-message   10.0.2 Adds a message after Composer installa...
drupal/core-recommended       10.0.2 Core and its dependencies with known-c...

Dev:

% composer create-project drupal/recommended-project:@dev 
Creating a "drupal/recommended-project:@dev" project at "./recommended-project"
Installing drupal/recommended-project (10.1.x-dev f2de0da24e4fdae1ae8b81d2f4f3fbd038809017)
  - Downloading drupal/recommended-project (10.1.x-dev f2de0da)
  - Installing drupal/recommended-project (10.1.x-dev f2de0da): Extracting archive

[...]

% cd recommended-project 
% composer show "drupal/core-*"
drupal/core-composer-scaffold 10.1.x-dev 14db477 A flexible Composer projec...
drupal/core-dev               10.1.x-dev a6812ad require-dev dependencies f...
drupal/core-project-message   10.1.x-dev 59b4475 Adds a message after Compo...
drupal/core-recommended       10.1.x-dev 3981ea2 Core and its dependencies ...

So in the meantime we split the difference: Releases don't risk including dev requirements by default, while dev setups include the dev tools.

The question then becomes: Should we exclude the dev tools from the dev project, and it's either +1 for consistency, or -1 for not letting users decide to take fate into their own hands by using @dev without having to type composer require drupal/core-dev

I think I'm +1 for consistency, where we just never add drupal/core-dev to the project templates, and we tell users to add the package they need. Obviously real actual maintainers might have a different view. :-)

Since we have a CR, setting the issue to target Drupal core 10.1.x.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.