Problem/Motivation
The Drupal template projects (drupal/recommended-project and drupal/legacy-project) include dev dependencies in their composer.json file in the repository. Drupal infrastructure automatically removes these when running the subtree split for any tag (e.g. 8.9.0-alpha1), but allows them to remain for branches (e.g. 8.9.x-dev).
Proposed resolution
This behavior may be good enough. We could also consider removing the dev dependencies from the repository for 9.x.
Remaining tasks
We should update the documentation to clearly describe the situation.
User interface changes
None.
API changes
None.
Data model changes
None.
Old Summary
On the production server, and as a site builder, you shouldn't need the development libraries. So I suggest that development libraries are not included by default in Drupal core. Also, in that way the --no-dev and --update-no-dev parameters are no longer necessary for everyday-site-building.
How to create and manage a project without development libraries in the new version 8.8, where -n equals --no-interaction:
$ composer create-project drupal/recommended-project:^8.8@dev -n --no-dev mysite
$ composer update --with-dependencies --no-dev
$ composer require --update-no-dev drupal/admin_toolbar
On one hand, adding the developer tools project with for example a command like composer require drupal/devtools will be an extra step for Drupal developers, who are deep in the code. On the other hand, the benefits for those who mostly don't need these tools, like new Drupal users, site builders and dev ops people, there will be benefits. For example for dev ops people, the reduced complexity will mean less risk of unforeseen Composer events during a security update situation, like a lingering git commit blocking everything, which can grind the process down to a halt.
If the composer update --with-dependencies process breaks down, while applying late night security updates, having to debug why it happens, while under the the pressure of rolling out updates is not optimal, and might make dev ops people a little less fond of Drupal in the long run. Running composer update should be as boring a non-event as possible, and the chances of a smooth process should be maximized, by getting potential obstacles out of the way.
While there "shouldn't" be a problem, stuff happens, like with .git repository should not be included with composer package, where rfay comments:
This seems to break every composer update I do, have to remove the coder directory in vendor to fix.
Changing to a default installation, without dev dependencies
Pros
- For new Drupal users there will be less moving parts and reduced complexity
- No need to remember to use
--no-devand--update-no-dev - You don't get a screen load of updates, and changed
composer.jsonif you forget to add--update-no-dev - Composer can calculate dependencies faster during
composer update - Less risk of unforeseen Composer events, like a lingering git commit blocking
composer update - Less changes in
composer.lock, giving less results runninggit diffand better overview
Cons
- The work involved in containing the developer tools in its own
drupal/devtoolsproject - Developers have to run
composer require drupal/devtools
Statistics
Result with --no-dev, 53 packages
$ composer create-project drupal/recommended-project:^8.8@dev -n --no-dev mysite
Installing drupal/recommended-project (8.8.x-dev 995d694d3cf9c0863bb170e7c62c8f1aa6d35437)
- Installing drupal/recommended-project (8.8.x-dev 995d694): Cloning 995d694d3c from cache
Created project in mysite
Loading composer repositories with package information
Updating dependencies
Package operations: 53 installs, 0 updates, 0 removals
- Installing composer/installers (v1.6.0): Loading from cache
...Stats: 24.534 items (and 91 hidden), totalling 391,2 MB
Result without --no-dev, 106 packages
$ composer create-project drupal/recommended-project:^8.8@dev -n mysiteDEV
Installing drupal/recommended-project (8.8.x-dev 995d694d3cf9c0863bb170e7c62c8f1aa6d35437)
- Installing drupal/recommended-project (8.8.x-dev 995d694): Cloning 995d694d3c from cache
Created project in mysiteDEV
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 106 installs, 0 updates, 0 removals
- Installing composer/installers (v1.6.0): Loading from cache
...Stats: 29.914 items (and 208 hidden), totalling 414,8 MB
Release notes snippet
Development dependencies have been removed from the Composer Template Projects. See the change record for information on how to add development dependencies. It is recommended to only install development dependencies in development environments.
| Comment | File | Size | Author |
|---|---|---|---|
| #78 | 3086489-nr-bot.txt | 144 bytes | needs-review-queue-bot |
| #46 | 3086489-46.patch | 2.12 KB | greg.1.anderson |
| #46 | 9-to-46.interdiff.txt | 1.65 KB | greg.1.anderson |
| #9 | 3086489_9.patch | 993 bytes | Mixologic |
Comments
Comment #2
greg.1.anderson commentedThere has been some talk of not including dev dependencies by default in the project templates, since running the Drupal tests is only supposed to be done from a git clone. However, we might need to wait until Drupal 9 to make this change, if we decide that we want to do it.
If dev dependencies are included by default, then those who do not wish to include them may run composer with the
--no-devflag. If dev dependencies are not included by default, then those who want them will need to know which project to require to get them. This is one reason why there has been some reluctance to make this change.Comment #4
ressaThanks for outlining the situation @greg.1.anderson. I understand there would probably be quite a bit of work involved in changing to a default installation without dev dependencies, so that user have to actively add developer tools, with a command like
composer require drupal/devtools. I have updated the Issue Summary, with a bit more description, as well as a list of Pros and Cons.Comment #5
greg.1.anderson commented@mixologic and I were just talking about this issue. We are renaming
drupal/dev-dependenciestodrupal/core-devfor another purpose (so thatcomposer update drupal/core*would get bothcore-recommendedandcore-devat the same time; c.f. #3090684: Rename drupal/dev-dependencies and drupal/pinned-dev-dependencies).It seems like it might not be too burdensome to ask a user to run
composer require drupal/core-devif the dev dependencies are needed. Perhaps we could even utilize #3087482: Add 'next steps' info to Composer installation output to socialize this.We also have to decide if we're going to make this change to both drupal/recommended-project and drupal/legacy-project, or perhaps keep the dev dependencies in legacy (sacrificing consistency for maintaining expectations for legacy).
Comment #6
ressaThat sounds really great, thank you and @mixologic for bearing this issue in mind, I am glad to hear that it might not be as hard to do as initially feared.
Hearing Dries' talk in his Driesnote today, about lowering the barrier of entry as much as possible to make it easier for first time users to enter Drupal, excluding development libraries by default seems to me the logical choice, over having (mostly) developers who need the
drupal/core-devtools run a single command.I agree that mentioning how to get the
drupal/core-devtools is an obvious candidate for inclusion on the #3087482: Add 'next steps' info to Composer installation output page.Just to make sure, should development libraries eventually be excluded by default in Drupal core, the commands below will download only what the added parameter
--no-devdownloads today (around 53 packages), and that afterwards, there will be no need to add--no-devand--update-no-devparameters to keep Drupal core lean, and free ofdrupal/core-devtools, right?By the way, would it be possible to somehow set
--with-dependenciesas default?I feel like the less clutter, the better. But of course, if it is a necessary Composer parameter, I understand if it has to stay.
It seems like the
-n(--no-interaction) can be set in composer.json, with the COMPOSER_NO_INTERACTION parameter. Worth considering? If this parameter was added, it could be mentioned on the #3087482: Add 'next steps' info to Composer installation output page for expert users, that it can be disabled in composer.json, if desired.Regarding
drupal/recommended-projectanddrupal/legacy-project, I agree with you that excluding development libraries by default is most important fordrupal/recommended-project, while keeping the dev dependencies indrupal/legacy-projectwould be a sane choice.Comment #7
ressaI created #3092140: Set Composer to --no-interaction by default to follow up on this issue in my previous comment.
Comment #8
mmjvb commented@ressa
No, composer doesn't have a way to set it as default. It does only apply for whitelisted packages though. So, it shouldn't make sense to use it on `composer update` because that isn't whitelisting packages. All packages are considered for update. Note that there is also an option --with-all-dependencies to include those that happen to be required in the root package, those are skipped with --with-dependencies unless whitelisted.
Not aware of a plugin that expands packages to include their dependencies. AFAIK it is not on the agenda for drupal composer initiative. At least not for phase 1, maybe an idea for later phases.
For removing dev requirements, suggest to use:
composer create-project --no-install ....
composer remove --dev ....
composer install
Put the package names on the ..., removing it prevents mistakes as there is no way to make --no-dev the default. Saving you a lot of trouble when omitting the --no-dev by mistake on one of the many commands of composer. Including the confusing option --update-no-dev of require and remove.
Comment #9
MixologicRight now, the only place there are --dev expectations are in the -dev versions of the legacy tarball. We can take the extra step to
composer require drupal/core-dev-pinnedwhen we build the legacy dev tarballs, and remove the require-dev sections from the templates.Im sure that the 80/20 rule applies here and that the vast majority of folks are just going to run into friction in upgrading if they have both core-recommended and core-dev in their composer json because even if you use --no-dev when you
composer create-project drupal/recommended-project --no-dev, when you go to upgradedrupal/core-recommended, it will *still* attempt to reconcile your packages with the require-dev in your composer.json -> which can be unexpected when trying to do a major version upgrade using composer require (even if you have the aforementioned inconsistently named --update-no-dev).So Im pretty sure the best thing to do is to ship without the dev dependencies, and instruct people how to add them back if they need them.
As an aside "whitelist" in composer is one of those confusing terms that isn't very well documented - it just means "adding specific package names to the command line"
Let's hopefully get this in before 8.8.0-beta1
Comment #10
greg.1.anderson commented+1 on #9.
#3087482: Add 'next steps' info to Composer installation output has gone in, so we can update the post-create-project message to advise the user to add the dev dependencies iff they need them.
Comment #11
MixologicWe should make sure to credit @ultimike on this issue too, he did some of this at dcon-ams here: https://www.drupal.org/project/drupal/issues/3091579
Comment #12
MixologicShould we add it to the post-create-project message or to the suggest (https://www.drupal.org/project/drupal/issues/3082983) section?
Comment #13
greg.1.anderson commentedHm, yeah, suggests would work.
Comment #14
MixologicI think we should just remove require-dev first, here, and add in the suggests in the suggests issue, and not block this on having instructions in the codebase, and assume that worst case scenario is that all of the documentation will instruct users how to add the development dependencies of core.
Comment #15
ressaThanks @Mixologic, I am very glad to see the patch. @ultimike should absolutely get some credit (Sidenote: He helped me get started with Migrate module at a sprint back in Barcelona 2015) and even though I am the creator of this issue, I have no experience in managing and giving credit, so I hope someone else can handle that.
I would like to try the patch, but am not sure how to test it, since it has to be implemented before
composer create-projectis run, if I understand it correctly ... Perhaps it's a complicated process?Comment #16
Mixologic@ressa there are a couple of ways you can test this.
The simplest/messy way is to apply the patch to core, cd to
composer/Template/RecommendedProjectand executecomposer create-projectThat will bootstrap the project in place.
You could copy that folder to somewhere else (after you patch it) to make it easier to test as well.
The other way is detailed here: https://www.drupal.org/project/drupal/issues/3082958#comment-13284131
But for this case I would go with the former.
Comment #17
ressaThanks for the help @Mixologic. That wasn't as difficult as I feared, once I figured out I should get the latest version via git, and that the resulting files end up in the
composer/Template/RecommendedProjectfolder.It works as expected, and the
drupal/core-devprojects are excluded by default. I'll add the commands here, in case anyone else wants to try it out, but is not sure how to.Get Drupal 8.8 (work-in-progress for next release):
Get patch and patch Drupal core:
Check changes:
Enter Composer template folder, and download Drupal files inside this folder in the new structure:
The result, where
56 installssignify thatdrupal/core-devprojects are excluded:To see the difference, run
composer create-projectfrom a default, un-patched Drupal 8.8. The developer projects ("109 installs") are included as expected:The new structure means adding a module can now be done without having to remember to add
--update-no-dev, to prevent downloading alldrupal/core-devprojects:Updating can be done without having to add
--no-dev:Comment #18
greg.1.anderson commentedThanks for testing, @ressa. It should be fine to remove these; it's easy enough to put them back if needed, and it will be very helpful for new users to have an easier update path.
Comment #19
ressaThanks @greg.1.anderson. I tried adding
drupal/core-devto a patched installation withcomposer require drupal/core-dev, which for some reason gave me a version for 8.7. Still, it adds the project incomposer.jsoncorrectly, as well as downloads the projects into the vendor folder.The difference between
composer.json's in a current, default Drupal 8.8 and a "patched and afterwards added core-dev":Current default
Patched version, developer tools added with
composer require drupal/core-devComment #20
MixologicBecause you have
prefer-stable: true, so it finds the most recent stable release, which is 8.7Comment #21
ressaThat would explain it, thanks @Mixologic.
Comment #22
greg.1.anderson commented@ressa: For the results you were expecting, try
composer require --dev drupal/core-dev:^8.8@devComment #23
ressaThanks @greg.1.anderson, the result of that command is virtually identical code bases:
composer.json
composer.lock
Comment #24
Mixologicwebchick 3 hours ago
This was the thing where the tarballs would not have e.g. phpunit but the git clones would? Off-hand I do think that’s ok, but we might want to survey or something to get wider input.
mixologic:drupalcon: 3 hours ago
@webchick its different than that
mixologic:drupalcon: 3 hours ago
So, this is the 'default instructions in composer'.
mixologic:drupalcon: 3 hours ago
Essentially, composer will install your "development dependencies" unless you specifcally remember to tell it not to
mixologic:drupalcon: 3 hours ago
Which is nbd when we create the tarballs, because it'd be scripted to not do that.
mixologic:drupalcon: 3 hours ago
But in any instructions that we put out there that say "In order to start a new drupal project using composer you do composer create-project drupal/recommended-project --no-dev and whenever you composer update a module or dependency, you'll have to remember to add --no-dev , and whenever you composer require a new module or dependency, you'll also have to remember to add --update-no-dev because composer is somewhat inconsistent.
mixologic:drupalcon: 3 hours ago
Thus, if we dont ship with those require-dev things in the template, nobody will have to do that unless they specifically want to have dev dependencies so that they can run tests and / or develop custom modules etc.
mixologic:drupalcon: 3 hours ago
And getting them back is a mere composer require drupal/core-dev
webchick 2 hours ago
That seems reasonable to me
Comment #25
MixologicComment #26
mglamanThat's a negative on my side. I think the default project should and must include the dev dependencies. I test heavily, which means now every project requires extra setup.
Most platforms always deploy with --no-dev. Running `composer install --no-dev` isn't hard. You should only run update/create-project/require locally, anyways. I don't see the benefit here, and it discourages people from writing tests.
---
EDIT: even if not phpunit, what about phpcs?
I would agree the "legacy" project, because dev dependencies have a higher risk of vulnerabilities.
Comment #27
eiriksmI feel very strongly about removing it. AT LEAST 80% of the users probably do not have tests, or do not run core tests. And having dependencies you do not use can unnecessary pain when updating.
If you are such an advanced user that you have test you either:
- have your own project template with the dev depa you use
- know enough composer to run the extra command
Let's not burden the users that literally does not know what they have in their composer.json?
Comment #28
nikathoneDoing this might simplify step 2 at https://www.drupal.org/docs/8/upgrade/upgrading-between-drupal-8-major-v...
Comment #29
nikathoneComment #30
MixologicCursory glance at the 'starting point' templates that other projects use. This is really just to illustrate that this concept is not a drupalism, and really just a choice between what the "out of the box experience" is supposed to be for the end users.
CMS's that do not have require-dev
https://packagist.org/packages/symfony/website-skeleton
https://packagist.org/packages/contao/managed-edition
https://packagist.org/packages/typo3/cms-base-distribution
https://packagist.org/packages/bolt/composer-install
Frameworks tend to come with require-dev's
https://packagist.org/packages/laravel/laravel
https://packagist.org/packages/cakephp/app
Comment #31
karolus commentedFrom the experience gained by talking Drupal and organizing events in the community, my sense would be that most site builds don't use dev tools.
As with any UX initiative, though, it would be good to have data be a guide to the decisions. Would a comparison of total Drupal core downloads to dev packages/tools (such as Devel, et.al) be starting point?
Comment #32
MixologicOne strike against doing this that I thought of:
If an end user wants to add the dev deps back, there is now the possibility that they mistakenly `composer require drupal/core-dev` into the normal requirements.. which.. well, that might be bad. vs `composer require-dev drupal/core-dev`
Comment #33
rhovlandFrom the perspective of someone newer to Drupal (about a year):
I had a lot of problems using composer. The vast majority of them were things I didn't know about. The first time I went to upgrade core I got a cryptic error message which turned out to be because I needed to specify dev dependencies in the update command, even though my site was installed with --no-dev at the time. I saw numerous other people panicking in Slack because Drupalgeddon 2 was here and composer was refusing to update drupal/core.
While a lot of hosts automatically deploy your site with --no-dev, anyone using shared hosting or rolling their own would have to know to use it. I have encountered a dev shop that does not use --no-dev in deployment.
As the site builder I never once used any of the dev tools.
The developers who I contracted out for module development needed the dev tools. Making them run
composer require --dev drupal/devtoolsin the project everytime they work on something means they need to ignore changes in composer.json and composer.lock at commit time. This would be especially troublesome if they needed to install the module they just wrote.I think while removing the dev dependencies would help new users we should consider that the change is being made because of how composer functions out of the box.
Instead of making this change which would shift the pain from users to devs what if a composer plugin was created to change the default behavior?
It could change the default behavior of composer so running
composer installwould default to using--no-devinstead of--devand if you wanted to install with dev dependencies you would usecomposer install --dev. We would make the same changes tocomposer updateand maybe add--with-dependenciesas a default (not sure if there's an instance where you don't want to run that). This also has the benefit that if someone doesn't like this change in behavior they can just remove the plugin from composer.json.Comment #34
mglaman#33: Interesting. So see if a plugin can layer in DX/UX on the flags passed to composer commands, to normalize what people expect in Drupal?
Comment #35
greg.1.anderson commentedI am reluctant to make Drupal's default Composer behavior contradict the expected default Composer behavior. I would be +1 on reversing the default for dev / no-dev in Composer itself, but it seems unlikely to me that this alteration would be accepted prior to Composer 2.
Comment #36
nikathoneThat would be really great but not sure if composer can let a plugin do that. At the moment if you start with something like
composer create-project drupal-composer/drupal-project:8.x-dev my-project --no-interaction --no-dev && cd my-projectNotice the
--no-devflag. Then let say you want to downgrade drupal to 8.7.10 usingcomposer require drupal/core:8.7.9composer will return an error likeeven if you excluded the dev packages from being installed.
There is also the
-update-no-devflag you can add to composer require which actually provide a better explanation of why downgrading failed:Based on this I guess as long as dev dependencies are in the mix people will always have to worry about them...

Comment #37
mile23Excluding dev dependencies is an OK solution.
The problem comes when people might accidentally add it to their require instead of require-dev.
The real solution lies in your workflow and not in what core provides. I can imagine the first thing a lead dev does is modify the composer.json file to have the proper dev stuff all sorted away in the proper place. Or make their own project template package for consistency's sake. Without the dev dependencies, there can't be testing, for instance. At least not using core's testing stuff.
This is definitely something we'd need to document thoroughly. Adding ND and NCR so there's a trail in commit history.
It would be great if there were a Composer way to specify that a package should only ever be dev. I don't see an upstream issue about that.
Comment #38
mile23Comment #39
rhovlandSo if we don't want to change default composer behavior what about allowing projects to add a section to their composer.json to hold the user's hand?
Create a plugin that adds an optional section to composer.json where a project can specify a list of packages that should always be whitelisted when updating it.
When a user runs
composer update drupal/coreorcomposer some-custom-script-nameit also updates the dev dependencies because they are part of the default whitelist defined in core's composer.json.This would also be very useful for Drupal Commerce as well since there's a huge number of packages you have to whitelist to get it to update without complaining.
I know, I'm not well versed in what the plugin system is capable of but I feel like the real problem lies in how composer operates and not in the contents of composer.json. Most PHP projects I've seen using composer only use it for dev work, and they create static packages for their end users. If drupal wants their regular users to use composer then I think it makes sense to add scripting or other affordances that simplifies some processes while still allowing developers to use it as designed.
Comment #40
mglamanThe Drupal Commerce Composer WTF is due to #2948861: Change submodule metadata to use "^maj" instead of 'self.version'
Comment #41
greg.1.anderson commentedA lot of folks have commented on this issue talking about the need for require-dev for testing; webflo/drupal-core-require-dev is part of drupal-composer/drupal-project for the same reason. However, I was looking around for documentation on how to test a Drupal site using the Drupal dev dependencies, and I couldn't find anything. The documentation on drupal.org focuses on Drupal core and modules. Blogs on testing with Behat or Drupal Test Traits do not seem to use Drupal's dev dependencies, but instead either provide their own (Drupal Test Traits) or expect the user to require-dev what is needed.
The documentation I could find on dev dependencies simply made reference to running the Drupal tests. However, the instructions for the Drupal tests themselves explicitly say that you should start with a `git clone` of drupal/drupal, not a Composer-managed site. The tests in drupal-composer/drupal-project that call through to the Drupal tests must exclude a large number of tests, and the drupal-composer/drupal-project tests themselves have been broken for a long time because Drupal tests that used to work stop working over time, because using them like this is an unsupported configuration.
I wanted to put some examples in the change record. If someone has any examples of documentation that recommends using these dependencies for testing a site, please provide them and I'll include it when I write it up. If, on the other hand, there really isn't a good use-case for when you would use these, then I don't see much point in documenting how to add them back into a project, or worrying about whether they might accidentally end up in `require` rather than `require-dev`.
Comment #42
mile23Well that's what I'm saying: There's no documentation because we haven't made the change yet. :-) We also don't really have a plan for how to support BDD/DTT against existing sites that were built from our project templates. It seems like that's what we're making here.
Comment #43
mglamanI'll add notes. I copy cores phpunit.xml and change paths, then our projects have unit and kernel tests. Even Functional JavaScript tests.
I don't use Behat anymore
Comment #44
greg.1.anderson commentedI wrote up a brief change record.
@mglaman @ultimike: If you could add any notes you may be able to provide to the placeholder documentation page, Using Drupal's test classes to write tests for a Composer-managed site, it would be appreciated.
Comment #45
mglaman@greg.1.anderson yes! I can add the information there. I just want to say that I have "come to the light" and see how this simplifies life for most users. It just does add a mess once you add dev dependencies and then try to do things with
--no-devbeyondcomposer install. That's where I was confused (never did composer require --no-dev, update --no-dev, etc.)Comment #46
greg.1.anderson commentedNow we have a change record and enough documentation to instruct users how to add the dev dependencies back in. The documentation is linked from the change record, and, with the updated patch attached to this comment, also from the post-install message.
While the documentation still needs to be finished, I think we might have enough here to get started, so I'm putting this back to 'needs review'.
Comment #47
ressaThanks for sharing that @mglaman. Your comment in #26 ("That's a negative on my side.") gave me pause, so your change of mind, as well as the opinion from @eiriksm in #27 ("I feel very strongly about removing it.") is really valuable and weighty opinions, coming from seasoned developers.
The common site builder or devops person probably doesn't visit these Composer issues to suggest improvements, but are more likely to go to the forums, or just move on after having experienced yet another cryptic message from Composer.
Comment #48
MixologicThe docs and CR both look good to me.
Should something like this go into a release notes snippet?
Comment #49
greg.1.anderson commentedI don't know if we really need a release note snippet, since the template projects have never been stable, so in theory only a few people are using them. I added one here in case the release manager wants to include one anyway.
Comment #50
MixologicYeah, I think its more of a 'tarball was always composer broken, so it didnt matter', but the 'defacto' community template _did_ come with them. So its sort of a change to standard procedure here.
Comment #51
naheemsays commentedShould the priority on this issue be moved to Major?
Getting this change in before 8.8.0 is IMO very important as then the workflow of less experienced users will not need to be changed again after this issue has landed. It will also let less experiences users to install drupal without needing to know about having to exclude dev dependencies.
Comment #52
alexpottI’m +1 to this patch. I think on balance it helps people make the right choices first up. It is a hard choice. But I think I prefer learning that I can get development tools rather than learning I have to remove them.
Comment #53
mile23Any chance of getting it in 8.8.x?
Also, we'll need to update the Project Templates documentation to tell people how to add the dev stuff back in.
Comment #54
xjmWe are already in commit freeze for 8.8.0, so without having the reviewed the issue yet, I would say it's not eligible. I'll take a closer look in a few hours.
Comment #55
greg.1.anderson commentedThere are no actual code changes here, so I hope we can make an eleventh-hour exception. Please chat with @webchick and @alexpott if possible.
If this does not go in 8.8.0, it probably should not go in 8.9.x either.
Comment #56
xjmI might prefer to revert this from 8.9.x (edit: not commit to 8.9.x for now?) than cram it into 8.8.x at the last minute. Unfortunately, "this isn't disruptive" as a statement kind of has an error bar, historically speaking. :) I'll wait to see if @catch has thoughts though.
Comment #57
xjmLike, also, we don't even commit documentation spelling fixes during a commit freeze. Much less things that change processes or best practices.
Comment #58
xjm(Sorry for the multiple comments in a row; just sort of sorting through this in my head.)
I'm not against this issue in principle. In a way I actually see it as a security hardening. I'm just concerned about the timing of the commit here.
Comment #59
greg.1.anderson commentedYeah it certainly would have been better if this issue had progressed a little faster. Having never been closely involved with the last moments of a Drupal release before, I am unfamiliar with how frozen "frozen" is. I certainly understand if it's too late. From a documentation perspective, it would be less confusing if the dev dependency handling was consistent across the Drupal versions, but we can adapt to whatever decision is made here.
Comment #60
daffie commented-1 I appreciate the effort to make it easier for site builders to use composer, only we will now add a drupalism. Composer adds development stuff by default and Drupal will do the opposite. We want to attract other PHP developers to Drupal and this will not help. My suggestion will be to go to the composer project and make the change there.
Comment #61
alexpott@daffie see #30 - this is not a Drupalism.
Comment #62
daffie commented@alexpott: I guess that you are right. It just does not feel this way for me.
Comment #63
xjmDiscussed with @larowlan and @catch. After discussing with them, I'm going to go ahead and say this issue is an issue for 8.9.x.
Based on @greg.1.anderson's earlier comment, I wanted to document here on the issue what the release phases are and the expectations for them, since we already did a lot of 11th-hour Composer Initiative work this minor, and we shouldn't have the same expectations in the future.
Issues are considered disruptive when they require changes to documentation, best practices, site owners' processes, core developers' processes, release managers' processes -- including from a previous beta or release candidate. Typically, we want all of these things to be the same for beta1 as they are for the minor release, and we would require them to be complete before the beta deadline. We made some exceptions around the Composer Initiative this quarter, because it was new in 8.8, because there were regressions and concerns that we were getting stuck with the wrong architecture for the minor release. For this issue and future issues, I think it's important to keep in mind that we're trying to complete such changes before alpha1, and at committer discretion, before beta1, if the issue is a strategic priority.
The purpose of the beta phase is to stabilize stuff we got badly wrong in beta1 that's causing regressions. You can see the list of allowed changes in the beta phase here. We might have considered this issue during the beta phase with a release note about it in the rc1 release notes, but we might have also required it to be 8.9.x even then.
The purpose of the release candidate phase is to allow widespread testing and fix any critical regressions that are uncovered. Release candidates have the same patch requirements as patch releases, which means we would not backport anything that was not (e.g.) a straightforward bugfix invisible to the site owner beyond that the bug went away. Patch releases can't include any disruptive changes, even internal API changes or bugfixes with disruptive changes, and we have the same expectation for release candidates. The things we commit at committer discretion here might typically be "a critical that makes a disruptive change but fixes a data loss or upgrade path issue". Not a DX improvement, even if it also has security hardening implications.
Finally, I'll describe how we run release windows and the expectations for the commit freeze. For most release windows, the commit freeze starts at 1200 UTC Tuesday and ends at 1200 UTC Thursday. This is to ensure nothing introduces a surprise before the release on Wednesday, and nothing introduces a change that might conflict with a hotfix after it goes out.
For alpha, beta, and occasionally RC windows, the scheduled release window is the whole week. The whole committer team discusses any issue being considered for backport during the week-long window. We flexibly start our 48 hours of commit freeze based on when strategic priorities are done -- like waiting to release beta1 until we'd deprecated SimpleTest properly, or waiting to release rc1 until a release manager had a chance to test another Comopser Initiative change that should ideally have been done before the beta but that had important implications for getting the new feature right. That was an exception based on evaluating impact vs. disruption, and it was still followed by a 48h commit freeze.
For minor releases, the commit freeze on the branch that's going to be released lasts all week. The scheduled release is on a single day. The only things we'd likely commit during this freeze would be fixes for data integrity, upgrade path, or security criticals, especially if they were regressions from the previous releases. Unlike alphas, betas, and RCs, where we can release them on any day of the week, minor releases need to be released on the scheduled date, to maintain confidence in the platform.
I've made two assumptions in evaluating this, and please correct me if I'm wrong about them:
This does not change or affect what get packaged on Drupal.org. Dev dependencies will still not be included in tagged releases. They will still be included in dev tarballs (something which Mixologic and I agree merits revisiting, but is out of scope here).
There is no regression for Composer-built sites already out there. As far as I know, all of our documentation instructs Composer site owners to always use
--no-devwhen they're deploying to production or any open environment. The docs have been that way since SA-CORE-2017-001 and the packaging change before that. If the new Composer initiative templates require--no-dev, that's parity with what we already have, and no regression from 8.7 in terms of whether dev dependencies are included when people follow our instructions.If either of those statements are false, then there's a critical security regression with the 8.8 Composer Initiative feature-set that needs to be addressed. Otherwise, though, this is a DX improvement and a form of security hardening that is eligible for a minor release during its development phase prior to alpha1, which is 8.9.x right now. If the documentation has to explain that 8.9 or 9.1 improved this over 8.8, that's fine, especially if typing
--no-deveither silently does the same thing as what's already happening or even provides a message that with that version of Drupal, it's the default behavior.Depending on the disruptions (which we should summarize in the IS), this might need a release note to explain the differences over 8.8.x. (I've tweaked the release note we already have to be a little more clear; please add any other disruptions there.) But it would probably be an allowed change as a security hardening, unless I'm missing some deeply disruptive impact besides just documentation that has one "On 8.8.x" header and another "On 8.9.x or higher" header.
Hope this helps explain the context and set expectations for the future.
Comment #64
xjmThe relevant handbook documentation that covers all this policy:
Comment #65
xjmComment #66
xjmI've also updated the section on "What is a disruptive change?" to cover the scenarios that have come up for this initiative:
https://www.drupal.org/core/d8-allowed-changes#disruption
Comment #67
mile23Based on #63 this issue is either 'closed (works as designed),' because the user can say
composer create-project --no-dev, or still RTBC for 8.9 with a documentation update saying that if you want dev, you can add the appropriate package.Adding 'needs documentation updates' because either way we have to tell the user how to, or how not to have dev packages: https://www.drupal.org/docs/develop/using-composer/starting-a-site-using...
Comment #68
greg.1.anderson commentedSo, we actually ended up making a change in Drupal Infra. When we split out the template project, we remove the dev dependencies iff the subtree split is a tag. If the split is being done for a dev branch, the dev dependencies are left in.
We could decide whether to change this for 9.x. We should still update the documentation. I updated the issue summary.
Comment #69
greg.1.anderson commentedBack to "Needs review" for discussion / decision on 9.x policy.
Comment #70
mglamanSo, did this end up getting committed? I did a fresh
drupal/recommended-projectbuild and there were no dev dependencies, I had to runcomposer require --dev drupal/core-dev-pinned. So I thought this was closed.. but it's not.Comment #71
greg.1.anderson commented@mglaman See #68. The Drupal Infrastructure scripts remove the dev dependencies as part of the subtree split operations.
If you want to add the dev dependencies back in, it's probably better to use the non-pinned variant,
drupal/core-dev. Adding pinned dependencies will make your site harder to update.Comment #72
mmjvb commentedYou might want to extend it to settings 'minimum-stability' and 'prefer-stable'. These settings don't belong in non-dev projects.
Comment #73
greg.1.anderson commented#72 is out of scope for this issue. If we want to discuss minimum-stability further, someone should open a follow-on issue.
Comment #78
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #79
mile23The current behavior is that for dev branches, the project templates include drupal/core-dev as a require-dev. For tagged releases, drupal/core-dev is excluded.
Release:
Dev:
So in the meantime we split the difference: Releases don't risk including dev requirements by default, while dev setups include the dev tools.
The question then becomes: Should we exclude the dev tools from the dev project, and it's either +1 for consistency, or -1 for not letting users decide to take fate into their own hands by using @dev without having to type
composer require drupal/core-devI think I'm +1 for consistency, where we just never add drupal/core-dev to the project templates, and we tell users to add the package they need. Obviously real actual maintainers might have a different view. :-)
Since we have a CR, setting the issue to target Drupal core 10.1.x.