I have a drupal 8.3 installed with composer
When i try to update core to 8.4 with : composer update drupal/core --with-dependencies
he told me everythings is up to date.
when i look into composer.json :
"drupal/core": "~8.0",
"drush/drush": "~8.0",

I have drush 8.1.14 installed so i try to changed composer.json with :
"drupal/core": "~8.4",
"drush/drush": "~8.1.14",

then run composer update drupal/core --with-dependencies
but nothing happened, ssh is blocked.

So how can I update the composer.json automatically for updating core ?

Comments

ergophobe’s picture

What output did you get from Composer?

I tried the same thing and it failed because I was on PHP 7.0.0 and the Symfony update failed because it needs 7.0.8 or higher.

With Drupal 8, minor version upgrades have become a major headache. So easy to get caught in dependency hell. The Windows Vista of Drupal releases...

It's all kind of aggravating. If I run
composer outdated
It says that 8.3.7 is outdated and should be updated to 8.4.0

But if I run
composer update --with-dependencies drupal/core
it says "Nothing to install"

If I update the composer.json file so that it specifies 8.4
"drupal/core": "~8.4",
that doesn't work because 8.3.7 specifies
"symfony/class-loader": "~2.8",
Which conflicts with the requirements in 8.4, so the whole thing fails.

Problem 1
    - Conclusion: don't install drupal/core 8.4.0
    - Conclusion: don't install drupal/core 8.4.0-rc2
    - Conclusion: don't install drupal/core 8.4.0-rc1
    - Conclusion: don't install drupal/core 8.4.0-beta1
    - Conclusion: don't install drupal/core 8.4.0-alpha1
    - Conclusion: don't install drupal/core 8.5.x-dev
    - Conclusion: remove symfony/class-loader v2.8.28
    - Installation request for drupal/core ~8.4 -> satisfiable by drupal/core[8.4.0, 8.4.0-alpha1, 8.4.0-beta1, 8.4.0-rc1, 8.4.0-rc2, 8.4.x-dev, 8.5.x-dev].
    - Conclusion: don't install symfony/class-loader v2.8.28
    - drupal/core 8.4.x-dev requires symfony/class-loader ~3.2.8 -> satisfiable by symfony/class-loader[3.2.x-dev, v3.2.10, v3.2.11, v3.2.12, v3.2.13, v3.2.8, v3.2.9].
    - Can only install one of: symfony/class-loader[3.2.x-dev, v2.8.28].
    - Can only install one of: symfony/class-loader[v3.2.10, v2.8.28].
    - Can only install one of: symfony/class-loader[v3.2.11, v2.8.28].
    - Can only install one of: symfony/class-loader[v3.2.12, v2.8.28].
    - Can only install one of: symfony/class-loader[v3.2.13, v2.8.28].
    - Can only install one of: symfony/class-loader[v3.2.8, v2.8.28].
    - Can only install one of: symfony/class-loader[v3.2.9, v2.8.28].
    - Installation request for symfony/class-loader (locked at v2.8.28) -> satisfiable by symfony/class-loader[v2.8.28].

Installation failed, reverting ./composer.json to its original content.

So as I mentioned above, you're stuck in dependency hell. You can't update to 8.4 because Symfony 2.8.28 is incompatible with 8.4. You can't update to Symfony 3.2 because it's incompatible with 8.3.7.

Aggravating. After many attempts with many different combinations, I have no idea how to update Drupal to 8.4

ergophobe’s picture

It seems that this is a pretty common problem and most people find that to update Drupal they need to
- delete /vendor
- delete composer.lock

See
https://www.drupal.org/node/2914193
https://www.drupal.org/node/2913970
https://www.drupal.org/node/2879621
https://drupal.stackexchange.com/questions/234321/updating-core-from-8-2...

That did not work for me, so I tried deleting the /web/core directory as well and then running composer.

No luck. No matter what I do, I'm stuck in dependency hell.

This is awful. I can't imagine doing this every few months with each minor upgrade. After using Drupal since 4.5, I think Drupal 7 may be my last version. This is my third effort with Drupal 8 and every experience has been terrible.

beck24’s picture

I'm also having the exact same issue :/

ergophobe’s picture

Finally sorted this out. What a PITA.

The Drupal 8.4 release notes say that you need to upgrade to Drush 8.1.14, but when I ran composer update, it was choking because Drush 8.1.14 is incompatible with Drupal 8.4

If you look at the Drush docs, it says there that the 8.x branch is only for Drupal 8.3- and Drush 9.x is for Drupal 8.3+

So same dependency problem. There's also the Symfony dependency problem as well, where you need ~3.2.8, but you're locked at 2.7 by the Drupal 8.3 composer.json file.

I had previously deleted /vendor and /web/core from my project. I'm not sure that's necessary.

I finally just explicitly specified the versions I needed in my update commmand

composer update --with-dependencies drupal/core:~8.4 symfony/class-loader:~3.2.13 drush/drush:~9.@dev

If you have the default minimum-stability=stable, you need the @dev for drush even though in theory the tilde operator should handle that. I'm not sure why, but I needed the @dev despite the tilde operator

That finally got me upgraded to Drupal 8.4

It also destroyed my Drush install. So then I did another install of Drush and it sort of worked, but it gave warnings because all the cache commands were missing and for some reason it will not recreate them. So I had lots of error messages likes

[warning] file_put_contents(C:/Users/Tom/.drush/cache/factory/Drush\Drupal\Commands\core\DrupalCommands.cache): failed to open stream: No such file or directory JSONCache.php:25

As you can see, this is on a Windows dev box and Drush had never worked great with Windows, so that is probably the issue here.

I fixed this by simply going into /vendor/drush/drush/src and copying
./Drupal/Commands
./Commands

to the

~/.drush/cache/factory/Drush

directory and using Bulk Rename Utility to rename all .php files to .cache

Now everything works... but of course the second your clear your cache, all the cache files are missing again and for some reason Drush is not generating them.

Clearly this is not a process I want to repeat. There has to be a better way. I'm starting to think the easiest way around this is just to switch to Backdrop!

ergophobe’s picture

PS - this also messed up my prompt when I ran drush init, which adds the drush prompt customizations to .drush and sources them in your .bashrc

Despite commenting the sourcing out of my .bashrc, I couldn't get the prompt back (the drush prompt shows the whole file path, which can mean your prompt takes up the whole terminal window).

Anyway, I was changing the prompt back and was looking something up and found this
http://ezprompt.net/

Great tool to customize your prompt. Now I just have the current dir (not the whole path) and the git branch.

samuhe’s picture

I had the same problem.
Running composer require drush/drush:^9 before running composer update worked for me.

ergophobe’s picture

Good to know.

This was giving me trouble because I had the old drush-shim installed
https://packagist.org/packages/webflo/drush-shim

Since that project was abandoned and had no Drush 9.x version available, I couldn't get out of the dependency problem.

I suspect if I had removed webflo/drush-shim first, then tried composer require drush/drush:^9, I would have avoided a lot of headache

ergophobe’s picture

composer show -t > dependencies.txt

Run that and it's easy to see if a given package can be removed without unanticipated consequences or a dependency conflict.

ergophobe’s picture

See this Github thread for Drush

https://github.com/drush-ops/drush/issues/3033

zorax’s picture

finally, the more simple :
remove folder /vendor
remove composer.lock
and run composer update
that's ok !

VVS’s picture

websiteworkspace’s picture

1.
edit composer.json
change drupal version to ^8.4
if any change drush version ~9.0

2.
rename /vendor to /_vendor
rename /core to /_core
rename composer.lock to _composer.lock
- - available for easy rollback if problems occur

3.
run composer update drupal/core

4.
from browser run {site}/update.php

5. clear the cache from either admin GUI or drush

6. check basics from admin GUI, such as status report

7.
delete /_vendor
delete /_core
delete _composer.lock

8. proceed to Drupal 8.4.0 - test carefully
- this update causes various known problems, such as breaking existing site features that use jquery

srhcpa@debitsandcredits.com’s picture

Thank you @DrupalSiteBuilder, that appears to be a very clean and clear set of instructions to a very gnarly upgrade. I have been dragging my feet until I saw your post.

One other question that has arisen in this upgrade and that is which Symphony version we should use, when we should upgrade and is there an order of upgrading, e.g. before or after some other component upgrade?

Thanks

raushan’s picture

Thank you very much. its works for me.

Rockabelly’s picture

This is what worked for me

Requirement : PHP 7.1
Explanation :
1. Doctrire, a dependancy of Drupal core, requires PHP 7.1. Doctrine uses the latest PHP 7.1 syntax to declare the methods' return type eg. function my_function() : void {...}
2. webflo breaks the chain of dependencies
3. Drupal 8.4 requires Drush 9.0

If you have PHP 7.1 on your server you can do the following:
1. edit the composer.json file

	"drupal/core": "~8.4"
	"drush/drush": "~9.0"

2. delete composer.lock
3. update webflo
composer require "webflo/drupal-finder ~1.0.0"
4. update the database
../vendor/bin/drush updb -y
5. update the entities schemas
../vendor/bin/drush entup -y
6. clear the cache
../vendor/bin/drush cr

webjazz’s picture

...then skip composer and use drush from the web root.

drush up

Worked like a charm for me when upgrading from 8.3.7 to 8.4.0.  

ptgraber’s picture

I've got 8.1.13  and PHP 7.0.2 and I'm leary...

badrange’s picture

This worked nicely for me; you might want to skip the phpunit there if your project doesn't have it. 

composer require drupal/core:^8.4.0 drush/drush:^8.1.15 phpunit/phpunit:^4.8 drupal/console:^1.0 --update-with-dependencies

Internet’s picture

As "steps - d 8.4.0 update with composer" above, but a few gotchas I had to research:

"require": {
        "drupal/core": "^8.4.0",
        "drush/drush": "~9.0"

Remove:     "replace": {
        "drupal/core": "~8.3"
    },

Also when adding modules I found duplications outside of the contrib folder that I removed and then composer required again to the latest version, this placed them in contrib folder.

Drush updb then drush cr

More detailed options here: https://www.drupal.org/docs/8/update but apparently once you use composer on a site you have to stick with it for updating core.

freshadmin’s picture

Spent a long time puling my hair out and then your solution worked.

Legend.

amcc’s picture

i still had issues while doing this with symphony/yaml and console-core in particular. After running update drupal/core a lot of issues came up. A couple of snippets are below

symfony/yaml 2.7.x-dev conflicts with symfony/dependency-injection[3.2.x-dev].
drupal/core 8.4.x-dev requires symfony/dependency-injection ~3.2.8 -> satisfiable by symfony/dependency-injection[3.2.x-dev, v3.2.10, v3.2.11, v3.2.12, v3.2.13, v3.2.14, v3.2.8, v3.2.9].

    - Conclusion: don't install symfony/dependency-injection v3.2.8

Problem 1
 - drupal/console 1.3.2 requires drupal/console-core 1.3.2 -> satisfiable by drupal/console-core[1.3.2].

by running this the installation went ahead:

composer remove drupal/console drush/drush

i updated drush like so:
composer require drush/drush:^9.0.0

i found this page helpful:
https://www.isovera.com/blog/upgrading-drupal-840-and-drush-9-composer

andrezstar’s picture

what you explained is pretty cool, but i still have some trouble

When I run composer prohibits drupal/core:8.5.0
drupal/core    8.5.0        requires  symfony/class-loader (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/class-loader (~3.2.8)
drupal/core    8.5.0        requires  symfony/console (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/console (~3.2.8)
drupal/core    8.5.0        requires  symfony/dependency-injection (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/dependency-injection (~3.2.8)
drupal/core    8.5.0        requires  symfony/event-dispatcher (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/event-dispatcher (~3.2.8)
drupal/core    8.5.0        requires  symfony/http-foundation (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/http-foundation (~3.2.8)
drupal/core    8.5.0        requires  symfony/http-kernel (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/http-kernel (~3.2.8)
drupal/core    8.5.0        requires  symfony/process (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/process (~3.2.8)
drupal/core    8.5.0        requires  symfony/routing (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/routing (~3.2.8)
drupal/core    8.5.0        requires  symfony/serializer (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/serializer (~3.2.8)
drupal/core    8.5.0        requires  symfony/translation (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/translation (~3.2.8)
drupal/core    8.5.0        requires  symfony/validator (~3.4.0)
drupal/drupal  dev-MAJD8.5  requires  symfony/validator (~3.2.8)
drupal/core    8.5.0        requires  symfony/yaml (~3.4.5)
drupal/drupal  dev-MAJD8.5  requires  symfony/yaml (~3.2.8)

and then  composer update drupal/core symfony/* --with-dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for symfony/yaml ~3.2.8 -> satisfiable by symfony/yaml[3.2.x-dev, v3.2.10, v3.2.11, v3.2.12, v3.2.13, v3.2.14, v3.2.8, v3.2.9].
    - drupal/core 8.5.0 requires symfony/yaml ~3.4.5 -> satisfiable by symfony/yaml[3.4.x-dev].
    - drupal/core 8.5.0-alpha1 requires symfony/yaml ~3.4.0 -> satisfiable by symfony/yaml[3.4.x-dev].
    - drupal/core 8.5.0-beta1 requires symfony/yaml ~3.4.0 -> satisfiable by symfony/yaml[3.4.x-dev].
    - drupal/core 8.5.0-rc1 requires symfony/yaml ~3.4.0 -> satisfiable by symfony/yaml[3.4.x-dev].
    - drupal/core 8.5.1 requires symfony/yaml ~3.4.5 -> satisfiable by symfony/yaml[3.4.x-dev].
    - drupal/core 8.5.x-dev requires symfony/yaml ~3.4.5 -> satisfiable by symfony/yaml[3.4.x-dev].
    - drupal/core 8.6.x-dev requires symfony/yaml ~3.4.5 -> satisfiable by symfony/yaml[3.4.x-dev].
    - Conclusion: don't install symfony/yaml 3.4.x-dev
    - Installation request for drupal/core ^8.5 -> satisfiable by drupal/core[8.5.0, 8.5.0-alpha1, 8.5.0-beta1, 8.5.0-rc1, 8.5.1, 8.5.x-dev, 8.6.x-dev].

plato1123’s picture

Thank you so much for adding these additional steps, I spent at least hours trying to solve this.  

webjazz’s picture

The link provided above by amcc solved my problems for good! Here is the excerpt

Every composer project is different but in general:

- update one package at a time starting with drupal/core

- read the errors and remove any conflicting packages 

- require the removed packages after you successfully installed drupal/core

- update contrib modules using the above procedure

Here is the link once more: https://www.isovera.com/blog/upgrading-drupal-840-and-drush-9-composer

ressa’s picture

I am testing out a Composer based work-flow, and just updated a vanilla Drupal with these steps:

# download Drupal 8.4.0
$ composer create-project drupal-composer/drupal-project:8.x-dev my_web_site --stability dev

# update to Drupal 8.4.4 and Drush 9.0.0-rc2
$ cd my_web_site 
$ composer require "drupal/core:~8.4.4" "drush/drush:~9.0"

# it will update these two lines in composer.json, as well as update some packages:
"drupal/core": "~8.4.4",
"drush/drush": "~9.0",

# finish the update, apparently some extra packages needs updating
$ composer update --with-dependencies

Result:

$ vendor/drush/drush/drush st
Drupal version : 8.4.4
Drush version  : 9.0.0-rc2

EDIT: I know this isn't an update from 8.3.x, but I ended up here searching for a way to get the latest Drupal 8.4.4 version via Composer.

freelylw’s picture

this is a terrible experience for install the D8, since the D6, this is the worse version to let people get on with so far. for sure it will kick thousands thousands people out from drupal

the first time I install D8 through upload to ftp, then I install commerce & lots related commerce modules through composer. after a while I realise the D8.5 has some new feature, then follow the instruction, I delete the "core/vendor"folder and those individual files to be replaced with the new version files/folder, then immediately I got a error message on the screen, the whole site is dead, only a error message showing.

luckly I have a backup, then I put those files back through ftp, and then spent me few hours to update the core through composer, never works, at the end, I see someone saying this , because the first installation was through upload by ftp, so you can't upgrade through composer, it won't work, it waste me few hours.

then I remove the whole site to install again from blank through composer, and then I try to update to the latest alpha1 version again through composer by use the common line : 

composer update drupal/core:8.5.0-alpha1 --with-dependencies
composer update drupal/core:8.5.0 --with-dependencies
composer update drupal/core:8.5 --with-dependencies
composer update drupal/core --with-dependencies

none of them are works, it says on the screen : "Package "drupal/core:8.5.0-alpha1" listed for update is not installed. Ignoring."

SO,can anyone tell me how can I upgrade to the latest alpha1 version through composer ?

( I realise you can't simply delete the vendor folder to replace the latest one, since the commerce module installed through composer, there are lots links in this folder, once the vendor folder removed, the commerce module dead )
 

webjazz’s picture

From your terminal enter composer require drupal/core:"version". This will make an entry in a composer.json file in the root of your project. Do not change the composer.lock file. It lists what composer installed for you . Google for free composer tutorial to get familiar with the principles behind it. Composer can install a lot o packages from different vendors. Not only drupal

freelylw’s picture

Thanks for reply, It still doesn't work.

I use : composer require drupal/core:"8.5.0-rc1" first,

then I got the message :

./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
 

And then I use :
composer update drupal/core:8.5.0-rc1 --with-dependencies
composer update drupal/core:8.5.0-rc1

I got the message :
"Package "drupal/core:8.5.0-rc1" listed for update is not installed. Ignoring."

I use :  composer update drupal/core 

I got the message:

Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update

At the end, just nothing being updated. Can you please suggest what should I do to solve this to update to the version "8.5.0-rc1"

Thank you.
 

estevan_carlos’s picture

Updating Drupal should not be like this.

deronimo’s picture

Agreed! Very frustrating. I am still trying to figure it out. If I come up with anything, I'll be sure to post it.

dotist’s picture

I was only able to update to 8.5.0 by manually changing the composer file:

"drupal/core": "^8.5"

and then running composer update --with-dependencies

Rick Hood’s picture

Things I understand and do not understand about update core with composer:

1. Always delete composer.lock file.  This I understand.

2. Sometimes delete the vendor folder is needed.  I do not understand the “when/why" of when I need to do this.

3. Core does not update unless “drupal/core” is in the “require” section of  composer.json not the “replace” section:

"require”: {
"drupal/core": "^8.4"
},

not

"replace": {
"drupal/core": "^8.4"
},

So, why does Drupal ship with it in the replace section?

gunwald’s picture

So, am I supposed to change that? Here I am told to completely migrate to https://github.com/drupal-composer/drupal-project, what ever that means, and why I would be better is out of scope of my retarded mind. So please can somebody of the almighty Drupal gods help us here!

Rick Hood’s picture

Forget about https://github.com/drupal-composer/drupal-project you don't need to migrate to that. 

Try this:

1.  In the composer.json file in the root of your site, change "drupal/core": "^8.x" to whatever you want to update to.  For example for 8.4 to 8.5 change it to  "drupal/core": "^8.5" - for 8.x to 8.4.6 change it to   "drupal/core": "8.4.6".

2. Make sure  "drupal/core": "^8.x" is in the require, for example:

"require”: { "drupal/core": "^8.5" },

not

"replace": { "drupal/core": "^8.5" },

3. Delete the composer.lock file in the root of your site -- it gets recreated in step 4.  UPDATE: I have been told this is not correct (e.g. don't delete it).  It does get recreated in step 4 anyways though.

4. Now run this:  composer update drupal/core --with-dependencies

Let us know if that works.  It worked for me. 

UPDATE:  here is a step by step I posted today:  https://drupal.stackexchange.com/a/258955/35313 

amcc’s picture

I wrote a how-to to solve troublesome drupal updates. This works to get to 8.5.1 (latest version currently)

If anything doesn't work or you think there should be changes to this let me know and i'll change it. I think something like this needs to be on the official guide:

https://am-digital.tumblr.com/post/172489444983/updating-drupal-8-core-a...

Rick Hood’s picture

Thanks, couple of questions:

1. Is Drush 9 required for 8.5.x?  I know Drush 8.1.15 (or higher) is required for 8.4.x.  Just curious.

2. If you end up deleting the original vendor folder (step 4 in your post), don't you lose stuff that got put in there from installing modules?  Running composer update drupal/core –with-dependencies will put the stuff in the vendor folder that Drupal needs, but not stuff that modules need (if anything). 

I believe you could run composer update to get the vendor stuff back for the modules, but depending on what composer.json has in it for module versions, it might update those modules, which you may or may not want to do.  For example:

"drupal/address": "^1.2"  < if this is in composer.json then composer update will update that module to anything higher than 1.2 but lower than 2.0.

"drupal/address": "1.2"  < if this is in composer.json then composer update will keep the module at 1.2 and will restore what is missing but needed (if anything) in the vendor folder. I believe running composer require drupal/address:1.2 would do the same thing (put back the stuff needed in vendor).

I am not quite sure when /vendor needs to get deleted for updates to go well. I feel like in theory it should not and worst case is removing and reinstalling stuff that is in there but blocking an update. 

Also, here is my step by step I posted today:  https://drupal.stackexchange.com/a/258955/35313

Also, "stuff" referred to above means php libraries added to the vendor folder by composer.  ;-) 

amcc’s picture

I don't think drush 9 is needed now. I saw an update on the blog post I added to my how to article that says it isn't. Perhaps it never was... Does the job though 

I've not had any trouble with reinstalling the vendor folder. All modules seemed to function fine. But they've not been complex drupal installs and perhaps they had nothing additional in the vendor folder. It does seem a very blunt instrument to remove the folders. However it works! 

Your tutorial is certainly more nuanced and better researched! 

I don't understand composer well enough to know what the story is with the lock file and why it needs deleting or not. 

Rick Hood’s picture

Yeah looks like actually Drush 9 is recommended for 8.4.x and higher:  http://docs.drush.org/en/master/install/   

And I see it is recommended to make sure to include drush as a requirement, which your blog post pointed out. 

"require": {
stuff....
"drush/drush": "^9.0.0",
more stuff...
},

That's cool that you can specify drush version by site, rather that using globally installed version.

rgpublic’s picture

Even though it's getting old, some very valid things have been mentioned in this thread. Though we are using a fully Composer-based workflow now successfully with a lot of Drupal websites, I think the whole experience for newcomers could and should be better.

1) I also don't understand why Drupal is shipped with drupal/core in the replace section when updating only works when it's in the require section. I've written a script to automatically patch the composer.json but I think it's ridiculous.

2) Next, I don't understand why drupal create-project is advocated everywhere as if it was free beer, even though I see hardly any need for it and it causes some disturbing complexities (e.g. that you have an additional folder level you don't want etc.).

IMHO: The "plain" method without any create-project stuff is by far the easiest for newcomers. And as an added benefit you can easily transition existing websites to this method. At least it should be mentioned *somewhere*. Back then when I wanted to transition to Composer I found this as my first search result:

https://www.drupal.org/docs/develop/using-composer/using-composer-to-man...

Option A, B and C are all equally confusing and the very easy method of just moving drupal/core from replace to require isnt even mentioned there.

amcc’s picture

Could you detail your method or where you found it. Agree with everything you're saying btw. 

I think as a beginner trying to install drupal your faced with a terrible experience now. Imagine transitioning from WordPress to Drupal now. 

A single method should be advocated and clearly explained. There's 3 methods, one is marked depreciated (so why is it even there). The second method is on a repro with a developers name on it rather than Drupal. It all seems so alien to my experience installing any other framework/library/CMS. 

Maybe this thread needs copying to a documentation issue...

rgpublic’s picture

This is the problem: I didn't find the method on a specific page but figured it out by reading comments and bug reports. The method is  actually fairly simple: Just open composer.json in the Drupal root folder and move "drupal/core" from the "replace" section to the "require" section. It's really just that. I'm doing this now for every D8 installation ever since and never had any problems with it. Installing modules with Composer, updating core and modules with Composer - everything works just great. I really wonder why this very simple option isnt at least *mentioned* on that page, or even better why Drupal doesnt ship with this by default, and why instead there are these 3 very weird and partially even deprecated alternatives. There might be reasons I'm not aware of but they arent even explained properly.

ressa’s picture

I am also very interested in hearing what your process looks like, in details. Do you just download Drupal core, and then update with composer upgrade with something like these steps?

1. Get older version (8.5.3) of Drupal 8 core, for testing an update
mkdir drupal && cd drupal && curl -sSL https://ftp.drupal.org/files/projects/drupal-8.5.3.tar.gz | tar -xz --strip-components=1

2.1 Update composer.json ...
"require": {
"composer/installers": "^1.0.24",
"wikimedia/composer-merge-plugin": "^1.4"
},
"replace": {
"drupal/core": "^8.5"
},

2.2 ... to 
"require": {
"composer/installers": "^1.0.24",
"wikimedia/composer-merge-plugin": "^1.4",
"drupal/core": "^8.6"
},

3. Update Drupal 8 core to Drupal 8.6
composer update

rgpublic’s picture

Yes, actually that's basically it. Of course you have to insert a 1.1 "composer require drush/drush" and a 1.2 "drush site-install ..." to get a really working 8.5.3 site first. You can try to log in and create content-types and stuff. Then, you do step 2 and 3 and after step 3 you need to do "drush updb" and usually I also do "drush entity-updates" which shouldnt be necessary but just to make sure that all our nice entities are really doin' well. Then I reload the web server to flush any possible APCu/Opcode caches. I've used this procedure on LOTS of Drupal sites now and never had a problem. I really wonder why this very simple process isnt mentioned on that page and the poor newbie user who perhaps even has it's first contact with Composer unnecessarily needs to be shocked with those crazy complex Options A/B/C on that page.

ressa’s picture

Thanks for adding some more details. I just ran through the process and got it to work, after a few challenges.

# 1.0 download Drupal
$ mkdir drupal && cd drupal && curl -sSL https://ftp.drupal.org/files/projects/drupal-8.5.3.tar.gz | tar -xz --strip-components=1

# 1.1 install Drush
$ composer require drush/drush

# 1.2 install Drupal 8.5.3
$ drush site-install --db-url=mysql://root:root@localhost/d853 -y;

# 2.1 update composer.json ...
   "require": {
       "composer/installers": "^1.0.24",
       "wikimedia/composer-merge-plugin": "^1.4"
   },
   "replace": {
       "drupal/core": "^8.5"
   },

# 2.2 ... to
   "require": {
       "composer/installers": "^1.0.24",
       "wikimedia/composer-merge-plugin": "^1.4",
       "drupal/core": "^8.6"
   },

# 3.1 Update Drupal 8 core to Drupal 8.6.1
# 1st attempt:
$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 11 installs, 54 updates, 1 removal
- Removing ircmaxell/password-compat (v1.0.4)
- Updating composer/installers (v1.5.0 => v1.6.0): Downloading (connecDownloading (100%)      
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/yaml (v3.4.5 => v3.4.17): Downloading (connecting..Downloading (100%)      
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating squizlabs/php_codesniffer (2.8.1 => 3.3.2): Downloading (coDownloading (100%)      
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating drupal/coder (8.2.12 => 8.3.1):    The package has modified files:
   D coder_sniffer/Drupal/Test/Array/ArrayUnitTest.inc
   D coder_sniffer/Drupal/Test/Array/ArrayUnitTest.inc.fixed
   D coder_sniffer/Drupal/Test/Array/ArrayUnitTest.php
   D coder_sniffer/Drupal/Test/Array/DisallowLongArraySyntaxUnitTest.php
   D coder_sniffer/Drupal/Test/Array/disallow_long_array_d7/DisallowLongArraySyntaxUnitTest.1.inc
   D coder_sniffer/Drupal/Test/Array/disallow_long_array_d7/disallow_long_array_d7.info
   D coder_sniffer/Drupal/Test/Array/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc
   D coder_sniffer/Drupal/Test/Array/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc.fixed
   D coder_sniffer/Drupal/Test/Array/disallow_long_array_d8/disallow_long_array_d8.info.yml
   D coder_sniffer/Drupal/Test/Classes/ClassCreateInstanceUnitTest.inc
   237 more files modified, choose "v" to view the full list
   Discard changes [y,n,v,d,s,?]? ^Z

# 2nd attempt:
# Found this: https://www.drupal.org/forum/support/installing-drupal/2017-10-15/compos...
# Try adding " --no-interaction"
$ composer update --no-interaction
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 11 installs, 52 updates, 0 removals
- Updating drupal/coder (8.2.12 => 8.3.1):
                                                                    
[RuntimeException]                                                
Source directory /var/www/public/drupal2/vendor/drupal/coder has uncommitted changes.

# 3rd attempt, SUCCESS:
# Found this: https://stackoverflow.com/questions/12402345/composer-uncommited-changes...
# Try setting Composer to discard changes by default, and run update again:
$ composer config -g discard-changes 1
$ composer update --no-interaction
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 11 installs, 52 updates, 0 removals
- Updating drupal/coder (8.2.12 => 8.3.1): Checking out 29a25627e7
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating guzzlehttp/guzzle (6.3.0 => 6.3.3): Downloading (100%)
...
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

# 3.2 finish update
$ drush updb -y
$ drush entity-updates -y
$ drush cr

$ drush st
Drupal version  : 8.6.1
Drush version   : 9.4.0

I really wonder why this very simple process isn't mentioned on that page and the poor newbie user who perhaps even has it's first contact with Composer unnecessarily needs to be shocked with those crazy complex Options A/B/C on that page.

I couldn't agree more, and if "drupal/core": "^8.x" was moved to the require section in the official Drupal core release, the documentation for newbie users to download Drupal, get Drush and upgrade could be boiled down to this:

$ mkdir drupal && cd drupal && curl -sSL https://ftp.drupal.org/files/projects/drupal-8.5.3.tar.gz | tar -xz --strip-components=1
$ composer require drush/drush
$ drush site-install --db-url=mysql://root:root@localhost/d853 -y;
# update to 8.6: update Drupal core version in composer to "drupal/core": "^8.6" (assumed already under require section)
$ composer config -g discard-changes 1
$ composer update --no-interaction
$ drush updb -y
$ drush entity-updates -y
$ drush cr

rgpublic’s picture

Yeah, you are right about the problem with code_sniffer. I remember it now that I see it. I never gave it much thought and just answered "y". That's why I didnt mention it here. Personally, my intent never was to make updates 100% automated. If you want that, your solution with "discard-changes" is of course correct to circumvent the problem - although I think a better solution going forward instead of recommending "discard-changes" would be to avoid the weird "uncomitted changes" problem in the first place. Don't know what's going on there but I think it's a bug in the code sniffer composer package. Otherwise I cant think of a sensible reason why there are "uncomitted changes" in a *fresh* Drupal install in the first place.

I don't know even what that package does and who is taking care of it. Perhaps we should file a bug?

ressa’s picture

It would be great to have a script which automatically applies security updates for contrib modules or Drupal core with Composer, basically the equivalent of drush up --security-only, but for Composer. I seem to recall that Composer can't discern between normal updates and security updates, so perhaps it's not possible ...

You should totally file a bug for that codesniffer thing :-)

rgpublic’s picture

Okay, I filed a bug: https://github.com/squizlabs/PHP_CodeSniffer/issues/2184

Let's see what happens. I'm also missing decent security update functionality. It's even difficult to find out which packages could be updated *at all*. There is "drush pm:security" which only works for Drupal stuff. But not for other non-Drupal Composer stuff. Then there is "https://security.symfony.com/" which can be used for non-Drupal stuff but, of course, not for Drupal stuff.

Then there is "composer outdated" which lists a lot of packages that could be updated, but which you cannot actually update due to version restrictions. It's the proverbial mirage on the horizon for version updates. You try to approach the oasis. Zap. It's gone.

Next, there is "composer update --dry-run" which lists a lot of stuff that can be updated but also says that it will uninstall a lot of stuff. Which it won't when you'd actually run it without --dry-run. But when you grep only the lines which contain "Updating" you're getting close. I still skip symfony/debug because it's still wrongly listed as updatable when it actually isnt.

Phew. It could very well be a parody but it's very real. This is how updating of Drupal works in 2018.

I think, if you combine the information from security.symfony.com and drush pm:security into a list of packages and pass that as arguments to "composer update ..." then I think you might be getting close to what you want.

I still wonder though why nobody sorted all this out before recommending/unleashing Composer on us Drupal users.

imclean’s picture

Then there is "composer outdated" which lists a lot of packages that could be updated, but which you cannot actually update due to version restrictions. 

I think we're supposed to let Drupal decide when to update its composer dependencies. For example, see Update core dependencies before 8.6.2.

rgpublic’s picture

First a quick update: code_sniffer seems to be the wrong project for the "files have been modified bug". The problem is in drupal/coder which erroneously seems to include a .git folder:

https://www.drupal.org/project/coder/issues/2939229

Next, yes, imclean, you're basically right if you have a clean Drupal installation and nothing else. But usually you have additional composer packages installed for your website. You need to track those for security issues as well.

IMHO the situation is far from optimal. Why was Composer introduced in the first place? To get "off the island" i.e. to embed Drupal more into the common PHP environment where Composer is the de-facto standard package manager - making it in turn easy to use functionality invented elsewhere. Which is all very nice. I see where this is heading. And now web devs are making use of this. Installing stuff from the great PHP/Composer universe into their Drupal website. Great. And then they realize: Umm, how do I keep my site secure?

I think we're only halfway thru with the Composer transition - stuck somewhere in the middle. Which is okay to some extent, because that's how open source sometimes works. There has to be some first steps to encourage others to push things further. But: I think Drupal core team should have been WAY more clear and transparent to users on what to expect. They are advertising cool new features for 8.5, 8.6 like crazy (media library, layouts etc), but they don't say *anything* about the fact that some aspects of Drupal (Caching, Composer) are right now still in a very incomplete state. I wonder: why do we need 2 completely separate systems to track our Composer-based website for security updates? It just doesnt make sense IMHO.

imclean’s picture

I agree @rgpublic and well put. 

...they don't say *anything* about the fact that some aspects of Drupal (Caching, Composer) are right now still in a very incomplete state. I wonder: why do we need 2 completely separate systems to track our Composer-based website for security updates? It just doesnt make sense IMHO.

There are a number of issues where Composer problems are being discussed.

There may be others but these are the ones I've come across recently. It might be worth adding your feedback to one of those somewhere.

rgpublic’s picture

With the new composer version 1.7.3, it's unfortunately also necessary to modify core/composer.json in addition to composer.json. Otherwise you'll end up with lots of conflicts. Took me a while to figure this out. In core/composer.json, you'll also need to merge all the stuff in replace into the require section while changing "self.version" to "@dev". Crazy. Until the Drupal team finally has mercy some day and delivers a decent setup out-of-the-box which doesnt need rocket science to update with composer, I've created a script to get back my sanity. Perhaps this also useful for someone else. I dont know whether it will work for everyone - make sure you do a backup before. Pull requests highly welcome.

https://github.com/rgpublic/drupal_composer_fix

You can just call the script from your Drupal root folder and it will automatically detect which changes are necessary and it fixes everything so you can update properly. Since updates of Drupal core will overwrite core/composer.json, make sure you run the script after any core update.

gillesv’s picture

@rgpublic, thanks for the info and temporary fix. We experienced the same issue. We suddenly couldn't execute "composer install" on one of our servers, without it giving dependency issues:

remove drupal/drupal dev-master|remove drupal/core 8.6.2
- don't install drupal/core 8.6.2|remove drupal/drupal dev-master
- Installation request for drupal/drupal dev-master -> satisfiable by drupal/drupal[dev-master].
- Installation request for drupal/core 8.6.2 -> satisfiable by drupal/core[8.6.2].

After a while I figured out it was due to a newer version of composer on that specific server. v1.7.2 does not give this problem, v1.7.3 does.

gillesv’s picture

There seems to be an easier fix. I had the issue with composer 1.7.3, but only with Drupal 8 projects that were started as non-composer projects, and later converted to just use composer.

The composer.json file still included this part:

        "merge-plugin": {
            "include": [
                "core/composer.json"
            ],
            "recurse": false,
            "replace": false,
            "merge-extra": false
        },

 

If you remove that, the issue with composer 1.7.3 seems to be gone.

rgpublic’s picture

...doesnt that mean core/composer.json is not respected at all anymore? Can this really be "the right thing (TM)"?

gillesv’s picture

Not 100% sure, but it's also not included in the composer installation of Drupal:
https://github.com/drupal-composer/drupal-project/blob/8.x/composer.json