How do I migrate existing D8 site into a Composer managed structure?

By ksavoie on 24 Mar 2017 at 21:10 UTC

I have a manually installed D8 site (30+ modules) that I need to get into a structure/workflow where composer manages updates. 'Composer Manager' no longer functions thus my dependencies are currently broken.
I have scoured the net looking for a method to get existing sites under composer but have had no success getting it to work properly.
Does anyone have a tutorial they can point me to have pointers to share?
Looking for any guidance. Thanks.

Comments

It is in the documentation

John_B commented 25 March 2017 at 09:14

It is in the documentation here. Because I could not remember the URL I used Google to find it: https://www.drupal.org/docs/develop/using-composer/using-composer-to-man...

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

I too am having great

Polly commented 27 March 2017 at 15:21

I too am having great difficulty understanding this. Previously I was using Composer Manager. Now I am stuck with a live D8.2.4 site that needs a security update I am unable to achieve.

Unfortunately the page John_B links to, and all other pages I have found, seem intended for more experienced developers with an overview that I lack. I have used D6/7 and now 8 to run great little sites for local community groups. Drupal has been invaluable. I have always managed even writing some custom modules. However, this has me stumped. It would be disastrous for us to not be able to continue with Drupal just because of PHP dependencies. Some guidance for the less experienced would be very much appreciated indeed.

Using Composer Manager my PREVIOUS upgrade path (for this example assume a minor Drupal8 update and new version of Address module on Debian) was:

Copy live site to localhost (/var/www/html)
Delete all files and the core folder from the html folder
Copy all files and core folder from root of the new Drupal version tar into the html folder (to replace deleted)
Delete the html/modules/address folder
Replace the address folder with the new version
In terminal run 'cd /var/www/html' and then 'composer drupal-update'
In browser run the script on 'localhost/update.php'.
FTP everything back onto live server.

Please can someone explain in easy to understand steps how to do this without Composer Manager?

Thank you very much
Polly

I think the tar balls for

John_B commented 27 March 2017 at 15:42

I think the tar balls for core still have the dependencies even the a few contrib modules do not. Am I wrong about that?

That sounds like hard work! Just ensure your server, or the machine where you run the update, has drush installed. Then log in, change directory to document root, and type
drush up
Hit 'enter'
and relax :)
I do it that way, though I don't have many D8 sites yet.
For more, search for updating Drupal with drush. Am I missing something here?

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

It seems straight forward so

ksavoie commented 27 March 2017 at 16:51

Previously I was updating with a combination of Drush and Composer Manager. Didn't have an issue with it until my last round of updates as it looks like Composer Managers repo is no longer good.

This seems straight forward so I know I am missing something silly.
I have followed the directions and 'composer update' appears operable as it looks to update Drupal's dependencies, however module dependencies seem ignored. In fact on 'composer update' a dependency one of my modules requires is removed (one that 'Composer Manager' historically updated without issue).

Questions.
The instructions state to manually edit thee composer.json file with the following

"extra": {
    "installer-paths": {
        "modules/contrib/{$name}": ["type:drupal-module"],
        "modules/custom/{$name}": ["type:drupal-custom-module"],
        "profiles/contrib/{$name}": ["type:drupal-profile"],
        "themes/contrib/{$name}": ["type:drupal-theme"],
        "themes/custom/{$name}": ["type:drupal-custom-theme"],
        "libraries/{$name}": ["type:drupal-library"],
        "drush/{$name}": ["type:drupal-drush"]
    }
}

Q1) Since the existing .json file already has a 'extra' section, do I append that, or create a new one (does it make a difference)?

Q2) My module,profile etc. paths do not match this format so I updated so they are the same. However I am not positive my path edits are accurate as it looks like a fresh Composer installed Drupal directory structure is quite different than what I am used to. My pathing is flat. At the root of the Drupal install, /modules, /profiles, etc... (no contrib/custom sub-directories). Are these the paths this section is referencing?

Any other insight appreciated!

Generally AFAIK (not very far

John_B commented 27 March 2017 at 17:21

Generally AFAIK (not very far!) you should not have a duplicate key in composer.json file. Therefore the idea seems to be add these paths to the "extra" section.

If you are not using the contrib/ and custom/ subdirectories on you site, leave them out of the composer file.

Since the composer manager repo is no longer good, be sure also to update the repo by updating composer using the step here.

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

Hello John

Polly commented 27 March 2017 at 17:29

Hello John

Thank you for your response. No, you are not the one missing anything - I have been stuck in my ways not using Drush. I am now trying to embrace it. Unfortunately I am having problems installing Drush. I will paste my Drush problems below hoping you might know the problem. If not I will resolve the Drush install problem and return here once I have tried your solution to my upgrade path.

I followed the 'Install a site-local Drush' here: http://docs.drush.org/en/master/install/

polly@PollysPC:/var/www/html$ composer require drush/drush
The "https://packagist.drupal-composer.org/packages.json" file could not be downloaded: Peer certificate CN=`*.github.com' did not match expected CN=`packagist.drupal-composer.org'
Failed to enable crypto
failed to open stream: operation failed
https://packagist.drupal-composer.org could not be fully loaded, package information was loaded from the local cache and may be out of date
Using version ^8.1 for drush/drush
./composer.json has been updated
Loading composer repositories with package information
The "https://packagist.drupal-composer.org/packages.json" file could not be downloaded: Peer certificate CN=`*.github.com' did not match expected CN=`packagist.drupal-composer.org'
Failed to enable crypto
failed to open stream: operation failed
https://packagist.drupal-composer.org could not be fully loaded, package information was loaded from the local cache and may be out of date
Updating dependencies (including require-dev)
Package operations: 12 installs, 0 updates, 0 removals
  - Installing webmozart/path-util (2.3.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/var-dumper (v2.8.17): Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/finder (v2.8.18): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing jakub-onderka/php-console-color (0.1): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing jakub-onderka/php-console-highlighter (v0.3.2): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing dnoegel/php-xdg-base-dir (0.1): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing nikic/php-parser (v3.0.5): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing psy/psysh (v0.8.3): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing pear/console_table (v1.3.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing consolidation/output-formatters (1.1.0): Downloading (connecting.Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing consolidation/annotated-command (1.4.1): Downloading (connecting.Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing drush/drush (8.1.5): Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
symfony/var-dumper suggests installing ext-symfony_debug ()
psy/psysh suggests installing ext-pdo-sqlite (The doc command requires SQLite to work.)
psy/psysh suggests installing hoa/console (A pure PHP readline implementation. You'll want this if your PHP install doesn't already support readline or libedit.)
pear/console_table suggests installing pear/Console_Color2 (>=0.1.2)
drush/drush suggests installing drush/config-extra (Provides configuration workflow commands, such as config-merge.)
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
polly@PollysPC:/var/www/html$ drush status
bash: drush: command not found

Hello John

Polly commented 27 March 2017 at 18:07

Hello John

I installed Drush following the manual upgrade here: https://www.drupal.org/node/2603018.

Unfortunately I got an error on the drush up cmd. I am unable to find a solution. Do you have any suggestion?

root@PollysPC:/var/www/html# drush up
Update information last refreshed: 27 Mar 2017
 Name                             Installed       Proposed version  Message          
                                  Version                                            
 Drupal                           8.2.4           8.2.7             SECURITY UPDATE  
                                                                    available        
 Address (address)                8.x-1.0-rc3     8.x-1.0-rc4       Update available 
 Comment Notify (comment_notify)  8.x-1.x-dev     8.x-1.0-beta1     Update available 
 Geolocation (geolocation)        8.x-1.8         8.x-1.10          Update available 
 Honeypot (honeypot)              8.x-1.23        8.x-1.24          Update available 
 Pathauto (pathauto)              8.x-1.0-beta1   8.x-1.0-rc1       Update available 
 Redirect (redirect)              8.x-1.0-alpha1  8.x-1.0-alpha4    Update available 
 Token (token)                    8.x-1.0-beta2   8.x-1.0-rc1       Update available 
 Video Embed Field                8.x-1.3         8.x-1.4           Update available 
 (video_embed_field)                                                                 
 Bootstrap (bootstrap)            8.x-3.1         8.x-3.2           Update available 


NOTE: A security update for the Drupal core is available.
Drupal core will be updated after all of the non-core projects are updated.

Code updates will be made to the following projects: Address [address-8.x-1.0-rc4], Comment Notify [comment_notify-8.x-1.0-beta1], Geolocation Field [geolocation-8.x-1.10], Honeypot [honeypot-8.x-1.24], Pathauto [pathauto-8.x-1.0-rc1], Redirect [redirect-8.x-1.0-alpha4], Token [token-8.x-1.0-rc1], Video Embed Field [video_embed_field-8.x-1.4], Bootstrap [bootstrap-8.x-3.2]

Note: A backup of your project will be stored to backups directory if it is not managed by a supported version control system.
Note: If you have made any modifications to any file that belongs to one of these projects, you will have to migrate those modifications after updating.
Do you really want to continue with the update process? (y/n): y
Project address was updated successfully. Installed version is now 8.x-1.0-rc4.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/address.
Project comment_notify was updated successfully. Installed version is now 8.x-1.0-beta1.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/comment_notify.
Project geolocation was updated successfully. Installed version is now 8.x-1.10.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/geolocation.
Project honeypot was updated successfully. Installed version is now 8.x-1.24.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/honeypot.
Project pathauto was updated successfully. Installed version is now 8.x-1.0-rc1.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/pathauto.
Project redirect was updated successfully. Installed version is now 8.x-1.0-alpha4.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/redirect.
Project token was updated successfully. Installed version is now 8.x-1.0-rc1.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/token.
Project video_embed_field was updated successfully. Installed version is now 8.x-1.4.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/modules/video_embed_field.
Project bootstrap was updated successfully. Installed version is now 8.x-3.2.
Backups were saved into the directory                                        [ok]
/root/drush-backups/drupal/20170327175049/themes/bootstrap.

Code updates will be made to drupal core.
WARNING:  Updating core will discard any modifications made to Drupal core files, most noteworthy among these are .htaccess and robots.txt.  If you have made any modifications to these files, please back them up before updating so that you can re-create your modifications in the updated version of the file.
Note: Updating core can potentially break your site. It is NOT recommended to update production sites without prior testing.

Do you really want to continue? (y/n): y
Unable to load class \Drush\VersionControl\Backup                            [error]
Unable to load class \Drush\VersionControl\Backup                            [error]
PHP Fatal error:  Class 'Symfony\Component\HttpFoundation\Response' not found in /var/www/html/vendor/drush/drush/lib/Drush/Boot/DrupalBoot8.php on line 200
Drush command terminated abnormally due to an unrecoverable error.           [error]
Error: Class 'Symfony\Component\HttpFoundation\Response' not found in
/var/www/html/vendor/drush/drush/lib/Drush/Boot/DrupalBoot8.php, line 200
root@PollysPC:/var/www/html#

The directory structure had changed. All the root files and core folder were now in 'html/drupal-8.2.7'.

It is a complete pain. Drupal

John_B commented 27 March 2017 at 18:19

It is a complete pain. Drupal can be like that, and WordPress rarely is! Personally I would try removing drush then installing drush globally, as per the last two posts here.

Or install Drpual Console (drush alternative) and use command
drupal update:execute

Ultimately I regard having a working drush (and / or Drupal Console) install as the first step, because without that everything else is so difficult.

I do not really know your workflow. I think you said you are copying the site to a local computer for the update, then putting it back on the server. If so, perhaps use a local environment with a working drush and Drupal console already installed, such as a virtualbox set up using Drupal VM (drupalvm.com).

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

Thank you John. I have

Polly commented 27 March 2017 at 19:54

Thank you John. I have started down the road of following your suggestions but have run out of time right now. I will try again over the next few days and let you know how I get on.

For information: it appears

Polly commented 1 April 2017 at 19:37

For information: it appears Drush should not be used if any commerce modules are used. See https://www.drupal.org/node/2865986

I've done it! I put this here

Polly commented 2 April 2017 at 16:47

I've done it! I put this here in case this helps anyone else.

I had a D8.24 site created manually from tar with dependencies managed using Composer Manager. These are the steps I took to upgrade to D8.27 (now that Composer Manager is deprecated). I am not able to say if this is the best way bu it seems to have worked for me.

-Uninstall Composer Manager via Drupal interface (& delete folder)
-Delete all files and core folder in Drupal root (in my case '/var/www/html')
-Copy all files and core folder from 8.27 tar root into Drupal root (to replace deleted)
-In terminal go to Drupal root ('cd /var/www/html')
-In terminal run 'composer config repositories.drupal composer https://packages.drupal.org/8'
-In terminal run 'composer require drupal/name_of_library' for every contributed module & theme (eg 'composer require drupal/address', 'composer require drupal/ctools' etc)
-In terminal run 'composer update'
-Run /update.php via browser

well done :-)

John_B commented 2 April 2017 at 16:47

well done :-)

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

Thanks for your help John_B

Polly commented 2 April 2017 at 16:48

Thanks for your help John_B

Just in case anyone needs

Polly commented 4 April 2017 at 16:42

Just in case anyone needs this: ksavoie has a better way to do this. See his comment below: https://www.drupal.org/node/2863882#comment-12022504

When retrofitting a D8 site

ksavoie commented 28 March 2017 at 13:36

When retrofitting a D8 site under Composer with several already installed modules, should composer automatically know these modules exist and pick them up for dependency updates? Or do I need to reinstall them via composer. I ask because this is where I am now stuck, Drupal dependencies are updated with a 'composer update' but module ones do not and are actually removed when the update is run.

Look at this from the same

John_B commented 28 March 2017 at 16:21

Look at this from the same documentation page.

My understanding is that the first time after migrating you need to issue the command

composer require drupal/module_name

This will add the contributed module to the composer.json, and later updates should upate that module too. Alternatively you could have a half-way house, where you have not run the command

drupal-composer/drupal-project

in which case you will update those modules using your usual method, drush or whatever, which I would think would get complicated.

(For the sake of avoid confusion I assume you are using the composer drupal/drupal option and not drupal-composer/drupal-project option, also described on the documentation page, which places the composer.json above document root.)

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

I guess this is my question.

ksavoie commented 28 March 2017 at 19:03

I guess this is my question. Their isn't a way for composer to review the currently installed modules and dynamically update the .json file with the appropriate required packages.
Basically I have to manually require every module for composer to pick them up.

That is my understanding,

John_B commented 28 March 2017 at 19:10

That is my understanding, with the proviso that although I have used Composer I am not expert in it. My understanding is that either you issue the command >composer require and let composer write the composer.json file, or you manually add your requirements to composer.json.

Now it is fair to say that this process could be scripted, and that the drupal-composer project has a drush command (which I have not tried) which says it does automate that process according to the README.txt file on github. My inclination is that if the site is not huge it would be easier to it manually, at least until one has more experience and more confidence. If it is a lot of work owing to the size of the site, or if you have a lot of sites to move to composer, then it would start to make sense to look into automating the process.

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

After working on this issue

ksavoie commented 4 April 2017 at 15:50

After working on this issue for some time I have come to a solution that appears to work for me and is actually quite straight forward.
[ f.y.i. I make no warranties : ) ]
First I uninstalled/deleted the 'Composer Manager' module.
I replaced my composer.json file with the contents of the latest Drupal download. I then ran the following composer command to add in Drupal's composer repository to composer.json.

composer config repositories.drupal composer https://packages.drupal.org/8

I manually added the paths to my directory structure to the 'extra' section of the composer.json file.

"installer-paths": {
            "core": ["type:drupal-core"],
            "modules/{$name}": ["type:drupal-module"],
            "profiles/{$name}": ["type:drupal-profile"],
            "themes/{$name}": ["type:drupal-theme"],
            "libraries/{$name}": ["type:drupal-library"],
            "drush/{$name}": ["type:drupal-drush"]
        }

I manually added all my modules to the 'require' section of the composer.json file !carefully making sure the version numbering I indicated is what I wanted!. Note: correct version numbering can be a little confusing especially if you are using dev versions of modules.
e.g.

"require": {
        "composer/installers": "^1.2",
        "drupal/core": "^8.2",
        "wikimedia/composer-merge-plugin": "~1.3",
        "drupal/admin_toolbar": "~1.0",
        "drupal/contact_block": "~1.0",
        "drupal/contact_storage": "~1.0",
        "drupal/ctools": "~3.0",
ETC..........

NOTE: In my case I removed

"replace": {
        "drupal/core": "~8.2"
    },

from composer.json and put ( "drupal/core": "^8.2", ) into the 'require' section.

This should complete the necessary adjustments to the composer.json. You should now be able to clear Composers cache 'composer clear-cache', and then update 'composer update' to get the files up to date.
After that you will need to make sure database updates are applied 'Drush updb'.

Once this is done you should be able to manage the site similarly to how you would any other.
Good Luck!

Thanks for putting this here.

Polly commented 4 April 2017 at 16:18

Thanks for putting this here. I took a slightly different (and longer!) path (explained above) which I think got me to a similar place with one obvious difference. I wonder if it may be important. Perhaps you or someone else may comment.

installer-paths
I did not add this section to my composer.json. I think it means my dependency libraries are put in the root 'vendor' folder (rather than sub-folders within the named folders). Is that correct?

I now see that the documentation says "this doesn't jibe with Drupal modules, themes, profiles, and libraries" but I don't understand what this means.

What are the implications of not including the installer-paths section? My site seems to be working fine but do I need to correct this?

So I added the 'installer

Polly commented 4 April 2017 at 16:35

So I added the 'installer-paths' section and moved the "drupal/core": "~8.2" line into require. I then ran 'composer update'.

polly@PollysPC:/var/www/html$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing drupal/core (8.2.7): Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

It seems it downloaded D8.2.7 (which I was already on) but did nothing else. My root 'vendor' folder was not changed - it remained with the same number of items and MB size.

I don't understand. Perhaps now composer will change the location of dependencies when it next updates them?

Yeah, originally I was

ksavoie commented 4 April 2017 at 16:33

Yeah, originally I was working a similar path to your 'longer' one but after I got my head around what was going on I believe this simpler method gets us to the same ending.
To be honest I am not positive that including installer paths is required if your respective directories are all at the root of your install. They may be required if you are using a alternate pathing or are using packaging with a git push/pull update workflow. Someone else may be able to clarify this.
If I get a second I may try excluding the installer paths in my .json on my DEV box and see what happens.

Are you still liking the method you posted?

Rick Hood commented 24 May 2017 at 13:25

@ksavoie
I've been looking for the same thing: how to convert a D8 site to composer-managed. Are you still liking the method you posted? Seems good to me, just checking as things change fast. ;-)
Also, I am unclear about how to tell whether or not a D8 site I got handed was setup to be composer-managed or not. Do you know the "best" way to determine that?
Thanks!

UPDATE:
Found answer to how to see if site is composer-managed: run drush up and you get a warning.

drush up                                                                                                                                     
This codebase is assembled with Composer instead of Drush. Use `composer update` and `composer require` instead of `drush pm-updatecode`  [error]
and `drush pm-download`. You may override this error by using the --pm-force option.

One other question - Is all

adminMN2023 commented 24 May 2017 at 15:07

One other question - Is all this do-able on a localHost site? (retrofitting an existing D8 site to one managed by Drupal - done in a dev environment)

I have my first Drupal site (D8) in development, 90% done, and now need to retro-fit something that is *seemingly* mandatory. (I've updated my site successfully 2 or 3 times using cPanel's file manager)

Documentation aimed at non-programmers would be a blessing as people continue to evaluate Drupal 8 as a viable alternative to other CMS's.

Probably better do it locally

John_B commented 24 May 2017 at 14:50

Probably better do it locally. In my experience 'composer require' won't work on shared hosting anyway (see blog post).

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

It would be local - then

adminMN2023 commented 24 May 2017 at 15:15

It would be local - then uploaded to a VPS. Which seems do-able according to the blog. The problem now being - Does a finicky Composer (on a VPS) add a new layer of complexity to D8? (Further widening the unintended gulf of adoption by non-programmer types)

In your case, Composer would

ksavoie commented 24 May 2017 at 16:01

In your case, Composer would/should not exist on your VPS. If you don't have CLI access it's moot anyway. It is exclusively used as a package management tool for the install and versioning of Core, Modules, and their respective dependencies. It is a tool used independent of Drupal. Drupal does not need it to operate. It took me a bit to get my head initially around the concepts. While you can use Composer commands to perform installs, for me I found it best to understand how to manually edit the composer.json format. It lets you have very good control over versioning.
If however you had ROOT/CLI access to your Production server and could install Composer, then their may be a case for running it their.
As for Drush, Drupal now recommends each Drupal site run their own install of Drush. Once you are set and running, Drush is relegated to usually only clearing cache and updating the DB once Composer has installed any new Core/Module updates.

Thanks for the feedback! Had

adminMN2023 commented 24 May 2017 at 16:30

Thanks for the feedback! Had no idea Composer ran separately - I thought it was all tied in together.

I do not have the experience to even attempt the root/CLI access thing. So that is great to know.

While I had/have no intention of "hijacking" this thread (apologize if that is the way it seems) - this is all really important information that non-programmer types need to know.

For clarification:

1 - Install Composer on local computer
2 - Install Drush on local computer
3 - Pull site down from off-site production server to local dev environment

Drush - similar to shell access - can reach anywhere including my off-site production server
Composer - stays local to do updates - the resultant updated site gets put back on production server

Question 1: Is this correct thought process?
Question 2: Are Drush and Composer two roads to the same destination?

Thanks!

Drush can download and

John_B commented 24 May 2017 at 16:42

Drush can download and install a site as well as doing many other things. Compser is only for installing and updating software. Drupal Console is a competitor to Drush for Drupal 8 only. Drupal Console removed the install and update features because it was though better to leave that to Composer. AFAIK these things are still possible with Drush but not recommended. To be more precise, installing and updating other than with Composer is not the recommended method for and by people who mainly work in teams on large 'enterprise' sites.

Although I have read about a Composer graphical interface, these are all command line tools.

For a small site, and for shared hosting, I think the question of whether Composer solves more problems that it causes is still open. Personally I am not persuaded that it is anything like 'compulsory.' I have not recently tried installing Drupal 8 without it, so not sure.

There are many things which are easier on command line, and it is always worth learning a few drush commands when working with Drupal. To a great extent the things Drupal Console can do overlaps with what drush can do.

I should add that many shared hosts and VPS providers will provide command line / shell access, even on services where they will ot provide root access. Sometimes you have ask for it.

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

"I have not recently tried

adminMN2023 commented 24 May 2017 at 16:51

"I have not recently tried installing Drupal 8 without it, so not sure."

I started a D8 project in November and have since upgraded it twice without any mention of composer. If I could existed without it - I would. But at some point it seems like it will be mandatory?

I doubt it will be mandatory.

John_B commented 24 May 2017 at 17:30

I doubt it will be mandatory. But contributed modules which sometimes use Composer to install dependencies will be more complex to install without it. Same with updates.

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

Mandatory for an initial

ksavoie commented 24 May 2017 at 19:03

Mandatory for an initial install of Core, as of now no. However there are modules (Search API) where Composer is required (they don't work) due to dependencies. My site has 15+ modules and probably 4-5 have dependencies. For today's Drupal, Composer is pretty much necessary. Trust me, once you have your DEV environment set up and running with Composer maintenance is quite painless. Just the initial hump :).
As for Drush, historically Drush was standalone and did much of what Composer does now. But with D8s new framework (Symphony), it's dependencies, and an existing tool (Composer) already designed specifically to manage such an environment, it makes sense that most of Drush's functions have been depreciated. Composer can manage code but doesn't know/touch specifics about Drupal. That is where Drush still finds it's purpose, to manage DB schema/data updates and manually clearing caches plus a few other non-critical functions.

Yes you can spin up a site without it, but you'll eventually find it difficult to maintain once you get into much complexity. Not hard retrofitting Composer into an existing install but if you follow the instructions to get a base site initiated/installed via Composer you will bypass the few issues involved in backtracking.

Once set up, only 2 commands to update a D8 site with composer (if Drush is installed in the site).

>  composer update --no-dev -v
>  vendor/bin/drush updb

I might add that if you are

John_B commented 24 May 2017 at 19:04

I might add that if you are reluctant about Composer (which I can understand), and not yet used to working on command line (which I highly recommend learning gradually), chances are that Drupal 8 is not the best tool for the site. Backdrop, for all its limitations, is a fork of Drupal 7 which has Views in core as far as I know, and could be a good option for situations where WordPress or Concrete 5 or whatever does not cut it.

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

And I'll add to John_B's

ksavoie commented 24 May 2017 at 19:22

And I'll add to John_B's thought : ), I work and have worked in several Open Source and Commercial CMSs. Drupal (especially D8) is a very solid/secure platform. However it is not completely the GUI plug-n-play development/maintenance platform that are offered by its competitors. You need to do work outside of the GUI and simple FTPing of files and understand a few technicalities in order to fully maximize what it can and should do to minimize the headaches/frustration that occur for not understanding such. I hate to say it but for all that is awesome about Drupal, D8 widened the cost of entry for Jr. developers.

Hah... that bridge has been

adminMN2023 commented 24 May 2017 at 21:35

Hah... that bridge has been crossed! I understand what you are saying. I see a lot of cool things in Drupal, and thought D8 was more mature than it was. It's sort of like the wild west though - lots of promise, with lots of hidden danger! :^)

@ksavoie - agree with the cost of entry - wholeheartedly.

Maturity is not the correct

ksavoie commented 24 May 2017 at 22:07

Maturity is not the correct analogy. It's more of a difference in mentality and target of the platform. Drupal is quite mature and in fact more mature technically than many. The D8 rewrite into its new framework was a crazy undertaking that formalized and straightened out a whole slew of development issues that plagued it's predecessor and are present in other platforms. Symphony offers a solid base to build upon. I understand a developers hesitation and initial feeling of irritability in working with Drupal as I had the same but don't get caught up in "you can't do it in the GUI so it must be bad" mindset. Yes there are things that take longer to develop than other platforms. But as you get to know the system better you will start having moments of understanding of the amazing inherent flexibility that allows you to get complex pieces to work and look exactly how you want without relying on a module or plugin (if it exist). It's not for everyone and definitely not for every project but as you work with it you will start being able to determine which projects are a great fit for it.

By "mature" I meant (re: D8)

adminMN2023 commented 25 May 2017 at 12:54

By "mature" I meant (re: D8) 'all the kinks ironed out to a much larger degree'. There is no doubt about the maturity of the platform as a whole. Honestly - as I've dealt with Wordpress and Joomla - it's always been "those guys over there" that develop it. From a non-programmers standpoint, with Drupal and the Drupal community - it's really pretty intimidating when "those guys over there" are literally all around you! I've gotten this far - not stopping now!

It's not for everyone and definitely not for every project but as you work with it you will start being able to determine which projects are a great fit for it.

Once I understood Exposed Filters - I added a whole new section to this website in development that wasn't even on the chalkboard for the web site. Basically a parts catalog that can narrow a list down based on check-boxes (Material, Machine, etc).

But then multiple exposed filters did not work (and don't know if it's fixed yet in the latest D8 version) - I soon figured out I can add an additional taxonomy as a work-around.

Anyway - Next task is figuring out Composer!

Composer security concerns

cmah commented 28 June 2017 at 22:08

I'm also sorry to hijack this thread, but it's the most useful one I've found so far.

First I would like to thank all of you for showing me how to change mid-stream from using drush to using composer in order manage contrib and core updates. There is nowhere else I've found that documents this (least of all Using Composer to manage Drupal site dependencies, which of all places you would think would actually document this).

I'm confused about the yes/no dilemma about using Composer in production. Do you really need composer in production? makes a really good point. The use cases this author points out are really similar to ours. After all, a Drupal website is a "living thing" indeed, and "deployment" and "distribution" are not the same thing at all.

The other issue is security. I would rather not run composer update on the production server, because then my production workflow would be to log in as root and temporarily turn on allow_url_fopen and proc_open before doing my composer update command (and then remembering to turn them back off).

So what does this mean? I think it means that my old workflow, of committing only the modules and theme directories to our git repo, has to be augmented by also committing all of the core and vendor directories as well, since switching from the old "delete the core and vendor directories and copy the new ones from the tarball" way of upgrading core is deprecated, and I'm now supposed to only use Composer to manage core updates.

This is a little weird and depressing, because in the old Drupal 7, pre-Composer world, our git repo was never bloated with all the core stuff. However, security is very important, and I'm extremely unwilling to compromise the hardening we've done in our php.ini file.

Which is better? Editing the php.ini every time, or committing the vendor and core directories to my repo?

The prevailing Drupal attitude toward allow_url_fopen and proc_openseems to be just laughing off the concerns, but having friends who work in cybersecurity, I have completely the opposite attitude.

PS - diff between composer update and composer install

cmah commented 5 July 2017 at 19:15

https://adamcod.es/2013/03/07/composer-install-vs-composer-update.html

PS -- the above seems to indicate that if you do decide to use composer on your production server, you should only add your composer.lock and not your composer.json to your git repo, and then pull that and do a composer install rather than a composer update. Is this really true? Any additional pointers would be appreciated.

While for a production deploy

Melvin Loos commented 6 August 2017 at 19:44

While for a production deploy one would only really need the composer.lock since this file contains the exact versions (in other words on what it is locked on) however the composer.json has a less restricted versioning and follows patterns like higher then (>) version 1.0 or lower then (<) 2.3 for example. Which is useful for the development fase where exact versions are not that important but backwards-incompatibility changes can still be avoided. For this composer strictly handles a versioning protocol called semver (see http://semver.org). This versioning protocol works with a 3 number notation, MAJOR.MINOR.PATCH. To answer your question, YES, always commited both composer.lock and composer.json. Composer install only installs the exact versions from the composer.lock if present, if not present it will do the same as composer update and update everything to the latest version allowed by the composer.json restrictions.

what a mess

groovedork commented 4 November 2017 at 11:45

I've spent two days trying to wrap my head around drush, composer, and just the sheer amount of "oh but you also have to do this, didn't you know?".

At this point I think I may just stop at Drupal 8.3 and hope I don't get hacked. Then get back into the game when a WYSIWYG option appears (if ever..) that works on shared hosting (unlikely).

It feels like all empathy has drained form Drupal. I always considered "you can do it through WYSIWYG" to be a core pilar of what the Drupal community believed in. It's what got me involved. The easy with which that pillar has been removed is astounding.

I hear you

Rick Hood commented 4 November 2017 at 12:42

You said "I may just stop at Drupal 8.3". Does that mean you have a Drupal 8.3 site that you are trying to update to 8.4? If so I might be able to post something to help, but would need to know: what exact version are you on now (what is "x"): 8.3.x ? Did you originally install it with composer, drush, or just download it as a tarball (like "the old days")? And do you have a web folder that index.php is located in, or is index.php in the same folder as the core folder (e.g. there is no /web folder)?

Curious what this means: "a WYSIWYG option appears (if ever..) that works on shared hosting". Is your 8.3 WYSIWYG not working? At any rate, this is not the case IMHO: "I always considered 'you can do it through WYSIWYG' to be a core pillar of what the Drupal community believed in." Quite the opposite actually. One should separate content into separate fields, not put everything into a WYSIWYG.

IMHO Drupal 8 is by far the best version of Drupal ever, mainly because of the build on top of symfony and using twig. BUT it has definitely become more challenging for developers to use in some areas (composer being one of them). Drupal is probably not be the right tool for simple sites anymore. I work with Drupal all day long, and love it, but have moved simple Drupal sites I did for my freelance clients to Squarespace. Honestly if your site(s) is simple, I would move to that, particularly because you don't have to worry about updates. Drupal has move more toward "enterprise" while SAS options like Squarespace and Wix have become better for simple sites.

PS: very good articles about composer here:

February 13, 2017
Tips for Managing Drupal 8 projects with Composer
https://www.jeffgeerling.com/blog/2017/tips-managing-drupal-8-projects-c...

May 3, 2017
Composer and Drupal are still strange bedfellows
https://www.jeffgeerling.com/blog/2017/composer-and-drupal-are-still-str...

This is also very good:
Using Composer to manage Drupal site dependencies
https://www.drupal.org/docs/develop/using-composer/using-composer-to-man...

It should have been called drupal 9

groovedork commented 6 November 2017 at 07:50

I suspect Drupal 8.4 should have been called Drupal 9, simply because there is no easy upgrade path.

The site in question is an 8.1 distribution that has been upgraded to 8.3.5 over time. It was not built with Composer or Drush, but with Dev Desktop.

I tried..
- Upgrading via file overwrite. No, doesn't work anymore. Ok then..
- Upgrading via Drush in DevDesktop.. ah but then it won't work anymore once the site is 8.4.
- Loads of warnings, do they matter? bootstrap.inc not found for modules I haven't even heard of (LINT or something), and can't find in my code.
- Upgrading via Drush (need Drush 8.1.12/8.1.15.. ok. There's also Drush 9, but that won't work for 8.3.5 sites (which I have more of).
- Instlaling drush 8.1.15 needed a lot of research: how do you do that? There is surprisingly little guidance online. Only for Drush 9.
- Drush 8.1.15 upgraded.. gives other errors.
- Drush gives errors if the drush is in the code you are trying to upgrade. The error being that the entire site is deleted.. great, start again.
- Try to switch the DevDesktop version of Drush manually. Is this ok? No idea. Getting a backup error.
- Composer then.. installing that was hard until I finally discovered Drush has a command for that. But do you install it locally or globally? What does that mean exactly, and where are things stored? How does it interact with other local sites?
- Memory is too small for composer.. start again.. ok.
- Require? Override? You need to change this for it to maybe work.. can I do that by hand?
- Drush up, Drush up drupal, composer update, composer etc --with dependencies.. what are the differences exactly?
- Ah so it matters what directory things are in. Ideally, put the site in a sub directory.
- Ah, so I can't composer upgrade a site that wasn't built with composer.. right.
- There's a Drush option to 'discover' all the modules, but it's not recommended. And would it work with the rather complex distribution anyway?
- Maybe I can make a new site and move things over via the new settings configuration files that I had such high hopes for. Nope. If you want to copy the entire site over, then the UUID is put into every file. Why? Who knows.
- Ah, there are three ways of starting a new Drupal 8 site via composer. But which framework is best? Sparse documentation that as ever implies you already feel totally comfortable around the commandline and understand all the implied things about directory structures. If I use a framework with a sub directory, will that mean that my site will now live at bla.com/web ? Who knows.

Etc etc, forgetting loads.

These are not questions I'm looking for answers to, I just wanted to give an indication of the complexity that is the "simply commandline option" that is being forced on everyone.

It works OK if you build the

John_B commented 6 November 2017 at 08:11

It works OK if you build the site with composer. Use option A on https://www.drupal.org/node/2718229 drupal-composer/drupal-project , it is the most widely accepted and I guess it may become official. Run composer update from project directory which with that method is one level above document root. Then use drush (or Drupal console if you prefer) to run drush updb to update the database.

Be careful with composer update unless the site is under version control: unlike drush up, there is no backup of code made before the new code is copied in.

True, it will be a pain if the site was not built with composer.

Dries told Vienna DrupalCon that he is personally offended when it is said Drupal has abandoned small users (without seriously arguing it was untrue, as far as I could hear), so I will not repeat the allegation. Other tools for other times...

Digit Professionals specialising in Drupal, WordPress & CiviCRM support for publishers in non-profit and related sectors

How to port from drush -> composer

BartK commented 15 March 2018 at 13:59

If you're here looking for a way to migrate your modules from drush to composer without having to manually require every single module, try this from your drupal install directory, AND FOR THE LOVE OF ALL THAT IS HOLY, BACK UP YOUR CODE AND DATABASE FIRST:

ls -1 modules/contrib/ | xargs -L 1 -I {} composer require drupal/{}

If you have a lot of modules, it will take a while to run.