Provide optional composer integration but don't force users to understand how to use composer

#2845378: Stop offering .tar.gz downloads

it should always be possible to download files from d.o, put them in the correct location (after extracting the archive) and have it work.

We can wish for this all we want, but it's technologically impossible due to the way Composer is built.

Mixologic and I discussed this, and agreed that the right solution is to offer a rewritten Update Manager that embeds Composer (basically shipping with updater/ next to core/). That way people either use Composer, or they use the UI which runs Composer and then moves the changes into place. We also get core updates for free.
#2538090: Allow the Update Manager to automatically resolve Composer dependencies.

That way people either use Composer, or they use the UI which runs Composer and then moves the changes into place.

The problem here is that it still requires composer. Most shared hosting servers do not have it installed by default, so to add composer requires knowledge of how to install it - and we're back to the command line. So while having GUI access is definitely a step in the right direction, users who cannot use the command line are still cut off.

@Jaypan: did you read the part of comment #3 about embedding composer?

No, I missed that. That's a good solution. It actually covers the biggest issue with composer as I see. That would also be nice since non-technical users would then be able to install drush more easily as well.

Agreed that we don't want to force anyone to have to use the shell to deploy/maintain Drupal.

Also agreed that the plan in #3 is the right way to rely on Composer to solve the problems it does without forcing end users to know it's even happening.

Part of how authorize.php was originally designed was to let it become a more generic framework for running something like composer as the user that owns the files in the filesystem. I'm not at all attached to keeping authorize.php as-is, but I definitely support any effort that allows it or something like it to run composer for site admins.

The installer could also use something like authorize.php to invoke Composer to bootstrap contrib installs during initial setup of a new site. There's talk of a module browser in core, etc. All of this could be UI around ways to get "update manager" or "install manager" or whatever we call it, to feed composer commands to the filesystem via authorize.php or its offspring.

Cheers,
-Derek

Given #3, shouldn't this be closed as a duplicate of #2538090: Allow the Update Manager to automatically resolve Composer dependencies?

Maybe. But #2538090 needs an issue summary update because #38 explains it is probably untenable as written.

Escalating, based on impact.

For background the lead up to this issue is at http://buytaert.net/making-drupal-upgrades-easy-forever#comment-131086

There's a couple of constraints that we want to consider when designing a system that will allow us to abstract away the complexities of using composer:

Composer is designed to be a build time/development tool, and is not intended to exist or be available in production systems. What that means in practice is that the normal constraints of "code outside the firewall and open to the internet" are not baked into composer, and thus it could be considered a security risk to have it available on a production system that interfaces with threats. Thats not to say it cannot be on live sites, just that upstream, the developers are not likely to focus on making it hard to compromise. Even if we were to audit it now and find that its fine, that doesn't mean that the next version wont have some obvious remote code execution vulnerability baked into it.

The second aspect of composer is that it is designed to be executed in a command line php environment - which means it is expecting to have the ability to consume considerable amounts of memory, and take significantly longer execution time than most webserver php environments are geared for. Drupal 8's 64MB minimum requirement would probably need to bump to 256MB or higher, and we might need as much as 180 seconds worth of timeout for php. Or more.

I bring up these two points to underscore the difficulty we may encounter in attempting to bundle composer within the drupal application itself. There are likely ways we can cordon it off for security purposes, and ways we can ensure its only running on a cron or something similar. Another alternative is to construct a site management tool that is separate from your web app, much like wordpress's desktop administration application (https://developer.wordpress.com/calypso/).

Composer isn't the only sitebuilding barrier, and we ought to address the "Constructing, Extending, Updating, and Maintaining" the codebase your website needs to run in a holistic manner - so that we can handle any other "Buildtime/Development tools"- i.e. we can add build tools like NPM or bower to solve the 'gather front end assets too' so as to provide a seamless, delightful experience for drupal users who do not fall into the classification of "working on enterprise sites for enterprise clients"

I also strongly believe that we should start by designing the ideal user experience we would like to see in a tool like this, and aim high enough to knock it out of the park.

I bring up these two points to underscore the difficulty we may encounter in attempting to bundle composer within the drupal application itself. There are likely ways we can cordon it off for security purposes, and ways we can ensure its only running on a cron or something similar. Another alternative is to construct a site management tool that is separate from your web app, much like wordpress's desktop administration application (https://developer.wordpress.com/calypso/).

I agree with @Mixologic and matt glaman and I already started work on a tool that does this. I started by building https://github.com/borisson/composer-ui and a couple of weeks later matt built https://github.com/mglaman/conductor. The goal I had in mind (and that matt largely followed) is to make a tool to install composer dependencies for any composer-based project (including drupal, but also symfony, laravel and others). The biggest thing still to do there is to add the ability to add/remove projects. Currently it only does install and update.

I think if we promote and improve that tool we'd get pretty close to what calypso can do. Adding the ability to use npm dependency resolution would be awesome, but I think that'd be a lot hard to get right. That's why I'd leave that for a 2.x release.

We had a BOF about this at drupalcon dublin but only a couple of people showed up, probably because I named it "composer UI BOF" and not something like: "Installing modules from the desktop", I can't really remember a lot of things from that BOF, I forgot to take notes, but maybe klausi remembers more. I created a couple of issues in the queue of composer-ui as a result of that discussion:
- https://github.com/borisson/composer-ui/issues/12
- https://github.com/borisson/composer-ui/issues/11
- https://github.com/borisson/composer-ui/issues/4

For my part, I'm not against having a UI to soften the steep Composer learning curve. In fact, I think it's a great idea -- as long as it's not Drupal-specific. When it comes to code base and update management, what I'm in favor of is dropping as many Drupalisms as possible. We should be taking advantage of everything Composer can do for us, and fill in the gaps (i.e., a UI) where it could be made easier and friendlier for Drupal users. And if our good work can move upstream, good.

The second aspect of composer is that it is designed to be executed in a command line php environment - which means it is expecting to have the ability to consume considerable amounts of memory, and take significantly longer execution time than most webserver php environments are geared for.

If we built a stand-alone Composer UI, maybe we could bundle it with a preconfigured copy of the PHP interpreter with higher exec time and memory limits, to sorta work around this problem.

What that means in practice is that the normal constraints of "code outside the firewall and open to the internet" are not baked into composer, and thus it could be considered a security risk to have it available on a production system that interfaces with threats.

Please correct me if I'm misunderstanding, but doesn't this just mean that it's potentially dangerous to keep Composer in your production environment? If so, how does that differ from the situation now, supporting modules that have Composer dependencies?

it's potentially dangerous to keep Composer, the software that you use to resolve your dependencies, in your production environment in such a manner that your production environment can actually execute it.

In other words 'composer dependencies' themselves are not potentially dangerous, but having "composer/composer" itself, as a dependency, is potentially dangerous.

As far as the updating and codebase management aspect, this is a situation where we want a Drupalism. Composer/NPM etc are the 'industry standard best practices'' that we would adopt, and not reinvent. The goal would be to add value to those tools in such a way that gives drupal a distinct competitive advantage, specifically for the market of site builders/non developers/folks without the same resources as "enterprise" etc.

Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'.

-1 For using composer in a GUI in Drupal Core.

+1 For using https://electron.atom.io/ as explained by @Mixologic in #2538090-13: Allow the Update Manager to automatically resolve Composer dependencies. That is the solution for this problem! Let @Dries create a initiative for it.

So... I've been following this issue for a while, because the issue of how to explain Composer in the User Guide came up recently (the User Guide issue is in the Related Issues sidebar here already: #2843440: Discuss Composer and tell how to use it to install some contrib modules). I suggest we step back a bit, and think about what situation we currently have, and what we want to achieve, and then think about how to achieve it. Here is my perspective on the situation/goals/solution:

a) To run a simple Drupal 8 site, as things stand now, it is definitely possible without having Composer installed or even knowing about it. You can download the Drupal Core tarball, unzip it, and go. Same when you update Core.

b) For the vast majority of Drupal 8 contrib modules, you can also operate without Composer: download the tarball, unzip it, and go. Same when you update the contrib module.

c) We have a GUI in Core (in Update Manager module) that allows users to download/unzip files for contrib modules in the GUI, when first installing them or when updating them. IMO this is dangerous, and I personally never set this up or use it, but it is already part of Core and has been since the D7 days. Currently this only works for the modules that work via (b) in Drupal 8.

d) There are a few Drupal 8 contrib modules that rely on 3rd-party libraries that are not distributed with Drupal Core (Geo, maybe Commerce?, ...). Currently, in order to get these modules installed and runnable, you would either need to run Composer on your local dev site and then carefully transfer a bunch of files (vendor? composer.lock? some autoload thing?) from your local to dev site (ideally using a VCS), or you would need to run Composer on the live site directly. Neither of these procedures is easy for our hypothetical "non-expert" user. This is the crux of the current issue.

e) As far as I can tell, the main goal of this issue is to make it so that "non-expert" users can download/update core and *all* contrib modules without having to know about or install Composer. Note that #2477789: Use composer to build sites proposes the opposite: to make it so that no one can build a site (even pure Core) without installing/running Composer.

f) One way to make (e) happen would be to have code in the Update Manager module that, when you are downloading/updating modules via the GUI, would also run Composer update so that their 3rd-party libraries are also downloaded/updated. This would only help people who are already using the (dangerous, IMO) Update Manager module GUI to download/update their contrib module files, but would probably be sufficient to satisfy the goals of this issue.

g) Given that we have, as a community, already decided that (c) is OK to have in Core, adding the capability for (f) doesn't seem much more "dangerous" to me.

h) I don't think we need a new GUI to manage Composer dependencies in general. In fact, I am not convinced this would satisfy (e) if we did. It would instead add another complication for non-expert users to understand and use, which would be a usability problem.

i) As an alternative to (f), another thing we could do to achieve the goals of this module is make it so that, like we do for Drupal Core, the d.o packaging scripts add the 3rd-party dependencies to the tarballs, so that you could download contrib module tarballs and go, just like you can with Core and with contrib modules that don't have dependencies on additional 3rd-party libraries. We have solved this problem for Core, so it should be possible, in principle, to do so with Contrib as well.

Thoughts?

We can do what this issue want and use the update manager manage composer. For now that will be enough. The problem is that Drupal core and contrib will be using composer more and more. See: #2477789: Use composer to build sites and #2002304: [META] Improve Drupal's use of Composer. Composer is THE dependency manager for the PHP community.
Doing what @jhodgdon suggest will not work in the long run. As an in between solution it will work. But https://electron.atom.io/ or something similar will be the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

A/B is accurate.

C. There are many modules that rely on third party dependencies, and that number will only grow as more and more essential functionality becomes based off of php libraries that contain solutions that the drupal community should not be re-inventing. Right now, it covers things like 3rd party integration API's, and we are gradually moving to a point where you will not be able to build an effective, functional drupal site without using modules that rely on composer. There are already some modules that are commonly used that fall into this category, such as the address, devel, backup_migrate, colorbox. Right now about 10% of all d8 modules have a 3rd party dependency.

D. is accurate

E. Most of #2477789: Use composer to build sites was "make it possible to use composer to build sites" with only a part of the proposal to "make it require composer". However, in making it possible, we also made it required in some instances. This issue isnt the opposite, its intended to make the composer requirement transparent to end users.

F. Update manager *is* risky. Adding composer to the production environment increases that risk.

G. While composer shares the same concerns of "writing to the executable filesystem" that update manager has, There are additional issues: Composer is a dev application, and contains a significant amount of code that was never written with security in mind that would be executable by production sites. It introduces a considerable attack surface, that we can't really fix, because upstream logic would be "why should we take on the additional maintenance burden of making composer secure because you want to run it in production, which we strongly discourage?"

H. We need a user interface for managing what code your site has because *none* of the current methods that the 'non-expert' class of users are currently used to will continue to work.. You cannot use FTP and tarballs, or update manager to manage a site if you need a module that requires composer.

I. It is not possible to pre-package modules with composer dependencies, because those dependencies are not static. This isn't a problem that is solveable. Composer needs your entire installation to be able to figure out what it needs to download. This is a limitation of PHP itself (two modules might have the same dependency, thus have the same code loaded twice), and not something we should try and find workarounds for.

... the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

^ that quote perfectly summarizes why we really need to do this and why we shouldnt harbor notions like "everybody should be using the command line and following 'best practices' of professional engineering workflows"

I just wanted to add my two cents as a site implementer (and total drupal newbie) who has had a lot of trouble running composer on the command line for a shared hosting plan.

When I install drupal-composer/drupal-project, I'm getting excessive CPU/memory use which causes the process to be "killed" before it has a chance to install.

If you're aiming at the non-expert crowd with shared hosting plans (and I'm not saying that you have to), it might be worthwhile to measure hits on CPU/memory and to be clear about system requirements for hosting services.

re: #15 (@mixologic's second aspect about Composer expecting to be run in PHP CLI) --
Are there any obstacles other than how much memory and timeouts available? What about environmental paths, etc. Running "shell_exec('drush status 2>&1');" from a php script doesn't even run (I don't think) unless PHP is configured to run like PHP CLI with the PATH variable including the drush path and I'd assume the same for composer. But maybe if composer were included in Drupal it would already know the path. But would writing those updated files as the webserver user cause any problems? Wouldn't that mean the webserver user would have to have write access to dependency library files in /vendor?

Actually, I guess @mixologic addresses some of my comments in #26 with Point "G" in #24.

However, I do think this would be huge. This:

... the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

^ that quote perfectly summarizes why we really need to do this and why we shouldnt harbor notions like "everybody should be using the command line and following 'best practices' of professional engineering workflows"

is on the money and something I've had very strong concerns about regarding Drupal's direction and adoption for the last few years. I think it's great that Drupal is supporting superior development procedures, but then BetaMax had a higher resolution than VHS and in the end nobody cared except for television film crews.

It sounds like maybe #606592: Allow updating core with the update manager and #2367319: Implement automatic background updates for highly critical security issues might want to be added as "related"?

Happy FTP user here. Very interested to learn what "H" could look like :)

@Mixologic talked a lot about the technical side of why we need https://electron.atom.io/. I would like to look at from a non technical side. Electron is a desktop app that runs on Windows, Mac and Linux. In the desktop app you can create an run websites with web technologies like JavaScript, HTML, and CSS. Electron is also open source project on github and released under the MIT license.

I would like to add some non technical reasons why such a solution would be good for Drupal:

1. We can create an Electron app so that non-experts and not-yet-experts to compose all Drupal website files they need for their site via a GUI. The non-experts and not-yet-experts should not have use the command line. For me that is asking to much of them. Also as @rjnagle has pointed out: You cannot run composer on shared hosting plan. See #2845379-25: Provide optional composer integration but don't force users to understand how to use composer.
2. The previous point is a start but we can do more. If we add a webserver and a SQLite database to the Electron app, we can let the non-experts and not-yet-experts run their own Drupal website in the Electron app. The Electron app then becomes a staging server for them. They can configure Drupal. They can create content. And then upload it all to their hosting provider. This will be great for (beginner) site-builders.

Why this is good for the Drupal Association:

1. The uploading of the files, configuration and content to a hosting provider from the by Drupal created Electron app has a big benefit to the Drupal Association. As the "owner" of this Electron app they would be the ones that would be determine which hosting providers will be preselected as good choice. This will make the Drupal Association responsible for the quality controle of the preselected hosting providers. But that will give the Drupal Association the possibility to ask for a recurring fee from the hosting providers for being preselected. An other possibility would be to ask for a percentage from the hosting fee from the hosting provider.
2. The Electron app would get Drupal to be a solution for the people that now choice Wordpress. Those people will have a modest (for Drupal standards) budget for their website. Their modest budget will not be interesting for traditional Drupal shops. In the Wordpress world there are a lot of paid/freemium plugins (modules) and themes. The people with the modest budget will get a solution with many updates and constant support. The Electron app will need an appstore to support paid/freemium/free modules and themes. This will make the Drupal Association responsible for the quality controle and to keep an eye out for making sure that the deal is "fair" for the non-experts and not-yet-experts. If we take a look at Apple's appstore we see that they take a 30% fee from each sale. I am not saying that that would be the cut that the Drupal Association should take, but with a healthy appstore the Drupal Association should make a profit. The appstore should also have all modules and themes that are now free to download from Drupal.org and that are deemed safe by Drupal's security advisory policy. See https://www.drupal.org/drupalorg/blog/goodbye-project-applications-hello.... Anybody that wants to create a Drupal website without paid/freemium modules and/or themes can do so.
3. I would also like to point out that such an Electron app would be a great way for non-experts and not-yet-experts to learn Drupal.
   From an absolute beginners level and all the way up to an expert site builder or (front-end) developer. With an Electron app is very easy to start learning Drupal. You download it and open the app and you have started with your first Drupal website. There is no need to install webservers, PHP, databases or any of those things. It is easy for usage in schools and universities, everybody gets their own app. So if someone makes a mess their app, nobody else will have a problem.

The Drupal Association is always complaining that they need money and money in a recurring way would be even better. I think this will solve that problem.

I've been meaning to comment for a few days, but haven't had time to fully read each comment start to finish. So I'll follow up with edits, but I really wanted to post this before I never do.

As borisson_ stated in #16, I have created Conductor as an Electron app UI for working with Composer: https://github.com/mglaman/conductor. It bundles Composer, assuming the user has PHP on their machine (which could be changed in future, I'm sure.)

My biggest blocker, right now, is wireframes for a decent UI/UX and actual feedback from user's on what they'd want it to do. It lets you have a listing of projects, view their dependencies, update the project, and install new projects from Composer project templates. It's not Drupal specific, either.

Here's a gif of its usage:

I have Travis and Appveyor running builds and making artifacts. I have also started to add testing. It gained a lot of initial interest but there hasn't been a user feedback loop. As a Drupal Commerce maintainer, and advocate for D8+Composer, I am motivated to see something like this come to fruition. Tools like SourceTree taught me Git. If something like this helps a newcomer, then ++ to that.

Edit 1: I'm hoping to show this at the MidCamp sprints and get feedback from people there. Maybe someone knows a UI/UX person to guide how it should look.
Edit 2: I'll be in Baltimore and would love to BoF this.

As an external management tool, I don't think it would be something that Drupal itself would become dependent on. The same way you could still build and maintain a D7 site without Drush...even if your own personal workflow becomes dependent upon it.

it would add a non-PHP dependency on working with core

As #32 said, it would be just like how people used Acquia Dev Desktop or Drush/Console. Some of the other comments, which represent Mixologic's idea, is to take my simple "Composer only" approach in Conductor and make a community edition of Acquia DD which runs Composer, Drush/Console, and the web server stack ( at least that's what I'm gathering.)

Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'.

(#18)
That is probably the best description of the optimal user experience for site builders I have read so far. Thanks for putting it so eloquently Mixologic. It gives me hope that non-Composer experts might still have a place in the Drupal world, if your vision becomes a reality.

Throwing up another option which basically allows you to run "simple" composer commands via a browser.
Note there are some limitations to this approach but there's no reason you can't use both CLI & browser...

https://github.com/thom8/drupal-project/tree/build

This currently extends https://github.com/drupal-composer/drupal-project

The "tarball" (zip on GH) - https://github.com/thom8/drupal-project/archive/build.zip could be a replacement for the the normal core download and includes all core dependencies plus exposes a simple web API for running composer commands.

For example `/composer/require/drupal/devel/--/prefer-dist` would run the command `composer require drupal/devel`
see the README for more info.

Would be nice if this could somehow be tied into Drupal auth and only allow access to user 1 similar to update.php.
It could be a drupal module however there could be issues when composer is run to update the codebase.
Rather than returning the console output it could return JSON and have a better UI..

The way I see it, if there's a decent workaround it will allow us all to move forward and embrace composer :)

There's a major problem with #36. Server resources. Composer's dependency resolver eats resources like no other. It might be fine locally. However, one solution would be a remote .lock resolver. There was talk of this at DrupalCon. And it's not a Drupal problem. It's a TYPO3 problem as well, and others. We should really be reaching out to other CMS/Frameworks which have this similar issue.

@mglaman you're absolutely right which is why this project is prebuilt and only designed to run simple commands obviously restricted by memory. It was built on a VM with limited memory for testing.

more details here - https://github.com/thom8/drupal-project/tree/build#limitations
At least lack of memory has a relatively simple solution but a lock resolution service would be awesome :)

@mglaman do you have any references to the remote .lock resolver?

Could be a simple lambda service which accepts a composer.json & current .lock and returns a new lock file or error.

I'd consider this for CLI too as running composer from Australia or Fiji is painfully slow, only a particular client network it can take ~20 mins for an update of a single package...

I absolutely agree with #34! That would be the best solution and finally would be written against an interface.

In the spirit of NIH, PHPStorm has some basic composer GUI functionality.
https://drupalize.me/videos/composer-support-phpstorm?p=2017

Whilst not OSS the EAP version is currently available without license restrictions.
https://confluence.jetbrains.com/display/PhpStorm/PhpStorm+Early+Access+...

With a solution such as #30 you'd still need to use a workflow that builds locally and push deployment artifacts,
so you might get better support from an existing IDE.

Having decided to take the plunge and learn command line, composer, and drush a few months ago, I can attest that it's an excruciating hurdle for new users. Just removing the blockers with my host required multiple support tickets and lots of time. I still struggle to get Drupal Console working on new projects, sometimes just giving up. I had strong motivation to do it because our small agency is most comfortable with site-building in Drupal, and someone had to learn a composer-based workflow for a project that required Address. At this point, I wouldn't go back, but I'm not sure that my company has paid off the learning investment with higher efficiency over our old ways of wrangling tar balls.

I would guess that most new non-CLI people wanting to kick the Drupal tires will say "Composer what?" and walk away. To me this is one of the most important issues to the future of the Drupal project.

'composer require' (using a composer.json from Drupal VM) does not work on shared hosting with a hard limit of 1GB on PHP 5.6 or 7.0, even using the latest snapshot of Composer which is said to include memory optmization. At least, it did not work for me (details in a blog post.) This is likely to limit the usefulness of a Composer GUI for site builders, assuming the GUI will call for more more memory, not less, than CLI. 'composer require' seems far more memory-hungry than 'composer update'.

Anyway, that point was already made by mglaman made in #37. It cost me a few hours, because I had not realised how hopelessly ineficient Composer is. Perhaps some sane programmer will re-write Composer in C, so that it is not quite such a lumbering beast.

#41:
Suggesting people to install phpstorm is even worse I'd say. It's a huge application that does a lot of things and while the composer support is solid, I feel that would be overkill.

#42:

I would guess that most new non-CLI people wanting to kick the Drupal tires will say "Composer what?" and walk away. To me this is one of the most important issues to the future of the Drupal project.

We can't go back, to not using composer at all. Composer does more than just downloading files (such as writing an autoloader), I agree that this issue is important but keep that in mind.

#43

'composer require' (using a composer.json from Drupal VM) does not work on shared hosting with a hard limit of 1GB on PHP 5.6 or 7.0, even using the latest snapshot of Composer which is said to include memory optmization. At least, it did not work for me (details in a blog post.) This is likely to limit the usefulness of a Composer GUI for site builders, assuming the GUI will call for more more memory, not less, than CLI. 'composer require' seems far more memory-hungry than 'composer update'.

As @mixologic mentions in #18, composer is not built to run on production machines, and having composer on your live servers could potentially lead to security issues. The UI that's being built is intended to be ran on a local machine with the results ready to be copied to the production environment. That copy could be done trough a build process; but can also be done by FTP.

Anyway, that point was already made by mglaman made in #37. It cost me a few hours, because I had not realised how hopelessly ineficient Composer is. Perhaps some sane programmer will re-write Composer in C, so that it is not quite such a lumbering beast.

http://xjmdrupal.org/blog/someone-worked-hard-on-it

The composer-team has done an amazing job of pushing php into a new era of success over the past years, and I'm sure that they all did their best on trying to make it as efficient as possible. While rewriting it in C would make it more memory efficient, it will also make it harder for the target-audience (PHP-developers) to contribute and make it better.

A couple of people have brought up that composer is very resource-intensive, and have mentioned that shared hosting servers can't support it, I think we should communicate clearly, in some way, that composer is intended to be ran locally. Not sure how we'd go about doing that.

Regarding one point from #45 -- how to communicate that Composer should be run locally and then the results copied in some manner to the live site... This seems to be very important, and I have a couple of thoughts...

There is quite a bit of Composer documentation on drupal.org now, in this section:
https://www.drupal.org/docs/develop/using-composer
My suggestion would be for this section:

a) Add a sub-page there explaining the workflow of run locally then copy to production, and why you want to do it that way.

b) Go through every page in that section and make sure that this sub-page on local-copy-to-production workflow is mentioned/linked everywhere. The reason this is important is that most likely, people will not start at the top page and read everything in the section. They will most likely reach one of the sub-pages through a web search or a search on drupal.org. So, it's important that this crucial piece of information is linked everywhere; at the same time, we don't want to write it multiple times, which is why I suggest making one page that explains why and how to do it, and adding a short link on each page to this information.

c) We also have two pages in the User Guide that mention composer. I think we need to clarify this point in those pages as well, so I created a separate issue in the User Guide project for that:
#2881438: Recommend running Composer on local not production site

I'm not sure I agree with local then push to prod is the /best/ approach. It is an approach. Used by some people. But it isn't the only one that is valid. We should present the option of local => prod as a potential solution for certain sets of problems. Not the best solution in all cases.

Are there security issues with running composer on a production site?

It's really no different than running drush on a production site, so would suggest it more "best practice" rather than you have to run locally then push to prod.

You could also use an external dev/stage server to resolve dependencies then pull back down to local,
in which case composer isn't required locally or on prod.

#45

We can't go back, to not using composer at all. Composer does more than just downloading files (such as writing an autoloader), I agree that this issue is important but keep that in mind.

I'm not arguing to "go back," but I strongly believe that for the future of the Drupal project, we need to find a way to both support composer and avoid the requirement that a person must be familiar with command line in order to spin up his first Drupal site.

I saw this on Twitter: Composy.io - Composer as a service I am not representing it as a solution, but it is an interesting idea.

This blog post just reached the planet earlier today: Installing Commerce 2.x without Composer, with Ludwig. It looks like something more akin to Libraries API, but more in the spirit of Composer (though without the flexibility or—ahem—composability).

But it is a GUI of sorts, in terms of allowing module installations completely by hand. But it is yet another standard available to Drupal 8 contrib module authors to deal with. Now you have to decide among:

1. Use Libraries API, use or don't use Composer as well.
2. Use Ludwig, use or don't use Composer as well.
3. Use Composer

And there are upsides and downsides to all three of those options.

And to be clear—I like the idea behind Ludwig (it's basically what I think Libraries API would/should be in Drupal 8)... but it adds yet another package management solution, meaning some modules might have ludwig.json and composer.json files.

I've already been monitoring the 'has custom Drush commands' vs. 'has custom Drupal Console commands' problem in Drupal 8 contrib... and it's annoying to have to have one site using two separate tools for CLI interaction. If some modules choose Ludwig, some choose Composer, some choose Libraries API, and some choose a combination of the above... that sounds like a DX disaster even worse than what's going on with Drush / Console.

@geerlingguy
There is no way to use Libraries API for external PHP libraries, it doesn't do anything related to the autoloader. A manual way to load a library before using it won't cut it, cause you need access to the external classes in your own services and use statements.

Until today the choice was use Composer, or don't use an external library. Today the choice is between Composer and Ludwig. And we are clearly recommending Composer by default. A module should always have both JSON files (or just composer.json if they don't care about non-Composer users, which is valid).

And there is no combining Ludwig and Composer. You use one or the other.

Some issues begin to actually address these problems, and yet remain unreviewed: #2494073: Prevent modules which have unmet Composer dependencies from being installed #2830880: Warn site admins when composer dev dependencies are installed inside of docroot

@borisson_ said in #45

As @mixologic mentions in #18, composer is not built to run on production machines, and having composer on your live servers could potentially lead to security issues.

OK. So I won't use it that way. However, requiring a site to be built locally then trasferred to hosting, is on its own enough to ensure that Drupal 8 is taken out of reach of many DIYers.

It may be that the increasing professionalisation in the Drupal world means that this must be faced squarely. On the other hand, saying 'inexperienced website builders need to install composer locally, create a dev site, and upload it to Godaddy or Hostmonster, and they are good to go' does not in my experience suffice to keep Drupal realistically accessible to many of the users who enjoyed D6.

Hello, I'm the lead developer of composy.io. We think that using Composer is the right way to build a modern PHP application, Drupal 8 included.
We've built composy.io for the people that don't want to use/learn Composer or the cli, but also for people in country where Composer is too slow or in company with firewall/proxy that blocks some network connections.

In this way the choice is to use Composer but without the complexity of Composer.

The base plan will be free of charge forever and the package we provide to download contains a composer.json file so an user can switch to manually use Composer when he want.

In a near future we'll plan to allow users to directly push to some hosting the package built by composy.io

I like the Ludwig concept as well, there are use ca session for it as well. I use Composer for some complex sites with dependencies, but oftentimes there are projects with just one or two external php libraries that are self-contained and have no other dependencies. Using Composer for those I feel like I'm using an armoured tank when I just need a bicycle to get across the park.

I don't think there needs to be conflict BTW the two. A module would always provide a composer.json by default, but should also provide a ludwig.json if it wants to support installation via Ludwig for less advanced developers?

I love the Linux cli, (which really makes sense once you've used it a bit) but composer doesn't seem that intuitive to me, and maybe I haven't used it enough to really say this, but it doesn't seem near as useful as drush was. "drush dl admin_toolbar" is so much less convoluted than "composer require drupal/admin_toolbar". Why not just work with the drush people to make drush more powerful, or build a tool that would be more complimentary to drush?
It seems like this huge tool that I won't use unless I just have to, but at the same time will always be in my face and I'll always be stumbling over it. I've already run into issues where it's using too many resources, and I've had to adjust some things to make it work. I really don't want to use it in production, but is that the next step? Will it be a requirement to even have an instance of Drupal running?
I know there are a lot of really smart people who love this tool, but why can't it be just that a tool that allows me to choose it or not?

Comment related to this subject in #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x by xjm:

@moshe weitzman:

Er, thats a long time from now. You still have 18 months to train up folks on Composer, or release a new major version (Drupal 9), or ask Symfony to extend even further. Or you can do nothing - thats fine too. But lets not throw shade at Drush.

I don't mean to throw shade at Drush, although I am really concerned about the stated policy of no longer supporting the sites that need Drush most. This is a shared problem for Drush and core.

Having full Drush support for the Symfony 3 update and solving all the problems was a prerequisite for having Symfony 3 committed to Drupal. I was told by the framework managers that all the Drush problems were solved. I don't know how they got unsolved (catch implied it was a later commit that broke it), or if updating from Drupal 8.3.x to 8.4.x was never tested, or if the connection wasn't made that requiring Drush 9 wouldn't fly because Drush 9 essentially doesn't support most Drupal 8 sites.

And, unfortunately, it's not simply a matter of "training people on Composer". There is deep concern about composer as the primary means of installing Drupal from many audiences, strong enough that people pick other solutions instead of Drupal. Composer is the bee's knees -- for PHP devs and devops people like you and me who can support it -- but for a significant chunk of our current market, it's actually a barrier. Not because they're not "trained". This is offtopic here, but it is a big risk for Drupal right now and finding a way to support both audiences is a problem that's going to take a lot of resources to solve.

18 months is not a long time in a six-month release cycle. As I said, we need to have Symfony 3 in this minor, or the next. The notion of releasing Drupal 9 next year is quite far-fetched when 90% of Drupal sites are still on Drupal 7 and the migration path isn't even done yet.

@Scott_McKinnis
Could we try to not repeat the same 3 points every 5 comments? It is making this issue very unproductive.
All of your questions have been answered several times above.

@ressa
I don't believe the linked issue is relevant. It is about whether CLI users can stand to use another CLI tool. This issue is about not leaving behind users who don't want to use any CLI.

This looks interesting - https://downloads.getcomposercat.com/

Windows, MacOS & Linux + SSH support coming soon so you'd be able to run locally to update a remote project.

Want to have GUI for Composer features? These issues remain unreviewed: #2830880: Warn site admins when composer dev dependencies are installed inside of docroot #2494073: Prevent modules which have unmet Composer dependencies from being installed

Even just +1s would let core developers know you want these features.

@bojanz, that is not what the linked issue is about. The linked issue is about Drush 8 not working with the 8.3.x/8.4.x transition while Drush 9 is not released and (later) that even when it is it will not support non-Composer workflows, which is a much more serious and immediate problem than your wording suggests. It's not about what "CLI users" can "stand"; it's about breaking existing sites between minor releases being a non-option.

My comment as quoted was off topic for that issue (as I mentioned), but would certainly be on topic here as this proposal would address the portion of the Drush userbase that wouldn't easily make the switch to a Composer workflow. (Though nothing as yet addresses how existing sites would switch to this new tool, but maybe that's either not part of the MVP or it's just implementation detail.)

The very first comment on this issue links #2845378: Stop offering .tar.gz downloads, which is from a user who said this:

However, once you start using composer, the regular methods of keeping the site up-to-date (`drush up`, or extracting the .tar.gz file) break everything. As it apparently is the opinion of the Drupal community that Composer is going to be a requirement, you should stop offering .tar.gz files for download; I now need to figure out how to move my site over to composer in such a way that I can apply security updates, and the existing documentation is really bad about this.

and later

(I've been using Drupal since something like 4.5, and this is *really* frustrating and annoying. I can understand some of the motivations behind moving to composer, but even as a programmer I find it really confusing and poorly documented. I'm now going to lose several days trying to make this work, most likely.)

So, yes, Drush is relevant and Drush's audience is not a monolith.

Developer of Composercat here. It's nice to see my project mentioned in this thread. I've had some good feedback from Drupal people, so I wanted to drop by and mention a couple of new features I'm working on that might be of interest to the Drupal community in particular:

1. Currently Composercat only supports searching and displaying extended package information for Packagist packages (you can install any package though, you just need to know the package name if it's not from Packagist), but the next version will have support for custom package repositories, including the Drupal repository (so you can search for Drupal packages the same way as you can with packages from Packagist).
2. The next version will support creating projects from packages, so you can easily start new projects from Drupal project templates.

You can expect the new version to be released in the next couple of weeks.

I am a lowly sitebuilder who got into Drupal because it allowed me to build incredible things via a GUI.

I make a lot of small websites on shared hosting, and often weight Wordpress versus Drupal. I often wish that Drupal would auto-update like Wordpress, so that I wouldn't have to make that choice, as I think Drupal allows me to make better websites.

In my world the biggest problem is not a site that is broken by an update, but a site that is taken over by Russian script kiddies because it wasn't updated.

Right now I often update by
(1) making a backup
(2) just FTP-ing a new Drupal version over the old one.

99% of the time this works fine. In the other cases I roll back the update and do it 'properly'.

I would love a tool that just tries to update the site this way. If it detects that the site is down, it puts back a backup and emails me to do the update manually.

All this reminds me of an older debate: should Drupal have an upgrade path between mayor versions, or should you just migrate? In the end the migrate option won.

Similarly, all the current proposals talk about "how it should be done", and then come up with these complex constructions. But I would hope for a migrate-like "keep it simple" option here. Most of the parts for this 'dirty' tool are already there (backup module, poorman's cron). In 2017 HDD space for backups is cheap, while my time and attention are not.

In the end this seems like a debate between two ideas: "Drupal is a corporate tool" vs "Drupal is a Wordpress competitor". I hope Drupal will (also) continue to be the latter, because Wordpress' limitations become very obvious once you also know Drupal ;-)

Some of what you discussed is being thought about on #2367319: Implement automatic background updates for highly critical security issues.

P.S. I will never understand why anyone thinks site building is "lowly".

@groovedork,

All this reminds me of an older debate: should Drupal have an upgrade path between mayor versions, or should you just migrate? In the end the migrate option won.

As of Drupal 8 all updates are in place. The first version of Drupal 9 is planned to be identical to the last version of Drupal 8 with all the deprecated functions/API removed. This gives developers time to remove deprecated code prior to Drupal 9, allowing an in place update.

@groovedork I agree with you that Wordpress's 1-click core update for core is a major selling point for them for smaller sites and agencies. We too are using Drupal to build sites for which many people would argue that Wordpress is the better fit. I am hoping this issue will in the future allow non-technical site-builders to manage updates for both core and modules in a simple, non-command-line way.

My sad impression, using Drupal 8, is that the difficulty, compared to drupal 7, will kick most of the less experienced and less enthusiastic users. Drupal will only be used by programmers for complex projects and will be abandoned by agencies. Consequently, the Drupal community will shrink and slowly die.
I'm sorry to say this, I use Drupal since version 5, but that's what I think it will happen.

After a few months of using composer to maintain multiple Drupal 8 development instances, the advantages of a composer style dependency management subsystem have become clear. From the command line, "composer update" correctly updates all the various dependencies in a Drupal 8 instance with a single command. A shell script can be written that updates an entire set of Drupal 8 instances on a server all with a single command as well.

However, command line interaction with a software system is palatable only to the developer demographic, but is an experience that alienates "agency" level website builders (designers), and alienates the clients they leave their projects to even more. In 2017, small business project clients in particular, without formal IT departments, simply won't tolerate a website/application installation that requires them to run command line scripts to perform security updates on their site's code.

What I believe is that a "headless composer" must be developed that can be integrated into Drupal core, and integrated into the Drupal 8 admin GUI, so that Drupal 8 can have a GUI "click and go" user-interface for intelligently updating core and/or modules, with underlying composer level dependency management functionality. Without this fundamentally essential GUI functionality, the future audience for Drupal will be limited to a technological audience, that many believe will erode the Drupal user-base and audience in time.

As others have noted above, GUI based core updating and module installation and updating, are a feature essential to Drupal 8's future viability and marketplace health.

As others have noted as well, this requirement is both a technological one, and a Drupal marketing requirement, that is also impacted the unwillingness of many hosting companies to configure their servers for client SSH command line access (not to mention the core resource requirements of Drupal 8). Not everybody is a developer running on a VPS or dedicated server with root access and total control.

GUI code dependency, code installation, and code update, management are fundamental to Drupal's future.

@sprite You said it better than I could have said it myself!

I have been using Drupal for many years for several web sites. I am not a programmer, but I am comfortable with the command line because I use Linux at work (networking, servers, package installation, etc). I have maintained a Drupal 7 small business web site for years. I recently decided to build a new site using Drupal 8 and Commerce. The install looked totally easy because of Composer. I built a stand-alone test box using Ubuntu, and the install worked fine. I then decided to build an on-line Drupal 8 site, parallel to my Drupal 7 site. After about 10 days of work, I have had zero success. Here are some points about my experience.
1) I knew that I would need Composer, so I requested that my ISP install it on my account. They did so, without hesitation.
2) I attempted some basic Composer commands and they worked fine.
3) I attempted to install Drupal Commerce, but the command line simply froze. I ran Composer Diagnose, and it said that allow_url_fopen=0 was stopping the process. My ISP set up allow_url_fopen=1 in a php.ini file in my home directory, and they said that should fix the problem. However, I found that it opened up Composer only for the web interface, not the SSH interface!!
4) My ISP boosted the problem up to a higher tech support level, they consulted a PHP specialist who recommended that I use -c
| to bypass the server restriction on allow_url_fopen. I tried this, but the level 3 techs have evidently blocked allow_url_fopen in the shell for the entire server, not just my account. This is for security reasons. Without allow_url_fopen= set to "On" or "1", Composer is useless.

So, my point is this: I am a long time Drupal user who simply wants to upgrade to Drupal 8. I am comfortable with both the command line and Composer. At this point I have not found a way. A built-in Drupal Composer GUI might be the best solution, but it is not necessary for me, and it seems like a long way off. My next step is to explore the idea of developing the web site offline with Composer, and FTP-ing (or some form of syncing) back and forth between the offline and online machines. It may be do-able, but has substantial potential for challenges along the way. A clear systematic approach to building a Drupal 8 web site (with Composer) offline, and maintaining the regular updates over time (back and forth without data loss) to the live server would be quite helpful....

agree with @sprite on comment #72

Drush users: pm-update and (I think) pm-download are dropped in Drush 9. This is partly a restatement of comment #60.

Related discussion taking place at #2905741: *DRAFT* Proposed product goals for Drupal 8.5/8.6(+) core.

FYI See #2906637: [META] Drush and core compatibility is fragile - there is a danger that "drush pm-upgrade" will stop working with Drupal 8, yet an estimated 60% of users rely on it.

I have tentatively proposed an update to the title (feel free to change it back if you don't agree) as the term "non-expert" feels confusing and unfair.

• There are likely expert site-builders with 10 years experience of the Drupal GUI, but minimal command line (enough for "drush pm-upgrade") and no text-editor software installed
• There are expert Drupal developers who have no knowledge of composer.

It seems likely that composer is the future - many users are already enjoying various advantages. The drush team have removed pm-upgrade from drush 9. Many modules are now using composer to install and update external libraries.

Drupal 8 benefits from the features of Symfony and TWIG, but you don't have to even be aware of that to use the Drupal GUI. We should do the same for composer. If we can hide away the internals of composer then it's a win for everyone. I suggest that there are many people who are strongly against having to learn to use compose, but few people who are against the benefits if it all just works.

I am one of the "Drupal developers with minimal knowledge of composer". I would be very happy to see something like this:

• Drupal site-install (from drush or GUI) includes the composer binary, generates a composer.json with dependencies in the right structure to leave composer fully working
• Drupal GUI "Install" (/admin/modules/install) and "Update" (/admin/modules/update) map to composer commands.
• There are drush wrappers (cm-download, cm-update) - because of course a properly secured site will not grant the web server write-access to binaries whether or not composer is involved.
• Drupal GUI has a button to automatically convert non-composer sites to work with composer.
• "Available updates" (/admin/reports/updates/update) is based on querying composer and includes available updates to external libraries as well as to modules.

Other challenges (things I discovered when experimenting with composer):

• Keep current "security only" option working in composer
• The current instructions are complex and include two options with different limitations
• Handle the different structure of version numbers (composer uses "1.0@RC") and preserve the Drupal concept of a recommended release (can be 1.x even if 2.x exists).
• Ensure libraries that include CSS or JS are installed in webroot/libraries rather than the composer default location that is outside of the webroot. The mechanism "type=drupal-library" is good, but only works if the library codebase is in control of the Drupal community.

Equally I would be happy if the underlying tool was not actually composer. My key point is that the underlying complexities of the tool need to be hidden from users that don't want to delve into them.

@AdamPS: Your new title is better than the old one, which was a bit misleading. Drush and Composer have very different difficulty levels, just read Composer and Drupal are still strange bedfellows by Jeff Geerling for examples on how complicated Composer can be to work with.

Thank you, @AdamPS, love the new title (I was also rallying against "non-CLI," bc a very real portion of mostly-GUI site builders use drush to speed up a number of run-of-the-mill admin tasks, so I love the new title from those angles, too).

Thank you, also, for your explainer and suggestion comments, they're quite thoughtful and clear. I am and will continue to be following all of these related issues, and will chip in as often as I can.

With all the talk about this and related issues lately, I feel this should be the title. Nobody is forcing users to understand the internals of Composer. "Internals" means "the inner workings" of a thing. That isn't the case here.

Ahhh good point, @cilefen. +1 to re-tweaked title as far as I'm concerned.

Related: #2908394: Use Composer to build sites without forcing users to learn Composer

Big +1 to this.

I have lost more hours in the last year or so to headdesking over Composer than to anything else.

@all followers of this thread: Make sure you also check and follow the new issue @Mile23 created (and added to "related issues" here, yesterday):
#2908394: Use Composer to build sites without forcing users to learn Composer

@cilefen Thanks for your improvement to the title. I propose a slight further change to address the following goals.

1) Clarify original point by @DamienMcKenna that install via tarball should continue to work. This is important for a variety of reasons

• Composer is not necessarily suitable for some environments: memory limit (composer can fail even with 2GB); shared hosting where FTP is only option; auto-installers (e.g. Softaculous).
• As per the Drupal core deprecation policy don't remove function which would break existing sites until next major version = D9
• In any case, Composer is not needed for simple sites that don't need modules with complicated dependencies.

2) Propose that the key is to provide "integration" not "a GUI".

• I suggest that this issue is not calling for a generic composer GUI: that would still expose composer terminology and concepts. It would allow mistakes (remove a module without uninstall; downgrade even though update hooks have changed database) and miss Drupal specifics (security release; recommended release).
• Install of files through GUI is not compatible with Drupal recommended file permissions. Some users will have a separate staging site which is not publicly accessible so a GUI is OK. Others use drush (pm-upgrade) and would like to continue to do so. Hence the Drupal composer integration module should have a GUI and also drush integration.

In summary the proposed plan is to develop a Drupal composer integration module with a GUI and command line, whilst maintaining the existing tarball installation method.

This will hopefully be best for everyone, including the composer enthusiasts. It should be much easier to make progress on composer integration aiming to be suitable for 75% of users rather than 100%.

Site owners installing from tarball and using drush might be interested in #2909032: Support drush for sites not built from composer

For everyone following this issue, amateescu is experimenting with an approach, where a service on drupal.org handles all the Composer dependency resolution, and it looks very promising: #2910136: Experiment: package PHP libraries in a single Phar file

Hi! Not trying to imply "yes" or "no," just asking:

Is this issue still needed, given all the other related issues that exist -- in particular:
#2908394: Use Composer to build sites without forcing users to learn Composer

I think #2912406: [META] Replace update_manager with a more powerful solution is probably where things are headed.

So if we want to let that 40% keep doing that, what's the solution?

If their modules don't have composer requirements then problem solved.

If their modules do require composer stuff, then they have to change their workflow no matter what we do to make it easier for them.

Thanks @DamienMcKenna

The big problem here is that manually downloading a module with composer dependencies will not install it's dependencies and importantly resolve conflicts with your existing modules.

The solution could be as simple as Drupal dropping support for including composer dependencies in modules or enforcing composer and dropping support for all other methods.

Have proposed another solution here -- https://github.com/thom8/faas-composer/wiki/Use-case-for-Drupal.org

Basically a way to run composer via the Admin UI and the end result is no different to running composer via the CLI, so a site could be build with CLI and then maintained via the UI.

@DamienMcKenna It only takes one to have to change your workflow

Over in #2494073: Prevent modules which have unmet Composer dependencies from being installed the eventual idea (after 300+ comments) was to add a build key to info.yml saying: This module needs Composer in order to work.

That allows you to know that you'll need to change your workflow before you change it. :-)

But now we have #2912406: [META] Replace update_manager with a more powerful solution which is apparently underway, which seems to be very similar to @8thom's idea from #95. Whether adding a key to the info.yml file is needed for that remains unknown.

Composer is about sharing packages with other frameworks like we do in core -- http://cgit.drupalcode.org/drupal/tree/core/composer.json#n8

This means we can leverage the wider PHP community for contrib, in a safe way that doesn't conflict with other modules or core sharing the same packages.

So, are we talking about the current state or where we want to get to?

In a composer model a PHP framework agnostic "panels" package is possible with a far greater reach for contributors,
All that would be needed is an integration component that allows it to work with Drupal...

Maybe I'm dreaming...

But a more practical composer example is that we maintain our own version of webform that has the webform module as a dependency and installs it with a bunch of default config specific to Australia and the way we implement webform.

This would be much harder without composer.

The "fork" is only a set of config yml files and a composer.json to reference the webform version it currently supports.

If anyone is keen to help out, the longterm plan is to hopefully push this upstream to composer with a the service run by packagist.

Caught up with Nils (packagist/composer founder) during badcamp camp a couple of weeks ago and there's interest in at least adding support in composer v2.

At this stage, I've created a PoC + a bunch of issues that need to be addressed.

This isn't a Drupal only problem, so hopefully we can find a solution that transcends Drupal's immediate problems.

I have some newbie type questions that I hope are appropriate for this thread. They are at the end, this first part is just context related to the subject of this thread.

I am one of those users who would prefer not to have to learn a bunch of command line workflows. I am a small business owner who built a D6 for our business (a recording studio) 10 years ago, and am finally biting the bullet to attempt to migrate to D8.

When I built the first D6 site, I knew nothing at all about website building, no html, css, nothing. It was a very difficult project, and took way too much time. I got into the position where I wasn't sure I wasn't throwing "good time after bad," because for many months I simply couldn't get the site to work. I finally got it all to work, and it's worked well for 10 years. So all I have done is maintain this site, keeping the process as simple as possible.

I'm shocked at how complex the Drupal system seems to have become! From what I can see, the current state of Drupal has definitively left behind the everyday user. If I hadn't already invested the time into building what is now a somewhat large Drupal site, there is no way I would pick Drupal over Wordpress. It's not that it's too hard, it's just that the learning curve requires so much time that it puts it out of reach of many smaller users.

I think I get the gist of using Composer, it allows for better workflows for professionals by managing dependencies between things like external libraries.

Anyhow, my hope is to migrate as much of the content as I can to a new D8 site, roughly 500 nodes worth.

I've gotten as far as setting up Acquia Dev Desktop. I've got a local version of the D6 site running there. I followed the Acquia documentation, and ran composer to set up a fresh D8.4 site, using their Lightning distribution.

Now I'm thinking about how to actually do the migration, and it looks like Drush is a necessity for any custom migration (which is what I need to do, the old site can't be migrated totally.)

I've never used Drush before. But it looks like it was installed in the Acquia installation. So I ran $drush status and I got a long error message that starts:

Fatal error: Declaration of Drush\Command\DrushInputAdapter::hasParameterOption()....................

$drush version says the version of drush is 8.1.8

Searching for help with this is how I wound up here.
------------------
Question(s) A- From what I can tell, is it the case that this version of Drush does not work with D8.4 sites? In this case, will I not be able to use the various Migrate modules to do the upgrade?

Question(s) B- Since I installed the initial version of the site with Composer, do I have to use Composer to, for example, add a module? Once I deviate from the Composer workflow, is it impossible to go back?

Question(s) C- Is there documentation that could help a newbie understand the interrelation between Composer and Drush, and suggest workflows for different situations?

Thanks everyone!

Thanks for the reply @DamienMcKenna! Sorry for posting in the wrong place. It's become clear to me that I am in way over my head with trying to migrate D6-D8. I'm now trying D6-D7-Backdrop, am making very modest progress.

Hi,

I'm using Drupal 8.4.2 and Drush 8.1.0. Whenever I execute any Drush command it shows below error message.

PHP Fatal error:  Declaration of Drush\Command\DrushInputAdapter::hasParameterOption($values) must be compatible with Symfony\Component\Console\Input\InputInterface::hasParameterOption($values, $onlyParams = false) in phar:///usr/bin/drush/lib/Drush/Command/DrushInputAdapter.php on line 27
Drush command terminated abnormally due to an unrecoverable error.                                                                                                                               [error]
Error: Declaration of Drush\Command\DrushInputAdapter::hasParameterOption($values) must be compatible with Symfony\Component\Console\Input\InputInterface::hasParameterOption($values,
$onlyParams = false) in phar:///usr/bin/drush/lib/Drush/Command/DrushInputAdapter.php, line 27

@hiramanpatil why are you spamming this already much to long issue?
PLEASE create a separate issue for that or search for one that fits your problem...

@Anybody

Sorry, I thought that this issue is active and will get a solution here. I will create another issue if required.

Thanks

This issue is active, it's just not relevant to the question you posted.

Drupal core idea #2940733: Site Builder Tool/Project Browser initiative was recently initiated.

Speaking from my own experience I have just completely deleted a new D8 site which I was building purely because of the pain of updating it with composer. I basically gave up after what must be a few days of fighting with it. Sorry but I just have better things to do, and can't be going through this every time.

After using Drupal (which I liked) for over 10 years now this is a shame, and I am off to have a look at Wordpress (which haven't used it for many years).

I'm a site builder who has been building several drupal 7 sites over the years. I recently found Drush, which massively simplifies updates. I wanted to replace my main site with a Drupal 8 site but I'm afraid that Composer makes it very unattractive, and I've spent more time on trying to work it out than I have on site content or design. I use good quality shared hosting that is more than enough for hosting a site, but the memory requirements of composer make it impossible for updates.

Before giving up, I'm trying to work out how to build locally and then upload, without taking a big step back from the automation gains I made with Drush on Drupal 7. If there was some really clear documentation that didn't assume you knew how to do everything already relating to version control, then that would be a good start to making it feasible. I've used git on github for a static bootstrap site and it worked great.

Drupal's inter-relationship between the database and the code, however, means it is not a simple matter to understand how a drupal version control system could work or how to set it up when you have an existing site. I'm not averse to using the command line, but there could do with being some clearer guidance that would enable more people to take up composer.

I would be happy to user test or help with UX/UI - my day job is a BA and I've done plenty of UX over the years.

Martin Price - I wrote a blog series on using Composer, for people in your position.

Part 1 is here: https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-com...

Haven't had much time to look into this issue recently but have just been notified of a new remote composer resolver project -- https://medium.com/@yanick.witschi/welcome-composer-resolver-cloud-30ebd...

Might be of interest to anyone working on this.

@martinprice_uk:

Drupal's inter-relationship between the database and the code, however, means it is not a simple matter to understand how a drupal version control system could work

Drupal 8 makes this a LOT (no really, a whole lot) easier to manage, because configuration (settings, fields, entity types, permissions, etc) live in code, not in the database. So the entire site can be versioned via git, deployed from a git tag, configuration changes imported via drush cim, and you don't need Features and Strongarm to (poorly) manage configuration deployment. I still maintain several Drupal 7 sites, and I hate deploying updates for those - compared to D8, a D7 update deployment is very very fragile.

That said, yes Composer is a memory hog when doing dependency resolution during composer-update. But you shouldn't ever be doing that on production, you should be doing composer-install --no-dev, which does no dependency resolution and just installs the specified versions from composer.lock rather than composer.json. I've not found that to be a memory hog - if you can reliably run drush updatedb operations on production then composer install should not be an issue.

Happy to chat about this privately if @jaypan's guide is not clear.

I’ve actually written a five-part series on the Configuration API as well, to understand some of the points you raised. We’re just proofreading internally before publishing. I’ll post a link to that when it goes live.

@Jaypan and @John Pitcairn
Thank you both for your advice. It's given me some good advice to go on and a bit more confidence to try again.

Ok, been using Drupal for a long while to manage a volunteer organisation's website. Completely non-technical about the innards of Drupal 8 and composer.
The hosting service I pay for out of my own pocket does not give me shell access to use composer, and a bunch of additional modules I need to make things work MUST be installed that way.
So, how do I do that?
To get the Drupal 8 I had to blow away the old Drupal 4/6 sites. We skipped 7 and the site has been down a long time while we got a committee in place that was willing to keep the site going and resurrect it.
I cannot afford to pay for a more expensive service to get shell access. So I cannot use composer.
Is there some solution for doing all this in the Admin console? Something I can install without Drush or Composer?
Thanks in advance.

The usual option in that case is you build it somewhere where you *do* have shell access, like your local development environment, and move it to the hosting company.

The goal is to someday be able to do something like this in the admin console, but right now there are a lot of things that has to happen to make that possible.

@jstevensog Check the comments on Ludwig here. Even with Drupal 8 there is no need to use Composer. It is still not a requirement, use Composer if you see a benefit of using it. Obviously, you need to do dependency management yourself, with or without Composer. Composer is just a tool that helps you when you have a separate development environment for your site. Ludwig helps you specifying the dependencies you need (ludwig.json) and getting them just like Composer. It doesn't determine the versions required, you need to do that yourself. You may need to request Ludwig support from the modules that specify non-drupal dependencies.

@jstevensog I've been through the same issue. We used to manage dozens of D7 sites through the UI. You can still do that with Drupal 8, but as you say, some modules are intended to be installed with Composer in order to automatically pull in their dependencies. So, with D8, you really should be making the jump to Composer. I'd say you have a few options.

1. As Mixologic stated, you can build and update your site locally and deploy it to your host. If the site is simply displaying content and nobody besides you is updating it, you can simply pull down the code and database, make changes, and then overwrite both code and database on remote. However, if you need to avoid overwriting content in the database, you should ideally export your configuration to code via config management after you make changes, update your code on remote and import configuration there. The Drupal community has initiatives around lowering the Composer barrier to entry, but that will take some time. I've found Local Web Development With DDEV Explained a helpful tutorial to local development.
2. You could find a host that allows you to run Composer on the host. However, many cheap hosts would not have the RAM needed to run Composer anyway.
3. If you liked the old ways of doing things, Backdrop CMS is a hybrid of Drupal 7 and Drupal 8, and you may find it easier to migrate to that platform if they have the modules you need. I'd rather see you use Backdrop than give up on the Drupal ecosystem completely because of Composer.