One of the items mentioned during some of Dries' keynotes is that there's a danger of Drupal leaving beginners and site builder behind. A major step in the current march forward is Composer usage. While Composer is a great tool and should become the standard for building complex projects, it should not be required to build a Drupal site unless a GUI is provided; additionally it should always be possible to download files from d.o, put them in the correct location (after extracting the archive) and have it work.

This should be seen as an opposition to #2477789: Use composer to build sites becoming a requirement unless a GUI is provided.

Files: 
CommentFileSizeAuthor
#41 PHPStormDrupalComposer.png240.1 KB8thom
#30 conductor.gif1.73 MBmglaman

Comments

DamienMcKenna created an issue. See original summary.

cilefen’s picture

bojanz’s picture

it should always be possible to download files from d.o, put them in the correct location (after extracting the archive) and have it work.

We can wish for this all we want, but it's technologically impossible due to the way Composer is built.

Mixologic and I discussed this, and agreed that the right solution is to offer a rewritten Update Manager that embeds Composer (basically shipping with updater/ next to core/). That way people either use Composer, or they use the UI which runs Composer and then moves the changes into place. We also get core updates for free.
#2538090: Allow the Update Manager to automatically resolve Composer dependencies.

DamienMcKenna’s picture

@bojanz: Thanks for adding that insight. An updated Update Manager would be completely fine, we just can't ignore the fact that a large portion of visitors *don't* run the command line, and that requiring it would cause Drupal to loose a lot of users.

Jaypan’s picture

That way people either use Composer, or they use the UI which runs Composer and then moves the changes into place.

The problem here is that it still requires composer. Most shared hosting servers do not have it installed by default, so to add composer requires knowledge of how to install it - and we're back to the command line. So while having GUI access is definitely a step in the right direction, users who cannot use the command line are still cut off.

cilefen’s picture

@Jaypan: did you read the part of comment #3 about embedding composer?

Jaypan’s picture

No, I missed that. That's a good solution. It actually covers the biggest issue with composer as I see. That would also be nice since non-technical users would then be able to install drush more easily as well.

dww’s picture

Agreed that we don't want to force anyone to have to use the shell to deploy/maintain Drupal.

Also agreed that the plan in #3 is the right way to rely on Composer to solve the problems it does without forcing end users to know it's even happening.

Part of how authorize.php was originally designed was to let it become a more generic framework for running something like composer as the user that owns the files in the filesystem. I'm not at all attached to keeping authorize.php as-is, but I definitely support any effort that allows it or something like it to run composer for site admins.

The installer could also use something like authorize.php to invoke Composer to bootstrap contrib installs during initial setup of a new site. There's talk of a module browser in core, etc. All of this could be UI around ways to get "update manager" or "install manager" or whatever we call it, to feed composer commands to the filesystem via authorize.php or its offspring.

Cheers,
-Derek

jhodgdon’s picture

cilefen’s picture

Maybe. But #2538090 needs an issue summary update because #38 explains it is probably untenable as written.

DamienMcKenna’s picture

Title: Do not require composer for building sites, do not leave non-experts behind » Add a GUI for Composer, do not require composer for building sites, do not leave non-experts behind
Issue summary: View changes

Clarifying that having a working GUI for Composer would make this a non-issue.

DamienMcKenna’s picture

Title: Add a GUI for Composer, do not require composer for building sites, do not leave non-experts behind » Do not leave non-experts behind: do not require Composer unless a GUI is also included
webchick’s picture

Priority: Normal » Major

Escalating, based on impact.

timmillwood’s picture

For background the lead up to this issue is at http://buytaert.net/making-drupal-upgrades-easy-forever#comment-131086

Mixologic’s picture

There's a couple of constraints that we want to consider when designing a system that will allow us to abstract away the complexities of using composer:

Composer is designed to be a build time/development tool, and is not intended to exist or be available in production systems. What that means in practice is that the normal constraints of "code outside the firewall and open to the internet" are not baked into composer, and thus it could be considered a security risk to have it available on a production system that interfaces with threats. Thats not to say it cannot be on live sites, just that upstream, the developers are not likely to focus on making it hard to compromise. Even if we were to audit it now and find that its fine, that doesn't mean that the next version wont have some obvious remote code execution vulnerability baked into it.

The second aspect of composer is that it is designed to be executed in a command line php environment - which means it is expecting to have the ability to consume considerable amounts of memory, and take significantly longer execution time than most webserver php environments are geared for. Drupal 8's 64MB minimum requirement would probably need to bump to 256MB or higher, and we might need as much as 180 seconds worth of timeout for php. Or more.

I bring up these two points to underscore the difficulty we may encounter in attempting to bundle composer within the drupal application itself. There are likely ways we can cordon it off for security purposes, and ways we can ensure its only running on a cron or something similar. Another alternative is to construct a site management tool that is separate from your web app, much like wordpress's desktop administration application (https://developer.wordpress.com/calypso/).

Composer isn't the only sitebuilding barrier, and we ought to address the "Constructing, Extending, Updating, and Maintaining" the codebase your website needs to run in a holistic manner - so that we can handle any other "Buildtime/Development tools"- i.e. we can add build tools like NPM or bower to solve the 'gather front end assets too' so as to provide a seamless, delightful experience for drupal users who do not fall into the classification of "working on enterprise sites for enterprise clients"

I also strongly believe that we should start by designing the ideal user experience we would like to see in a tool like this, and aim high enough to knock it out of the park.

borisson_’s picture

I bring up these two points to underscore the difficulty we may encounter in attempting to bundle composer within the drupal application itself. There are likely ways we can cordon it off for security purposes, and ways we can ensure its only running on a cron or something similar. Another alternative is to construct a site management tool that is separate from your web app, much like wordpress's desktop administration application (https://developer.wordpress.com/calypso/).

I agree with @Mixologic and matt glaman and I already started work on a tool that does this. I started by building https://github.com/borisson/composer-ui and a couple of weeks later matt built https://github.com/mglaman/conductor. The goal I had in mind (and that matt largely followed) is to make a tool to install composer dependencies for any composer-based project (including drupal, but also symfony, laravel and others). The biggest thing still to do there is to add the ability to add/remove projects. Currently it only does install and update.

I think if we promote and improve that tool we'd get pretty close to what calypso can do. Adding the ability to use npm dependency resolution would be awesome, but I think that'd be a lot hard to get right. That's why I'd leave that for a 2.x release.

We had a BOF about this at drupalcon dublin but only a couple of people showed up, probably because I named it "composer UI BOF" and not something like: "Installing modules from the desktop", I can't really remember a lot of things from that BOF, I forgot to take notes, but maybe klausi remembers more. I created a couple of issues in the queue of composer-ui as a result of that discussion:
- https://github.com/borisson/composer-ui/issues/12
- https://github.com/borisson/composer-ui/issues/11
- https://github.com/borisson/composer-ui/issues/4

phenaproxima’s picture

For my part, I'm not against having a UI to soften the steep Composer learning curve. In fact, I think it's a great idea -- as long as it's not Drupal-specific. When it comes to code base and update management, what I'm in favor of is dropping as many Drupalisms as possible. We should be taking advantage of everything Composer can do for us, and fill in the gaps (i.e., a UI) where it could be made easier and friendlier for Drupal users. And if our good work can move upstream, good.

The second aspect of composer is that it is designed to be executed in a command line php environment - which means it is expecting to have the ability to consume considerable amounts of memory, and take significantly longer execution time than most webserver php environments are geared for.

If we built a stand-alone Composer UI, maybe we could bundle it with a preconfigured copy of the PHP interpreter with higher exec time and memory limits, to sorta work around this problem.

What that means in practice is that the normal constraints of "code outside the firewall and open to the internet" are not baked into composer, and thus it could be considered a security risk to have it available on a production system that interfaces with threats.

Please correct me if I'm misunderstanding, but doesn't this just mean that it's potentially dangerous to keep Composer in your production environment? If so, how does that differ from the situation now, supporting modules that have Composer dependencies?

Mixologic’s picture

it's potentially dangerous to keep Composer, the software that you use to resolve your dependencies, in your production environment in such a manner that your production environment can actually execute it.

In other words 'composer dependencies' themselves are not potentially dangerous, but having "composer/composer" itself, as a dependency, is potentially dangerous.

As far as the updating and codebase management aspect, this is a situation where we want a Drupalism. Composer/NPM etc are the 'industry standard best practices'' that we would adopt, and not reinvent. The goal would be to add value to those tools in such a way that gives drupal a distinct competitive advantage, specifically for the market of site builders/non developers/folks without the same resources as "enterprise" etc.

Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'.

daffie’s picture

-1 For using composer in a GUI in Drupal Core.

+1 For using https://electron.atom.io/ as explained by @Mixologic in #2538090-13: Allow the Update Manager to automatically resolve Composer dependencies. That is the solution for this problem! Let @Dries create a initiative for it.

DamienMcKenna’s picture

Might it also be worth reaching out to the companies and organizations that build hosting systems, e.g. see if the CPanel maintainers are looking to add composer support to their system? How about MAMP, WampServer, etc?

jhodgdon’s picture

So... I've been following this issue for a while, because the issue of how to explain Composer in the User Guide came up recently (the User Guide issue is in the Related Issues sidebar here already: #2843440: Discuss Composer and tell how to use it to install some contrib modules). I suggest we step back a bit, and think about what situation we currently have, and what we want to achieve, and then think about how to achieve it. Here is my perspective on the situation/goals/solution:

a) To run a simple Drupal 8 site, as things stand now, it is definitely possible without having Composer installed or even knowing about it. You can download the Drupal Core tarball, unzip it, and go. Same when you update Core.

b) For the vast majority of Drupal 8 contrib modules, you can also operate without Composer: download the tarball, unzip it, and go. Same when you update the contrib module.

c) We have a GUI in Core (in Update Manager module) that allows users to download/unzip files for contrib modules in the GUI, when first installing them or when updating them. IMO this is dangerous, and I personally never set this up or use it, but it is already part of Core and has been since the D7 days. Currently this only works for the modules that work via (b) in Drupal 8.

d) There are a few Drupal 8 contrib modules that rely on 3rd-party libraries that are not distributed with Drupal Core (Geo, maybe Commerce?, ...). Currently, in order to get these modules installed and runnable, you would either need to run Composer on your local dev site and then carefully transfer a bunch of files (vendor? composer.lock? some autoload thing?) from your local to dev site (ideally using a VCS), or you would need to run Composer on the live site directly. Neither of these procedures is easy for our hypothetical "non-expert" user. This is the crux of the current issue.

e) As far as I can tell, the main goal of this issue is to make it so that "non-expert" users can download/update core and *all* contrib modules without having to know about or install Composer. Note that #2477789: Use composer to build sites proposes the opposite: to make it so that no one can build a site (even pure Core) without installing/running Composer.

f) One way to make (e) happen would be to have code in the Update Manager module that, when you are downloading/updating modules via the GUI, would also run Composer update so that their 3rd-party libraries are also downloaded/updated. This would only help people who are already using the (dangerous, IMO) Update Manager module GUI to download/update their contrib module files, but would probably be sufficient to satisfy the goals of this issue.

g) Given that we have, as a community, already decided that (c) is OK to have in Core, adding the capability for (f) doesn't seem much more "dangerous" to me.

h) I don't think we need a new GUI to manage Composer dependencies in general. In fact, I am not convinced this would satisfy (e) if we did. It would instead add another complication for non-expert users to understand and use, which would be a usability problem.

i) As an alternative to (f), another thing we could do to achieve the goals of this module is make it so that, like we do for Drupal Core, the d.o packaging scripts add the 3rd-party dependencies to the tarballs, so that you could download contrib module tarballs and go, just like you can with Core and with contrib modules that don't have dependencies on additional 3rd-party libraries. We have solved this problem for Core, so it should be possible, in principle, to do so with Contrib as well.

Thoughts?

DamienMcKenna’s picture

Do the people who know Symfony's autoloader and Composer better than I do (most of the people in this discussion), what changes would be needed in order to make jhodgdon's idea i work?

daffie’s picture

We can do what this issue want and use the update manager manage composer. For now that will be enough. The problem is that Drupal core and contrib will be using composer more and more. See: #2477789: Use composer to build sites and #2002304: [META] Improve Drupal's use of Composer. Composer is THE dependency manager for the PHP community.
Doing what @jhodgdon suggest will not work in the long run. As an in between solution it will work. But https://electron.atom.io/ or something similar will be the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

Mixologic’s picture

A/B is accurate.

C. There are many modules that rely on third party dependencies, and that number will only grow as more and more essential functionality becomes based off of php libraries that contain solutions that the drupal community should not be re-inventing. Right now, it covers things like 3rd party integration API's, and we are gradually moving to a point where you will not be able to build an effective, functional drupal site without using modules that rely on composer. There are already some modules that are commonly used that fall into this category, such as the address, devel, backup_migrate, colorbox. Right now about 10% of all d8 modules have a 3rd party dependency.

D. is accurate

E. Most of #2477789: Use composer to build sites was "make it possible to use composer to build sites" with only a part of the proposal to "make it require composer". However, in making it possible, we also made it required in some instances. This issue isnt the opposite, its intended to make the composer requirement transparent to end users.

F. Update manager *is* risky. Adding composer to the production environment increases that risk.

G. While composer shares the same concerns of "writing to the executable filesystem" that update manager has, There are additional issues: Composer is a dev application, and contains a significant amount of code that was never written with security in mind that would be executable by production sites. It introduces a considerable attack surface, that we can't really fix, because upstream logic would be "why should we take on the additional maintenance burden of making composer secure because you want to run it in production, which we strongly discourage?"

H. We need a user interface for managing what code your site has because *none* of the current methods that the 'non-expert' class of users are currently used to will continue to work.. You cannot use FTP and tarballs, or update manager to manage a site if you need a module that requires composer.

I. It is not possible to pre-package modules with composer dependencies, because those dependencies are not static. This isn't a problem that is solveable. Composer needs your entire installation to be able to figure out what it needs to download. This is a limitation of PHP itself (two modules might have the same dependency, thus have the same code loaded twice), and not something we should try and find workarounds for.

... the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

^ that quote perfectly summarizes why we really need to do this and why we shouldnt harbor notions like "everybody should be using the command line and following 'best practices' of professional engineering workflows"

rjnagle’s picture

I just wanted to add my two cents as a site implementer (and total drupal newbie) who has had a lot of trouble running composer on the command line for a shared hosting plan.

When I install drupal-composer/drupal-project, I'm getting excessive CPU/memory use which causes the process to be "killed" before it has a chance to install.

If you're aiming at the non-expert crowd with shared hosting plans (and I'm not saying that you have to), it might be worthwhile to measure hits on CPU/memory and to be clear about system requirements for hosting services.

DigitalFrontiersMedia’s picture

re: #15 (@mixologic's second aspect about Composer expecting to be run in PHP CLI) --
Are there any obstacles other than how much memory and timeouts available? What about environmental paths, etc. Running "shell_exec('drush status 2>&1');" from a php script doesn't even run (I don't think) unless PHP is configured to run like PHP CLI with the PATH variable including the drush path and I'd assume the same for composer. But maybe if composer were included in Drupal it would already know the path. But would writing those updated files as the webserver user cause any problems? Wouldn't that mean the webserver user would have to have write access to dependency library files in /vendor?

DigitalFrontiersMedia’s picture

Actually, I guess @mixologic addresses some of my comments in #26 with Point "G" in #24.

However, I do think this would be huge. This:

... the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

^ that quote perfectly summarizes why we really need to do this and why we shouldnt harbor notions like "everybody should be using the command line and following 'best practices' of professional engineering workflows"

is on the money and something I've had very strong concerns about regarding Drupal's direction and adoption for the last few years. I think it's great that Drupal is supporting superior development procedures, but then BetaMax had a higher resolution than VHS and in the end nobody cared except for television film crews.

It sounds like maybe #606592: Allow upgrading core with the update manager and #2367319: Implement automatic background updates for highly critical security issues might want to be added as "related"?

yoroy’s picture

Category: Bug report » Task

Happy FTP user here. Very interested to learn what "H" could look like :)

daffie’s picture

@Mixologic talked a lot about the technical side of why we need https://electron.atom.io/. I would like to look at from a non technical side. Electron is a desktop app that runs on Windows, Mac and Linux. In the desktop app you can create an run websites with web technologies like JavaScript, HTML, and CSS. Electron is also open source project on github and released under the MIT license.

I would like to add some non technical reasons why such a solution would be good for Drupal:

  1. We can create an Electron app so that non-experts and not-yet-experts to compose all Drupal website files they need for their site via a GUI. The non-experts and not-yet-experts should not have use the command line. For me that is asking to much of them. Also as @rjnagle has pointed out: You cannot run composer on shared hosting plan. See #2845379-25: Do not leave non-experts behind: do not require Composer unless a GUI is also included.
  2. The previous point is a start but we can do more. If we add a webserver and a SQLite database to the Electron app, we can let the non-experts and not-yet-experts run their own Drupal website in the Electron app. The Electron app then becomes a staging server for them. They can configure Drupal. They can create content. And then upload it all to their hosting provider. This will be great for (beginner) site-builders.

Why this is good for the Drupal Association:

  1. The uploading of the files, configuration and content to a hosting provider from the by Drupal created Electron app has a big benefit to the Drupal Association. As the "owner" of this Electron app they would be the ones that would be determine which hosting providers will be preselected as good choice. This will make the Drupal Association responsible for the quality controle of the preselected hosting providers. But that will give the Drupal Association the possibility to ask for a recurring fee from the hosting providers for being preselected. An other possibility would be to ask for a percentage from the hosting fee from the hosting provider.
  2. The Electron app would get Drupal to be a solution for the people that now choice Wordpress. Those people will have a modest (for Drupal standards) budget for their website. Their modest budget will not be interesting for traditional Drupal shops. In the Wordpress world there are a lot of paid/freemium plugins (modules) and themes. The people with the modest budget will get a solution with many updates and constant support. The Electron app will need an appstore to support paid/freemium/free modules and themes. This will make the Drupal Association responsible for the quality controle and to keep an eye out for making sure that the deal is "fair" for the non-experts and not-yet-experts. If we take a look at Apple's appstore we see that they take a 30% fee from each sale. I am not saying that that would be the cut that the Drupal Association should take, but with a healthy appstore the Drupal Association should make a profit. The appstore should also have all modules and themes that are now free to download from Drupal.org and that are deemed safe by Drupal's security advisory policy. See https://www.drupal.org/drupalorg/blog/goodbye-project-applications-hello.... Anybody that wants to create a Drupal website without paid/freemium modules and/or themes can do so.
  3. I would also like to point out that such an Electron app would be a great way for non-experts and not-yet-experts to learn Drupal.
    From an absolute beginners level and all the way up to an expert site builder or (front-end) developer. With an Electron app is very easy to start learning Drupal. You download it and open the app and you have started with your first Drupal website. There is no need to install webservers, PHP, databases or any of those things. It is easy for usage in schools and universities, everybody gets their own app. So if someone makes a mess their app, nobody else will have a problem.

The Drupal Association is always complaining that they need money and money in a recurring way would be even better. I think this will solve that problem.

mglaman’s picture

FileSize
1.73 MB

I've been meaning to comment for a few days, but haven't had time to fully read each comment start to finish. So I'll follow up with edits, but I really wanted to post this before I never do.

As borisson_ stated in #16, I have created Conductor as an Electron app UI for working with Composer: https://github.com/mglaman/conductor. It bundles Composer, assuming the user has PHP on their machine (which could be changed in future, I'm sure.)

My biggest blocker, right now, is wireframes for a decent UI/UX and actual feedback from user's on what they'd want it to do. It lets you have a listing of projects, view their dependencies, update the project, and install new projects from Composer project templates. It's not Drupal specific, either.

Here's a gif of its usage:

I have Travis and Appveyor running builds and making artifacts. I have also started to add testing. It gained a lot of initial interest but there hasn't been a user feedback loop. As a Drupal Commerce maintainer, and advocate for D8+Composer, I am motivated to see something like this come to fruition. Tools like SourceTree taught me Git. If something like this helps a newcomer, then ++ to that.

Edit 1: I'm hoping to show this at the MidCamp sprints and get feedback from people there. Maybe someone knows a UI/UX person to guide how it should look.
Edit 2: I'll be in Baltimore and would love to BoF this.

DamienMcKenna’s picture

One thing to consider is that the core maintainers have been unwilling to use Sass for the theming because it would add a non-PHP dependency on working with core. Wouldn't using Electron be the in the same boat, that it's a non-PHP system that core could become dependent upon.

DigitalFrontiersMedia’s picture

As an external management tool, I don't think it would be something that Drupal itself would become dependent on. The same way you could still build and maintain a D7 site without Drush...even if your own personal workflow becomes dependent upon it.

mglaman’s picture

it would add a non-PHP dependency on working with core

As #32 said, it would be just like how people used Acquia Dev Desktop or Drush/Console. Some of the other comments, which represent Mixologic's idea, is to take my simple "Composer only" approach in Conductor and make a community edition of Acquia DD which runs Composer, Drush/Console, and the web server stack ( at least that's what I'm gathering.)

ressa’s picture

Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'.

(#18)
That is probably the best description of the optimal user experience for site builders I have read so far. Thanks for putting it so eloquently Mixologic. It gives me hope that non-Composer experts might still have a place in the Drupal world, if your vision becomes a reality.

cilefen’s picture

8thom’s picture

Throwing up another option which basically allows you to run "simple" composer commands via a browser.
Note there are some limitations to this approach but there's no reason you can't use both CLI & browser...

https://github.com/thom8/drupal-project/tree/build

This currently extends https://github.com/drupal-composer/drupal-project

The "tarball" (zip on GH) - https://github.com/thom8/drupal-project/archive/build.zip could be a replacement for the the normal core download and includes all core dependencies plus exposes a simple web API for running composer commands.

For example `/composer/require/drupal/devel/--/prefer-dist` would run the command `composer require drupal/devel`
see the README for more info.

Would be nice if this could somehow be tied into Drupal auth and only allow access to user 1 similar to update.php.
It could be a drupal module however there could be issues when composer is run to update the codebase.
Rather than returning the console output it could return JSON and have a better UI..

The way I see it, if there's a decent workaround it will allow us all to move forward and embrace composer :)

mglaman’s picture

There's a major problem with #36. Server resources. Composer's dependency resolver eats resources like no other. It might be fine locally. However, one solution would be a remote .lock resolver. There was talk of this at DrupalCon. And it's not a Drupal problem. It's a TYPO3 problem as well, and others. We should really be reaching out to other CMS/Frameworks which have this similar issue.

8thom’s picture

@mglaman you're absolutely right which is why this project is prebuilt and only designed to run simple commands obviously restricted by memory. It was built on a VM with limited memory for testing.

more details here - https://github.com/thom8/drupal-project/tree/build#limitations
At least lack of memory has a relatively simple solution but a lock resolution service would be awesome :)

8thom’s picture

@mglaman do you have any references to the remote .lock resolver?

Could be a simple lambda service which accepts a composer.json & current .lock and returns a new lock file or error.

I'd consider this for CLI too as running composer from Australia or Fiji is painfully slow, only a particular client network it can take ~20 mins for an update of a single package...

Anybody’s picture

I absolutely agree with #34! That would be the best solution and finally would be written against an interface.

8thom’s picture

FileSize
240.1 KB

In the spirit of NIH, PHPStorm has some basic composer GUI functionality.
https://drupalize.me/videos/composer-support-phpstorm?p=2017

Whilst not OSS the EAP version is currently available without license restrictions.
https://confluence.jetbrains.com/display/PhpStorm/PhpStorm+Early+Access+...

With a solution such as #30 you'd still need to use a workflow that builds locally and push deployment artifacts,
so you might get better support from an existing IDE.

PHPStormDrupalComposer

Adrian83’s picture

Having decided to take the plunge and learn command line, composer, and drush a few months ago, I can attest that it's an excruciating hurdle for new users. Just removing the blockers with my host required multiple support tickets and lots of time. I still struggle to get Drupal Console working on new projects, sometimes just giving up. I had strong motivation to do it because our small agency is most comfortable with site-building in Drupal, and someone had to learn a composer-based workflow for a project that required Address. At this point, I wouldn't go back, but I'm not sure that my company has paid off the learning investment with higher efficiency over our old ways of wrangling tar balls.

I would guess that most new non-CLI people wanting to kick the Drupal tires will say "Composer what?" and walk away. To me this is one of the most important issues to the future of the Drupal project.

John_B’s picture

'composer require' (using a composer.json from Drupal VM) does not work on shared hosting with a hard limit of 1GB on PHP 5.6 or 7.0, even using the latest snapshot of Composer which is said to include memory optmization. At least, it did not work for me (details in a blog post.) This is likely to limit the usefulness of a Composer GUI for site builders, assuming the GUI will call for more more memory, not less, than CLI. 'composer require' seems far more memory-hungry than 'composer update'.

Anyway, that point was already made by mglaman made in #37. It cost me a few hours, because I had not realised how hopelessly ineficient Composer is. Perhaps some sane programmer will re-write Composer in C, so that it is not quite such a lumbering beast.

cilefen’s picture

borisson_’s picture

#41:
Suggesting people to install phpstorm is even worse I'd say. It's a huge application that does a lot of things and while the composer support is solid, I feel that would be overkill.

#42:

I would guess that most new non-CLI people wanting to kick the Drupal tires will say "Composer what?" and walk away. To me this is one of the most important issues to the future of the Drupal project.

We can't go back, to not using composer at all. Composer does more than just downloading files (such as writing an autoloader), I agree that this issue is important but keep that in mind.

#43

'composer require' (using a composer.json from Drupal VM) does not work on shared hosting with a hard limit of 1GB on PHP 5.6 or 7.0, even using the latest snapshot of Composer which is said to include memory optmization. At least, it did not work for me (details in a blog post.) This is likely to limit the usefulness of a Composer GUI for site builders, assuming the GUI will call for more more memory, not less, than CLI. 'composer require' seems far more memory-hungry than 'composer update'.

As @mixologic mentions in #18, composer is not built to run on production machines, and having composer on your live servers could potentially lead to security issues. The UI that's being built is intended to be ran on a local machine with the results ready to be copied to the production environment. That copy could be done trough a build process; but can also be done by FTP.

Anyway, that point was already made by mglaman made in #37. It cost me a few hours, because I had not realised how hopelessly ineficient Composer is. Perhaps some sane programmer will re-write Composer in C, so that it is not quite such a lumbering beast.

http://xjmdrupal.org/blog/someone-worked-hard-on-it

The composer-team has done an amazing job of pushing php into a new era of success over the past years, and I'm sure that they all did their best on trying to make it as efficient as possible. While rewriting it in C would make it more memory efficient, it will also make it harder for the target-audience (PHP-developers) to contribute and make it better.


A couple of people have brought up that composer is very resource-intensive, and have mentioned that shared hosting servers can't support it, I think we should communicate clearly, in some way, that composer is intended to be ran locally. Not sure how we'd go about doing that.

jhodgdon’s picture

Regarding one point from #45 -- how to communicate that Composer should be run locally and then the results copied in some manner to the live site... This seems to be very important, and I have a couple of thoughts...

There is quite a bit of Composer documentation on drupal.org now, in this section:
https://www.drupal.org/docs/develop/using-composer
My suggestion would be for this section:

a) Add a sub-page there explaining the workflow of run locally then copy to production, and why you want to do it that way.

b) Go through every page in that section and make sure that this sub-page on local-copy-to-production workflow is mentioned/linked everywhere. The reason this is important is that most likely, people will not start at the top page and read everything in the section. They will most likely reach one of the sub-pages through a web search or a search on drupal.org. So, it's important that this crucial piece of information is linked everywhere; at the same time, we don't want to write it multiple times, which is why I suggest making one page that explains why and how to do it, and adding a short link on each page to this information.

c) We also have two pages in the User Guide that mention composer. I think we need to clarify this point in those pages as well, so I created a separate issue in the User Guide project for that:
#2881438: Recommend running Composer on local not production site

heddn’s picture

I'm not sure I agree with local then push to prod is the /best/ approach. It is an approach. Used by some people. But it isn't the only one that is valid. We should present the option of local => prod as a potential solution for certain sets of problems. Not the best solution in all cases.

jhodgdon’s picture

Are there security issues with running composer on a production site?

8thom’s picture

It's really no different than running drush on a production site, so would suggest it more "best practice" rather than you have to run locally then push to prod.

You could also use an external dev/stage server to resolve dependencies then pull back down to local,
in which case composer isn't required locally or on prod.

Adrian83’s picture

#45

We can't go back, to not using composer at all. Composer does more than just downloading files (such as writing an autoloader), I agree that this issue is important but keep that in mind.

I'm not arguing to "go back," but I strongly believe that for the future of the Drupal project, we need to find a way to both support composer and avoid the requirement that a person must be familiar with command line in order to spin up his first Drupal site.

cilefen’s picture

I saw this on Twitter: Composy.io - Composer as a service I am not representing it as a solution, but it is an interesting idea.

geerlingguy’s picture

This blog post just reached the planet earlier today: Installing Commerce 2.x without Composer, with Ludwig. It looks like something more akin to Libraries API, but more in the spirit of Composer (though without the flexibility or—ahem—composability).

But it is a GUI of sorts, in terms of allowing module installations completely by hand. But it is yet another standard available to Drupal 8 contrib module authors to deal with. Now you have to decide among:

  1. Use Libraries API, use or don't use Composer as well.
  2. Use Ludwig, use or don't use Composer as well.
  3. Use Composer

And there are upsides and downsides to all three of those options.

geerlingguy’s picture

And to be clear—I like the idea behind Ludwig (it's basically what I think Libraries API would/should be in Drupal 8)... but it adds yet another package management solution, meaning some modules might have ludwig.json and composer.json files.

I've already been monitoring the 'has custom Drush commands' vs. 'has custom Drupal Console commands' problem in Drupal 8 contrib... and it's annoying to have to have one site using two separate tools for CLI interaction. If some modules choose Ludwig, some choose Composer, some choose Libraries API, and some choose a combination of the above... that sounds like a DX disaster even worse than what's going on with Drush / Console.

bojanz’s picture

@geerlingguy
There is no way to use Libraries API for external PHP libraries, it doesn't do anything related to the autoloader. A manual way to load a library before using it won't cut it, cause you need access to the external classes in your own services and use statements.

Until today the choice was use Composer, or don't use an external library. Today the choice is between Composer and Ludwig. And we are clearly recommending Composer by default. A module should always have both JSON files (or just composer.json if they don't care about non-Composer users, which is valid).

And there is no combining Ludwig and Composer. You use one or the other.

John_B’s picture

@borisson_ said in #45

As @mixologic mentions in #18, composer is not built to run on production machines, and having composer on your live servers could potentially lead to security issues.

OK. So I won't use it that way. However, requiring a site to be built locally then trasferred to hosting, is on its own enough to ensure that Drupal 8 is taken out of reach of many DIYers.

It may be that the increasing professionalisation in the Drupal world means that this must be faced squarely. On the other hand, saying 'inexperienced website builders need to install composer locally, create a dev site, and upload it to Godaddy or Hostmonster, and they are good to go' does not in my experience suffice to keep Drupal realistically accessible to many of the users who enjoyed D6.

lussoluca’s picture

Hello, I'm the lead developer of composy.io. We think that using Composer is the right way to build a modern PHP application, Drupal 8 included.
We've built composy.io for the people that don't want to use/learn Composer or the cli, but also for people in country where Composer is too slow or in company with firewall/proxy that blocks some network connections.

In this way the choice is to use Composer but without the complexity of Composer.

The base plan will be free of charge forever and the package we provide to download contains a composer.json file so an user can switch to manually use Composer when he want.

In a near future we'll plan to allow users to directly push to some hosting the package built by composy.io

almaudoh’s picture

I like the Ludwig concept as well, there are use ca session for it as well. I use Composer for some complex sites with dependencies, but oftentimes there are projects with just one or two external php libraries that are self-contained and have no other dependencies. Using Composer for those I feel like I'm using an armoured tank when I just need a bicycle to get across the park.

I don't think there needs to be conflict BTW the two. A module would always provide a composer.json by default, but should also provide a ludwig.json if it wants to support installation via Ludwig for less advanced developers?

Scott_McKinnis’s picture

I love the Linux cli, (which really makes sense once you've used it a bit) but composer doesn't seem that intuitive to me, and maybe I haven't used it enough to really say this, but it doesn't seem near as useful as drush was. "drush dl admin_toolbar" is so much less convoluted than "composer require drupal/admin_toolbar". Why not just work with the drush people to make drush more powerful, or build a tool that would be more complimentary to drush?
It seems like this huge tool that I won't use unless I just have to, but at the same time will always be in my face and I'll always be stumbling over it. I've already run into issues where it's using too many resources, and I've had to adjust some things to make it work. I really don't want to use it in production, but is that the next step? Will it be a requirement to even have an instance of Drupal running?
I know there are a lot of really smart people who love this tool, but why can't it be just that a tool that allows me to choose it or not?

ressa’s picture

Comment related to this subject in #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x by xjm:

@moshe weitzman:

Er, thats a long time from now. You still have 18 months to train up folks on Composer, or release a new major version (Drupal 9), or ask Symfony to extend even further. Or you can do nothing - thats fine too. But lets not throw shade at Drush.

I don't mean to throw shade at Drush, although I am really concerned about the stated policy of no longer supporting the sites that need Drush most. This is a shared problem for Drush and core.

Having full Drush support for the Symfony 3 update and solving all the problems was a prerequisite for having Symfony 3 committed to Drupal. I was told by the framework managers that all the Drush problems were solved. I don't know how they got unsolved (catch implied it was a later commit that broke it), or if updating from Drupal 8.3.x to 8.4.x was never tested, or if the connection wasn't made that requiring Drush 9 wouldn't fly because Drush 9 essentially doesn't support most Drupal 8 sites.

And, unfortunately, it's not simply a matter of "training people on Composer". There is deep concern about composer as the primary means of installing Drupal from many audiences, strong enough that people pick other solutions instead of Drupal. Composer is the bee's knees -- for PHP devs and devops people like you and me who can support it -- but for a significant chunk of our current market, it's actually a barrier. Not because they're not "trained". This is offtopic here, but it is a big risk for Drupal right now and finding a way to support both audiences is a problem that's going to take a lot of resources to solve.

18 months is not a long time in a six-month release cycle. As I said, we need to have Symfony 3 in this minor, or the next. The notion of releasing Drupal 9 next year is quite far-fetched when 90% of Drupal sites are still on Drupal 7 and the migration path isn't even done yet.

bojanz’s picture

@Scott_McKinnis
Could we try to not repeat the same 3 points every 5 comments? It is making this issue very unproductive.
All of your questions have been answered several times above.

@ressa
I don't believe the linked issue is relevant. It is about whether CLI users can stand to use another CLI tool. This issue is about not leaving behind users who don't want to use any CLI.

8thom’s picture

This looks interesting - https://downloads.getcomposercat.com/

Windows, MacOS & Linux + SSH support coming soon so you'd be able to run locally to update a remote project.

Mile23’s picture

Want to have GUI for Composer features? These issues remain unreviewed: #2830880: Warn site admins when composer dev dependencies are installed inside of docroot #2494073: Mark modules with unmet composer dependencies uninstallable

Even just +1s would let core developers know you want these features.

xjm’s picture

@bojanz, that is not what the linked issue is about. The linked issue is about Drush 8 not working with the 8.3.x/8.4.x transition while Drush 9 is not released and (later) that even when it is it will not support non-Composer workflows, which is a much more serious and immediate problem than your wording suggests. It's not about what "CLI users" can "stand"; it's about breaking existing sites between minor releases being a non-option.

My comment as quoted was off topic for that issue (as I mentioned), but would certainly be on topic here as this proposal would address the portion of the Drush userbase that wouldn't easily make the switch to a Composer workflow. (Though nothing as yet addresses how existing sites would switch to this new tool, but maybe that's either not part of the MVP or it's just implementation detail.)

xjm’s picture

The very first comment on this issue links #2845378: Stop offering .tar.gz downloads, which is from a user who said this:

However, once you start using composer, the regular methods of keeping the site up-to-date (`drush up`, or extracting the .tar.gz file) break everything. As it apparently is the opinion of the Drupal community that Composer is going to be a requirement, you should stop offering .tar.gz files for download; I now need to figure out how to move my site over to composer in such a way that I can apply security updates, and the existing documentation is really bad about this.

and later

(I've been using Drupal since something like 4.5, and this is *really* frustrating and annoying. I can understand some of the motivations behind moving to composer, but even as a programmer I find it really confusing and poorly documented. I'm now going to lose several days trying to make this work, most likely.)

So, yes, Drush is relevant and Drush's audience is not a monolith.

tanel’s picture

Developer of Composercat here. It's nice to see my project mentioned in this thread. I've had some good feedback from Drupal people, so I wanted to drop by and mention a couple of new features I'm working on that might be of interest to the Drupal community in particular:

1. Currently Composercat only supports searching and displaying extended package information for Packagist packages (you can install any package though, you just need to know the package name if it's not from Packagist), but the next version will have support for custom package repositories, including the Drupal repository (so you can search for Drupal packages the same way as you can with packages from Packagist).
2. The next version will support creating projects from packages, so you can easily start new projects from Drupal project templates.

You can expect the new version to be released in the next couple of weeks.

groovedork’s picture

I am a lowly sitebuilder who got into Drupal because it allowed me to build incredible things via a GUI.

I make a lot of small websites on shared hosting, and often weight Wordpress versus Drupal. I often wish that Drupal would auto-update like Wordpress, so that I wouldn't have to make that choice, as I think Drupal allows me to make better websites.

In my world the biggest problem is not a site that is broken by an update, but a site that is taken over by Russian script kiddies because it wasn't updated.

Right now I often update by
(1) making a backup
(2) just FTP-ing a new Drupal version over the old one.

99% of the time this works fine. In the other cases I roll back the update and do it 'properly'.

I would love a tool that just tries to update the site this way. If it detects that the site is down, it puts back a backup and emails me to do the update manually.

All this reminds me of an older debate: should Drupal have an upgrade path between mayor versions, or should you just migrate? In the end the migrate option won.

Similarly, all the current proposals talk about "how it should be done", and then come up with these complex constructions. But I would hope for a migrate-like "keep it simple" option here. Most of the parts for this 'dirty' tool are already there (backup module, poorman's cron). In 2017 HDD space for backups is cheap, while my time and attention are not.

In the end this seems like a debate between two ideas: "Drupal is a corporate tool" vs "Drupal is a Wordpress competitor". I hope Drupal will (also) continue to be the latter, because Wordpress' limitations become very obvious once you also know Drupal ;-)

cilefen’s picture

Some of what you discussed is being thought about on #2367319: Implement automatic background updates for highly critical security issues.

P.S. I will never understand why anyone thinks site building is "lowly".

imclean’s picture

@groovedork,

All this reminds me of an older debate: should Drupal have an upgrade path between mayor versions, or should you just migrate? In the end the migrate option won.

As of Drupal 8 all updates are in place. The first version of Drupal 9 is planned to be identical to the last version of Drupal 8 with all the deprecated functions/API removed. This gives developers time to remove deprecated code prior to Drupal 9, allowing an in place update.

Adrian83’s picture

@groovedork I agree with you that Wordpress's 1-click core update for core is a major selling point for them for smaller sites and agencies. We too are using Drupal to build sites for which many people would argue that Wordpress is the better fit. I am hoping this issue will in the future allow non-technical site-builders to manage updates for both core and modules in a simple, non-command-line way.

diego.paroni’s picture

My sad impression, using Drupal 8, is that the difficulty, compared to drupal 7, will kick most of the less experienced and less enthusiastic users. Drupal will only be used by programmers for complex projects and will be abandoned by agencies. Consequently, the Drupal community will shrink and slowly die.
I'm sorry to say this, I use Drupal since version 5, but that's what I think it will happen.

sprite’s picture

After a few months of using composer to maintain multiple Drupal 8 development instances, the advantages of a composer style dependency management subsystem have become clear. From the command line, "composer update" correctly updates all the various dependencies in a Drupal 8 instance with a single command. A shell script can be written that updates an entire set of Drupal 8 instances on a server all with a single command as well.

However, command line interaction with a software system is palatable only to the developer demographic, but is an experience that alienates "agency" level website builders (designers), and alienates the clients they leave their projects to even more. In 2017, small business project clients in particular, without formal IT departments, simply won't tolerate a website/application installation that requires them to run command line scripts to perform security updates on their site's code.

What I believe is that a "headless composer" must be developed that can be integrated into Drupal core, and integrated into the Drupal 8 admin GUI, so that Drupal 8 can have a GUI "click and go" user-interface for intelligently updating core and/or modules, with underlying composer level dependency management functionality. Without this fundamentally essential GUI functionality, the future audience for Drupal will be limited to a technological audience, that many believe will erode the Drupal user-base and audience in time.

As others have noted above, GUI based core updating and module installation and updating, are a feature essential to Drupal 8's future viability and marketplace health.

As others have noted as well, this requirement is both a technological one, and a Drupal marketing requirement, that is also impacted the unwillingness of many hosting companies to configure their servers for client SSH command line access (not to mention the core resource requirements of Drupal 8). Not everybody is a developer running on a VPS or dedicated server with root access and total control.

GUI code dependency, code installation, and code update, management are fundamental to Drupal's future.

Adrian83’s picture

@sprite You said it better than I could have said it myself!

robertmaynord’s picture

I have been using Drupal for many years for several web sites. I am not a programmer, but I am comfortable with the command line because I use Linux at work (networking, servers, package installation, etc). I have maintained a Drupal 7 small business web site for years. I recently decided to build a new site using Drupal 8 and Commerce. The install looked totally easy because of Composer. I built a stand-alone test box using Ubuntu, and the install worked fine. I then decided to build an on-line Drupal 8 site, parallel to my Drupal 7 site. After about 10 days of work, I have had zero success. Here are some points about my experience.
1) I knew that I would need Composer, so I requested that my ISP install it on my account. They did so, without hesitation.
2) I attempted some basic Composer commands and they worked fine.
3) I attempted to install Drupal Commerce, but the command line simply froze. I ran Composer Diagnose, and it said that allow_url_fopen=0 was stopping the process. My ISP set up allow_url_fopen=1 in a php.ini file in my home directory, and they said that should fix the problem. However, I found that it opened up Composer only for the web interface, not the SSH interface!!
4) My ISP boosted the problem up to a higher tech support level, they consulted a PHP specialist who recommended that I use -c
| to bypass the server restriction on allow_url_fopen. I tried this, but the level 3 techs have evidently blocked allow_url_fopen in the shell for the entire server, not just my account. This is for security reasons. Without allow_url_fopen= set to "On" or "1", Composer is useless.

So, my point is this: I am a long time Drupal user who simply wants to upgrade to Drupal 8. I am comfortable with both the command line and Composer. At this point I have not found a way. A built-in Drupal Composer GUI might be the best solution, but it is not necessary for me, and it seems like a long way off. My next step is to explore the idea of developing the web site offline with Composer, and FTP-ing (or some form of syncing) back and forth between the offline and online machines. It may be do-able, but has substantial potential for challenges along the way. A clear systematic approach to building a Drupal 8 web site (with Composer) offline, and maintaining the regular updates over time (back and forth without data loss) to the live server would be quite helpful....

sumanthkumarc’s picture

agree with @sprite on comment #72