One of the items mentioned during some of Dries' keynotes is that there's a danger of Drupal leaving beginners and site builder behind. A major step in the current march forward is Composer usage. While Composer is a great tool and should become the standard for building complex projects, it should not be required to build a Drupal site unless a GUI is provided; additionally it should always be possible to download files from d.o, put them in the correct location (after extracting the archive) and have it work.

This should be seen as an opposition to #2477789: Use composer to build sites becoming a requirement unless a GUI is provided.

Files: 
CommentFileSizeAuthor
#30 conductor.gif1.73 MBmglaman

Comments

DamienMcKenna created an issue. See original summary.

cilefen’s picture

bojanz’s picture

it should always be possible to download files from d.o, put them in the correct location (after extracting the archive) and have it work.

We can wish for this all we want, but it's technologically impossible due to the way Composer is built.

Mixologic and I discussed this, and agreed that the right solution is to offer a rewritten Update Manager that embeds Composer (basically shipping with updater/ next to core/). That way people either use Composer, or they use the UI which runs Composer and then moves the changes into place. We also get core updates for free.
#2538090: Allow the Update Manager to automatically resolve Composer dependencies.

DamienMcKenna’s picture

@bojanz: Thanks for adding that insight. An updated Update Manager would be completely fine, we just can't ignore the fact that a large portion of visitors *don't* run the command line, and that requiring it would cause Drupal to loose a lot of users.

Jaypan’s picture

That way people either use Composer, or they use the UI which runs Composer and then moves the changes into place.

The problem here is that it still requires composer. Most shared hosting servers do not have it installed by default, so to add composer requires knowledge of how to install it - and we're back to the command line. So while having GUI access is definitely a step in the right direction, users who cannot use the command line are still cut off.

cilefen’s picture

@Jaypan: did you read the part of comment #3 about embedding composer?

Jaypan’s picture

No, I missed that. That's a good solution. It actually covers the biggest issue with composer as I see. That would also be nice since non-technical users would then be able to install drush more easily as well.

dww’s picture

Agreed that we don't want to force anyone to have to use the shell to deploy/maintain Drupal.

Also agreed that the plan in #3 is the right way to rely on Composer to solve the problems it does without forcing end users to know it's even happening.

Part of how authorize.php was originally designed was to let it become a more generic framework for running something like composer as the user that owns the files in the filesystem. I'm not at all attached to keeping authorize.php as-is, but I definitely support any effort that allows it or something like it to run composer for site admins.

The installer could also use something like authorize.php to invoke Composer to bootstrap contrib installs during initial setup of a new site. There's talk of a module browser in core, etc. All of this could be UI around ways to get "update manager" or "install manager" or whatever we call it, to feed composer commands to the filesystem via authorize.php or its offspring.

Cheers,
-Derek

jhodgdon’s picture

cilefen’s picture

Maybe. But #2538090 needs an issue summary update because #38 explains it is probably untenable as written.

DamienMcKenna’s picture

Title: Do not require composer for building sites, do not leave non-experts behind » Add a GUI for Composer, do not require composer for building sites, do not leave non-experts behind
Issue summary: View changes

Clarifying that having a working GUI for Composer would make this a non-issue.

DamienMcKenna’s picture

Title: Add a GUI for Composer, do not require composer for building sites, do not leave non-experts behind » Do not leave non-experts behind: do not require Composer unless a GUI is also included
webchick’s picture

Priority: Normal » Major

Escalating, based on impact.

timmillwood’s picture

For background the lead up to this issue is at http://buytaert.net/making-drupal-upgrades-easy-forever#comment-131086

Mixologic’s picture

There's a couple of constraints that we want to consider when designing a system that will allow us to abstract away the complexities of using composer:

Composer is designed to be a build time/development tool, and is not intended to exist or be available in production systems. What that means in practice is that the normal constraints of "code outside the firewall and open to the internet" are not baked into composer, and thus it could be considered a security risk to have it available on a production system that interfaces with threats. Thats not to say it cannot be on live sites, just that upstream, the developers are not likely to focus on making it hard to compromise. Even if we were to audit it now and find that its fine, that doesn't mean that the next version wont have some obvious remote code execution vulnerability baked into it.

The second aspect of composer is that it is designed to be executed in a command line php environment - which means it is expecting to have the ability to consume considerable amounts of memory, and take significantly longer execution time than most webserver php environments are geared for. Drupal 8's 64MB minimum requirement would probably need to bump to 256MB or higher, and we might need as much as 180 seconds worth of timeout for php. Or more.

I bring up these two points to underscore the difficulty we may encounter in attempting to bundle composer within the drupal application itself. There are likely ways we can cordon it off for security purposes, and ways we can ensure its only running on a cron or something similar. Another alternative is to construct a site management tool that is separate from your web app, much like wordpress's desktop administration application (https://developer.wordpress.com/calypso/).

Composer isn't the only sitebuilding barrier, and we ought to address the "Constructing, Extending, Updating, and Maintaining" the codebase your website needs to run in a holistic manner - so that we can handle any other "Buildtime/Development tools"- i.e. we can add build tools like NPM or bower to solve the 'gather front end assets too' so as to provide a seamless, delightful experience for drupal users who do not fall into the classification of "working on enterprise sites for enterprise clients"

I also strongly believe that we should start by designing the ideal user experience we would like to see in a tool like this, and aim high enough to knock it out of the park.

borisson_’s picture

I bring up these two points to underscore the difficulty we may encounter in attempting to bundle composer within the drupal application itself. There are likely ways we can cordon it off for security purposes, and ways we can ensure its only running on a cron or something similar. Another alternative is to construct a site management tool that is separate from your web app, much like wordpress's desktop administration application (https://developer.wordpress.com/calypso/).

I agree with @Mixologic and matt glaman and I already started work on a tool that does this. I started by building https://github.com/borisson/composer-ui and a couple of weeks later matt built https://github.com/mglaman/conductor. The goal I had in mind (and that matt largely followed) is to make a tool to install composer dependencies for any composer-based project (including drupal, but also symfony, laravel and others). The biggest thing still to do there is to add the ability to add/remove projects. Currently it only does install and update.

I think if we promote and improve that tool we'd get pretty close to what calypso can do. Adding the ability to use npm dependency resolution would be awesome, but I think that'd be a lot hard to get right. That's why I'd leave that for a 2.x release.

We had a BOF about this at drupalcon dublin but only a couple of people showed up, probably because I named it "composer UI BOF" and not something like: "Installing modules from the desktop", I can't really remember a lot of things from that BOF, I forgot to take notes, but maybe klausi remembers more. I created a couple of issues in the queue of composer-ui as a result of that discussion:
- https://github.com/borisson/composer-ui/issues/12
- https://github.com/borisson/composer-ui/issues/11
- https://github.com/borisson/composer-ui/issues/4

phenaproxima’s picture

For my part, I'm not against having a UI to soften the steep Composer learning curve. In fact, I think it's a great idea -- as long as it's not Drupal-specific. When it comes to code base and update management, what I'm in favor of is dropping as many Drupalisms as possible. We should be taking advantage of everything Composer can do for us, and fill in the gaps (i.e., a UI) where it could be made easier and friendlier for Drupal users. And if our good work can move upstream, good.

The second aspect of composer is that it is designed to be executed in a command line php environment - which means it is expecting to have the ability to consume considerable amounts of memory, and take significantly longer execution time than most webserver php environments are geared for.

If we built a stand-alone Composer UI, maybe we could bundle it with a preconfigured copy of the PHP interpreter with higher exec time and memory limits, to sorta work around this problem.

What that means in practice is that the normal constraints of "code outside the firewall and open to the internet" are not baked into composer, and thus it could be considered a security risk to have it available on a production system that interfaces with threats.

Please correct me if I'm misunderstanding, but doesn't this just mean that it's potentially dangerous to keep Composer in your production environment? If so, how does that differ from the situation now, supporting modules that have Composer dependencies?

Mixologic’s picture

it's potentially dangerous to keep Composer, the software that you use to resolve your dependencies, in your production environment in such a manner that your production environment can actually execute it.

In other words 'composer dependencies' themselves are not potentially dangerous, but having "composer/composer" itself, as a dependency, is potentially dangerous.

As far as the updating and codebase management aspect, this is a situation where we want a Drupalism. Composer/NPM etc are the 'industry standard best practices'' that we would adopt, and not reinvent. The goal would be to add value to those tools in such a way that gives drupal a distinct competitive advantage, specifically for the market of site builders/non developers/folks without the same resources as "enterprise" etc.

Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'.

daffie’s picture

-1 For using composer in a GUI in Drupal Core.

+1 For using https://electron.atom.io/ as explained by @Mixologic in #2538090-13: Allow the Update Manager to automatically resolve Composer dependencies. That is the solution for this problem! Let @Dries create a initiative for it.

DamienMcKenna’s picture

Might it also be worth reaching out to the companies and organizations that build hosting systems, e.g. see if the CPanel maintainers are looking to add composer support to their system? How about MAMP, WampServer, etc?

jhodgdon’s picture

So... I've been following this issue for a while, because the issue of how to explain Composer in the User Guide came up recently (the User Guide issue is in the Related Issues sidebar here already: #2843440: Discuss Composer and tell how to use it to install some contrib modules). I suggest we step back a bit, and think about what situation we currently have, and what we want to achieve, and then think about how to achieve it. Here is my perspective on the situation/goals/solution:

a) To run a simple Drupal 8 site, as things stand now, it is definitely possible without having Composer installed or even knowing about it. You can download the Drupal Core tarball, unzip it, and go. Same when you update Core.

b) For the vast majority of Drupal 8 contrib modules, you can also operate without Composer: download the tarball, unzip it, and go. Same when you update the contrib module.

c) We have a GUI in Core (in Update Manager module) that allows users to download/unzip files for contrib modules in the GUI, when first installing them or when updating them. IMO this is dangerous, and I personally never set this up or use it, but it is already part of Core and has been since the D7 days. Currently this only works for the modules that work via (b) in Drupal 8.

d) There are a few Drupal 8 contrib modules that rely on 3rd-party libraries that are not distributed with Drupal Core (Geo, maybe Commerce?, ...). Currently, in order to get these modules installed and runnable, you would either need to run Composer on your local dev site and then carefully transfer a bunch of files (vendor? composer.lock? some autoload thing?) from your local to dev site (ideally using a VCS), or you would need to run Composer on the live site directly. Neither of these procedures is easy for our hypothetical "non-expert" user. This is the crux of the current issue.

e) As far as I can tell, the main goal of this issue is to make it so that "non-expert" users can download/update core and *all* contrib modules without having to know about or install Composer. Note that #2477789: Use composer to build sites proposes the opposite: to make it so that no one can build a site (even pure Core) without installing/running Composer.

f) One way to make (e) happen would be to have code in the Update Manager module that, when you are downloading/updating modules via the GUI, would also run Composer update so that their 3rd-party libraries are also downloaded/updated. This would only help people who are already using the (dangerous, IMO) Update Manager module GUI to download/update their contrib module files, but would probably be sufficient to satisfy the goals of this issue.

g) Given that we have, as a community, already decided that (c) is OK to have in Core, adding the capability for (f) doesn't seem much more "dangerous" to me.

h) I don't think we need a new GUI to manage Composer dependencies in general. In fact, I am not convinced this would satisfy (e) if we did. It would instead add another complication for non-expert users to understand and use, which would be a usability problem.

i) As an alternative to (f), another thing we could do to achieve the goals of this module is make it so that, like we do for Drupal Core, the d.o packaging scripts add the 3rd-party dependencies to the tarballs, so that you could download contrib module tarballs and go, just like you can with Core and with contrib modules that don't have dependencies on additional 3rd-party libraries. We have solved this problem for Core, so it should be possible, in principle, to do so with Contrib as well.

Thoughts?

DamienMcKenna’s picture

Do the people who know Symfony's autoloader and Composer better than I do (most of the people in this discussion), what changes would be needed in order to make jhodgdon's idea i work?

daffie’s picture

We can do what this issue want and use the update manager manage composer. For now that will be enough. The problem is that Drupal core and contrib will be using composer more and more. See: #2477789: Use composer to build sites and #2002304: [META] Improve Drupal's use of Composer. Composer is THE dependency manager for the PHP community.
Doing what @jhodgdon suggest will not work in the long run. As an in between solution it will work. But https://electron.atom.io/ or something similar will be the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

Mixologic’s picture

A/B is accurate.

C. There are many modules that rely on third party dependencies, and that number will only grow as more and more essential functionality becomes based off of php libraries that contain solutions that the drupal community should not be re-inventing. Right now, it covers things like 3rd party integration API's, and we are gradually moving to a point where you will not be able to build an effective, functional drupal site without using modules that rely on composer. There are already some modules that are commonly used that fall into this category, such as the address, devel, backup_migrate, colorbox. Right now about 10% of all d8 modules have a 3rd party dependency.

D. is accurate

E. Most of #2477789: Use composer to build sites was "make it possible to use composer to build sites" with only a part of the proposal to "make it require composer". However, in making it possible, we also made it required in some instances. This issue isnt the opposite, its intended to make the composer requirement transparent to end users.

F. Update manager *is* risky. Adding composer to the production environment increases that risk.

G. While composer shares the same concerns of "writing to the executable filesystem" that update manager has, There are additional issues: Composer is a dev application, and contains a significant amount of code that was never written with security in mind that would be executable by production sites. It introduces a considerable attack surface, that we can't really fix, because upstream logic would be "why should we take on the additional maintenance burden of making composer secure because you want to run it in production, which we strongly discourage?"

H. We need a user interface for managing what code your site has because *none* of the current methods that the 'non-expert' class of users are currently used to will continue to work.. You cannot use FTP and tarballs, or update manager to manage a site if you need a module that requires composer.

I. It is not possible to pre-package modules with composer dependencies, because those dependencies are not static. This isn't a problem that is solveable. Composer needs your entire installation to be able to figure out what it needs to download. This is a limitation of PHP itself (two modules might have the same dependency, thus have the same code loaded twice), and not something we should try and find workarounds for.

... the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

^ that quote perfectly summarizes why we really need to do this and why we shouldnt harbor notions like "everybody should be using the command line and following 'best practices' of professional engineering workflows"

rjnagle’s picture

I just wanted to add my two cents as a site implementer (and total drupal newbie) who has had a lot of trouble running composer on the command line for a shared hosting plan.

When I install drupal-composer/drupal-project, I'm getting excessive CPU/memory use which causes the process to be "killed" before it has a chance to install.

If you're aiming at the non-expert crowd with shared hosting plans (and I'm not saying that you have to), it might be worthwhile to measure hits on CPU/memory and to be clear about system requirements for hosting services.

DigitalFrontiersMedia’s picture

re: #15 (@mixologic's second aspect about Composer expecting to be run in PHP CLI) --
Are there any obstacles other than how much memory and timeouts available? What about environmental paths, etc. Running "shell_exec('drush status 2>&1');" from a php script doesn't even run (I don't think) unless PHP is configured to run like PHP CLI with the PATH variable including the drush path and I'd assume the same for composer. But maybe if composer were included in Drupal it would already know the path. But would writing those updated files as the webserver user cause any problems? Wouldn't that mean the webserver user would have to have write access to dependency library files in /vendor?

DigitalFrontiersMedia’s picture

Actually, I guess @mixologic addresses some of my comments in #26 with Point "G" in #24.

However, I do think this would be huge. This:

... the solution for the non-expert and the not-yet-expert in the composer Drupal world. The last ones will become the Drupal experts in the future.

^ that quote perfectly summarizes why we really need to do this and why we shouldnt harbor notions like "everybody should be using the command line and following 'best practices' of professional engineering workflows"

is on the money and something I've had very strong concerns about regarding Drupal's direction and adoption for the last few years. I think it's great that Drupal is supporting superior development procedures, but then BetaMax had a higher resolution than VHS and in the end nobody cared except for television film crews.

It sounds like maybe #606592: Allow upgrading core with the update manager and #2367319: Implement automatic background updates for highly critical security issues might want to be added as "related"?

yoroy’s picture

Category: Bug report » Task

Happy FTP user here. Very interested to learn what "H" could look like :)

daffie’s picture

@Mixologic talked a lot about the technical side of why we need https://electron.atom.io/. I would like to look at from a non technical side. Electron is a desktop app that runs on Windows, Mac and Linux. In the desktop app you can create an run websites with web technologies like JavaScript, HTML, and CSS. Electron is also open source project on github and released under the MIT license.

I would like to add some non technical reasons why such a solution would be good for Drupal:

  1. We can create an Electron app so that non-experts and not-yet-experts to compose all Drupal website files they need for their site via a GUI. The non-experts and not-yet-experts should not have use the command line. For me that is asking to much of them. Also as @rjnagle has pointed out: You cannot run composer on shared hosting plan. See #2845379-25: Do not leave non-experts behind: do not require Composer unless a GUI is also included.
  2. The previous point is a start but we can do more. If we add a webserver and a SQLite database to the Electron app, we can let the non-experts and not-yet-experts run their own Drupal website in the Electron app. The Electron app then becomes a staging server for them. They can configure Drupal. They can create content. And then upload it all to their hosting provider. This will be great for (beginner) site-builders.

Why this is good for the Drupal Association:

  1. The uploading of the files, configuration and content to a hosting provider from the by Drupal created Electron app has a big benefit to the Drupal Association. As the "owner" of this Electron app they would be the ones that would be determine which hosting providers will be preselected as good choice. This will make the Drupal Association responsible for the quality controle of the preselected hosting providers. But that will give the Drupal Association the possibility to ask for a recurring fee from the hosting providers for being preselected. An other possibility would be to ask for a percentage from the hosting fee from the hosting provider.
  2. The Electron app would get Drupal to be a solution for the people that now choice Wordpress. Those people will have a modest (for Drupal standards) budget for their website. Their modest budget will not be interesting for traditional Drupal shops. In the Wordpress world there are a lot of paid/freemium plugins (modules) and themes. The people with the modest budget will get a solution with many updates and constant support. The Electron app will need an appstore to support paid/freemium/free modules and themes. This will make the Drupal Association responsible for the quality controle and to keep an eye out for making sure that the deal is "fair" for the non-experts and not-yet-experts. If we take a look at Apple's appstore we see that they take a 30% fee from each sale. I am not saying that that would be the cut that the Drupal Association should take, but with a healthy appstore the Drupal Association should make a profit. The appstore should also have all modules and themes that are now free to download from Drupal.org and that are deemed safe by Drupal's security advisory policy. See https://www.drupal.org/drupalorg/blog/goodbye-project-applications-hello.... Anybody that wants to create a Drupal website without paid/freemium modules and/or themes can do so.
  3. I would also like to point out that such an Electron app would be a great way for non-experts and not-yet-experts to learn Drupal.
    From an absolute beginners level and all the way up to an expert site builder or (front-end) developer. With an Electron app is very easy to start learning Drupal. You download it and open the app and you have started with your first Drupal website. There is no need to install webservers, PHP, databases or any of those things. It is easy for usage in schools and universities, everybody gets their own app. So if someone makes a mess their app, nobody else will have a problem.

The Drupal Association is always complaining that they need money and money in a recurring way would be even better. I think this will solve that problem.

mglaman’s picture

FileSize
1.73 MB

I've been meaning to comment for a few days, but haven't had time to fully read each comment start to finish. So I'll follow up with edits, but I really wanted to post this before I never do.

As borisson_ stated in #16, I have created Conductor as an Electron app UI for working with Composer: https://github.com/mglaman/conductor. It bundles Composer, assuming the user has PHP on their machine (which could be changed in future, I'm sure.)

My biggest blocker, right now, is wireframes for a decent UI/UX and actual feedback from user's on what they'd want it to do. It lets you have a listing of projects, view their dependencies, update the project, and install new projects from Composer project templates. It's not Drupal specific, either.

Here's a gif of its usage:

I have Travis and Appveyor running builds and making artifacts. I have also started to add testing. It gained a lot of initial interest but there hasn't been a user feedback loop. As a Drupal Commerce maintainer, and advocate for D8+Composer, I am motivated to see something like this come to fruition. Tools like SourceTree taught me Git. If something like this helps a newcomer, then ++ to that.

Edit 1: I'm hoping to show this at the MidCamp sprints and get feedback from people there. Maybe someone knows a UI/UX person to guide how it should look.
Edit 2: I'll be in Baltimore and would love to BoF this.

DamienMcKenna’s picture

One thing to consider is that the core maintainers have been unwilling to use Sass for the theming because it would add a non-PHP dependency on working with core. Wouldn't using Electron be the in the same boat, that it's a non-PHP system that core could become dependent upon.

DigitalFrontiersMedia’s picture

As an external management tool, I don't think it would be something that Drupal itself would become dependent on. The same way you could still build and maintain a D7 site without Drush...even if your own personal workflow becomes dependent upon it.

mglaman’s picture

it would add a non-PHP dependency on working with core

As #32 said, it would be just like how people used Acquia Dev Desktop or Drush/Console. Some of the other comments, which represent Mixologic's idea, is to take my simple "Composer only" approach in Conductor and make a community edition of Acquia DD which runs Composer, Drush/Console, and the web server stack ( at least that's what I'm gathering.)

ressa’s picture

Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'.

(#18)
That is probably the best description of the optimal user experience for site builders I have read so far. Thanks for putting it so eloquently Mixologic. It gives me hope that non-Composer experts might still have a place in the Drupal world, if your vision becomes a reality.

cilefen’s picture