Related to #2477789: Use composer to build sites:

If the Update Manager is used to add a module that has Composer dependencies, it would be very nice if it could automatically run Composer to resolve/download those dependencies to the Drupal site. That way the site administrator can turn the module on via the user interface immediately, rather than having to go to the command line and muck around with things there to meet all the module's requirements.

How this works (and whether it can work) depends on the method Update Manager uses to download the project:

  1. Writable docroot method. Since Update Manager already has permission to write to the docroot in this case (and already uses that permission to add the module itself) it should just be able to run Composer directly. According to Jason Judge's comment:

    Just FYI, you can install the non-phar version of composer and access all its functions through its API, so it can be "driven" from the back-end rather than the command-line if it comes to that.

    Which sounds like the right way to do it.

  2. FTP method. Probably not possible to do anything in this case. (Edit: Or actually, would it be possible to run Composer on a copy of the module in the site's /tmp directory, and then transfer the final result over FTP?)
  3. SSH-over-PHP method. This is the most secure way of using the Update Manager, but almost nobody uses it in practice due to server requirements. In this case since there is a SSH tunnel to the site (with the SSH user having the ability to write to the docroot) it would be possible to run Composer via the command line.

TBD: What happens if there are multiple submodules in the downloaded project, and each one has separate Composer dependencies? (Is that even possible?) The simplest answer is probably just to resolve all of them - that way the necessary code is in place in the filesystem regardless of which module(s) the site administrator winds up turning on.

Comments

tlyngej’s picture

Composer will throw an error if a conflict between two requirements are detected. Can't Drupal not "just" inherit that behavior, and notify the user about the problem, asking him to choose whether he want one module or the other?

An extension to this could be some sort of automated system that could inform the maintainers of the two module, that they might want to get this solved, by, for example, aligning the version of the PHP library.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

David_Rothstein’s picture

Issue summary: View changes
bojanz’s picture

I don't think there's a reason for this issue to be postponed.

Yes, Update Manager should be rewritten to embed Composer and use it for all downloads (download into sites/default/files/$somedir then move into place).
That fixes both the libraries issue and the core updates issue.
We might want to ship it in a separate dir (updater/ next to core/) if that makes it easier for the code to deploy changes.
We can look at this for inspiration: https://github.com/mothership-ec/up

My main worry is that this will end up being not-good-enough, due to Composer's memory requirements being too large for most shared hosting, making it necessary to run this locally, in which case it makes more sense to have the updater be an Electron app.

David_Rothstein’s picture

Issue summary: View changes
Status: Postponed » Active
David_Rothstein’s picture

Version: 8.2.x-dev » 8.3.x-dev

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dww’s picture

+1000. As the once-upon-a-time maintainer of update.module, I heartily support this issue.

philsward’s picture

See https://www.drupal.org/node/2477789#comment-11920071 for additional inspiration (hopefully). That comment really belongs over here, however was directed to the discussion in that issue.

Oh, one thought is to have this type of system allow modules to define "when" their dependencies get updated on cron runs. For example, a module knows the development on a dependency isn't very frequent, have it check once a month for updates. On the flip side, Commerce being in beta, could specify that its dependencies get checked and updated once a day. When Commerce goes to stable, it scales backs those checks to once a week. Might help out a bit with the Composer overhead problem by not running composer against every dependency? Dunno... ideas...

Mixologic’s picture

Executing composer as part of a web request is unlikely to be successful. composer can take anywhere from 20 seconds to three minutes to complete so will run into php timeouts, and can take a *lot* of memory to run, so the best we could hope for with something in update manager is having something added to a queue, and having it execute on cron.

There isn't a way to make composer work on just parts of the application - composer calculates all of the requirements for the whole application at once.

I've been thinking a lot about this issue, and it goes far beyond composer. What drupal is currenly lacking is a clean abstraction layer over the top of 'sitebuilding'. A separate electron app is probably the best way to go here, but it needs to handle composer, and bower, and npm and whatever other forms of dependency management and asset construction we can put into it.

philsward’s picture

@Mixologic

If composer is slow and resource heavy, is it worth trying to get some of the Drupal execs to contact the composer folks and try to help figure out how to reduce it's consumption load?

That said, is composer written and designed specifically to be used locally? (as opposed to a remote staging or live server)

If we can't make composer work better, is it worth leaving composer to be a 'developer only' tool used locally and simply create a composer style dependency manager specifically written for Drupal but still kind of acts like composer and still uses packagist?

Mixologic’s picture

I've spent a fair amount of time in meetings with Jordi (the creator of composer) when he was helping advise us on building packages.drupal.org , and the issue at hand is not really a communications, engineering, or design problem with composer.

Composer is slow and resource heavy because it has to be. The job that it automates is fourfold, and its complicated. Like, the hardest problem in all of computer science complicated.

1. It needs to gather all of the metadata about all the components of your project (core, modules, php libraries, themes etc), including new pieces you are asking to install. - This gets data from your local filesystem (composer.json/composer.lock), gathering data from packagist.org, gathering data from packages.drupal.org, and gathering data from any other repositories you may have told it to use (maybe a local repo for custom modules etc). The only pieces of data that matters in all of this are what each component depends on, and which versions of those components will work. This is pretty much the same thing as a database query to get this data, except composer is hitting api's and caching things etc.

2. Once it has all of that data, it has to run an SAT solver on it. This is the hard part. In a world where a "module" had all of the functionality you needed, and everything was encapsulated inside and you didn't have to worry about overlap between modules, this used to be easy. But in a more modern world, nobody wants to rewrite functionality thats already written, and things become more and more overlappy. One module might use the aws sdk to stash files on S3 at aws, and a different module might want that same sdk to automatically run your email through amazons email service. Anyhow, point being is that once you have every single component, and all of their dependencies, and dependencies of dependencies and you have to figure out if there is a single set of versions that makes everyone happy - well, thats another example of a classic comp sci problem called the traveling salesman. Nobody has ever solved it, so it's hard and slow. Sometimes its easy and fast, but you cant know in advance. This is what takes up all the memory, attempts at making it faster.

3. Once if finally *does* know exactly which versions are needed, then It needs to download all of the files you are trying to add - the module, its other composer deps, those composer deps deps ad infinitum until you actually *have* everything. This might include upgrading some things as well. Downloading files from the internet can be slow, so again, no guarantee that can happen before php times out.

4. Finally, once everything is figured out, and in place, then composer generates one last PHP file, the Autoloader, so that we dont have to pull in every scrap of php into memory when we're running requests. It can grab them as they are asked for. And the autoloader essentially hands the application a mapping of files on disk to classnames being used. So generating a file, and writing it to a place that your webserver can execute, *from* your webserver itself is generally considered about as secure as a balsawood box full of bananas in a room full of chimpanzees.

So, all that being said, *none* of that is stuff that everybody should have to care about, and all of it implementation details. Which is why this:

simply create a composer style dependency manager specifically written for Drupal but still kind of acts like composer and still uses packagist?

Other than the "simply" part, that is essentially what we are talking about doing, except it would use composer under the hood. An electron app is really just a browser in disguise (its what wordpress's admin tool is, slack is written in electron, and others: http://electron.atom.io/). So the idea would be "heres a tool that lets you get everything you need for your site in one place, and then it hands you a file or files to upload to your server"

My main suggestion in this is that composer isn't the only problem we should solve if we're going to do that, we should look at the same dependency problem that the front end faces (js assets, webpack, bower, npm, etc) and solve it all at once.

Anyhow, your pain is strongly justified, and not isolated - Im hearing a need for something like this from a lot of corners of the drupalsphere. Drupal 7->8 made an engineering tradeoff for better, more battle tested software engineering practices and less likely to break code, and in the process sacrificed some of the ease of site construction. For the people building sites with plenty of resources, that was fine. They have programmers/developers for whom things like composer were familiar, but for anybody without the resources to follow the same "best practices", that was a bummer.

But now we have an opportunity to make something great, that can address these issues, and put a layer of user experience into the site building process that is definitely lacking from where we are now.

daffie’s picture

+1000 For the input from Mixologic! I have no idea how much work it is to create what he is suggesting in comment #13. But I think that it would help a lot of people that are not very technical savvy.

philsward’s picture

Thanks @Mixologic. Awesome breakdown. I wasn't sure where the electron app came into the picture so that makes sense now.

D8 was definitely the developers build. I'd be lying if I said I didn't feel alienated by D8 as a site builder so hearing that this issue is being discussed with site builders in mind is exciting. I really do hope to see Drupal have an update system that can deal with core and contrib, as well as security issues, automatically on a schedule.

philsward’s picture

Oh, I'll point out another use case to drive the point home for this need... (Maybe I'm just preaching to the choir)

I played around with Commerce over the weekend and ran into an issue with a view blowing up because I removed some relationships of fields that were dependent on them, resulting in php errors. (separate issue...) and in the process, realized D8 core was a version behind. I hoped that an update would fix the view problem. My sandbox is on a remote server and the most straight forward way for me to update core, is simply upload the files through FTP after renaming .htaccess, index.php and robots.txt. What I didn't realize, is that you apparently have to rename composer.json and composer.lock now too? Updating core, broke the composer link for Commerce. I was met with errors that the dependencies of Commerce weren't met... I then had to re-add the repository, and basically "re-install" commerce through composer because all links to it were gone. (The core update didn't fix my view problem if you were wondering...)

This is where a major disconnect lies with "some" modules requiring composer to be used to maintain them due to the various dependencies it requires. Upgrading Drupal 8 core has now become a rather pretentious and fragile task. Not because core is fragile, but because there's no way for contribs to deal with core updates along side their own dependency updates. It begs the question: "How are new Drupalers expected to deal with updating Drupal without getting frustrated with the update process?" As time goes on, more and more contribs will begin leveraging 3rd party dependencies and if a person isn't careful, they can very easily break the links that tie everything together.

Mixologic’s picture

I was curious myself just how big of a project it would be, and looking at the commit history behind wordpress' desktop electron UI, it kinda looks to me like a couple of people worked on it, with some help from random folks.

That doesnt say 'herculean task' to me: https://github.com/Automattic/wp-desktop/graphs/contributors

But, my hope is that we can attract the design/UX people *from the start*, and come up with an awesome workflow first, and then build the app to that spec. Otherwise we might end up with another 747 cockpit kind of User experience.

DamienMcKenna’s picture

@mixologic: Amazing info, thanks for putting that together.

Could the process be simplified by doing controlled, limited update checking through the UX? So, maybe there'd be one process for checking core's updates, another for contrib, another for other dependencies, thus splitting one giant status check into e.g. three smaller ones?

cilefen’s picture

I think this issue needs a retitling, a new component, and an issue summary update based on #13.

philsward’s picture

Anybody ever run PHPbb? I think that's the right forum I'm thinking of.. anywho, I used it for a project a long time ago and did an update to it. What I found interesting is that it would run a diff against EVERY core file. If there were any changes from core, it would present those files that were changed and ask what you wanted to do with it: merge, replace or use existing.

Could we do this with key files like .htaccess, robots.txt, composer.json/lock? Since it isn't recommended to modify core files, we don't really need to worry about all of them, but the ones that are most likely to change from install, could be approached in this manner.

Essentially do a diff of those files and merge any changes that aren't part of the core files (Recommended).
But also provide a way to keep existing or completely overwrite them.

This would probably require those files to be present in another folder and copied to the right place on install / upgrade.

If no changes aren't found in the diff, it doesn't even ask about those files and skips them.

It's one extra page added to the update process and if the UI is done well, shouldn't be too scary for the less tech savvy. The major benefit is that upgrading literally means blowing down the entire extracted archive without having to worry about renaming or deleting files ahead of time. There's been issues brought up in Drush to ignore core files but even that hasn't gained traction making it difficult to update core from Drush. It's also been the #1 reason you can't upgrade core from the GUI if I'm not mistaken.

jhodgdon’s picture

When you do a Linux upgrade (at least in Ubuntu), you get something similar too with config files. For instance, there's a new version of Apache or PHP, and you have custom php.ini or other config files on your system (/etc/apache*...), it lets you view the diffs for each one that is different from the previous distro, and then either accept the new file, keep your old file, or accept the new while keeping a backup copy of the old.