Efforts to stabilize Composer-based Drupal platforms have been scattershot. There's Hosting Git (already in aegir core), and then there's Hosting Composer. There are a number of issues in the Aegir queues. See links below.

Considering Composer is now being used to construct Drupal itself, and the Drupal Composer site building method is finalized (enough), we should make composer install a built in feature of Aegir, more specifically Provision.

The attached branch runs composer install in the next few lines after we run drush make. It's as Drupal native as it gets.

Errors are handled through drush which means good capture by hostmaster.

Warnings are currently detected through string match for the word "Warning", I do not know if there is another way to do this. Most common warning is an out of date composer.lock file.

This patch is solely to run composer install on verify. There is no UI. There is no need for a UI. A drush setting can block running composer install on subsequent verify runs if vendor is present. Other tasks can be provided by other modules, preferable Hosting Task Extras.

After discussion on Twitter, it's clear it's worth it to run composer install every time. If vendor and composer.lock are already present, the command only takes 1 second to run.To test this easily:

  1. Install Aegir.
  2. Enable Hosting Git.
  3. Create Platform
  4. Git URL: https://github.com/thinkdrop/drupal-composer
  5. Repository docroot: web
  6. Save
  7. Watch Verify end successfully having run composer install after cloning the git repo.
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Jon Pugh created an issue. See original summary.

Jon Pugh’s picture

Issue summary: View changes

  • Jon Pugh committed 33ab262 on 2937147-composer-install
    Issue #2937147: Add Composer support directly to Provision
    
Jon Pugh’s picture

Here's a patch file for easier review...

  • Jon Pugh committed 93f3407 on 2937147-composer-install
    Issue #2937147 by Jon Pugh: Add Composer support directly to Provision:...
Jon Pugh’s picture

FileSize
2.66 KB
Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Here's what a platform verify logs looks like:

Provision verifying a composer-based Drupal platform.

colan’s picture

I'm reluctant to get this in as-is given that:

  • The code here continues the tradition of functional programming making it harder to read, maintain, and enhance. Aegir Composer, as it's currently written, is very clean OO, making it a pleasure to work with.
  • Aegir Composer introduces the concept of deployment strategies, which cleans up the platform creation UX.
  • I'm a fan of the small-core philosophy, where we make plug-ins modular, and slurp them in as opposed to placing everything in the base modules.

But given that

  • the Aegir 3 code is throw-away in the long term, and
  • keeping Aegir core small is just one philosophy,

...I won't block this if everyone's on board. However, if it goes in, we should must reincarnate Aegir Composer as Aegir Deploy (hosting_deploy) so as not to lose the deployment strategies or the clean-up of the Create Platform page. I actually merged all of the well-written documentation for this stuff earlier today. (It would need a bit of tweaking, but not a huge deal.) See #2931158: Add documentation to Aegir Docs for details.

ergonlogic’s picture

It's worth noting that hosting_composer also provides an option to build platforms with composer create project

Jon Pugh’s picture

Thanks for the feedback, but I really need a technical review or test.

I also think there's some fundamental disagreement on what "small core" really means. To me it means the core components are useful without needing a bunch of add-ons.

What is the decision process for what gets into this "small-core" anyway? I'd argue that building the platform codebase is as core as it gets, no?

"Small core" philosophies is why this patch is so small to begin with. It doesn't do anything extra: just enables a Successful verify on the first try when using git and composer, in only 44 lines of code

I'm not sure how much smaller you can get than that.

Jon Pugh’s picture

It's worth noting that hosting_composer also provides an option to build platforms with composer create project

There's nothing preventing other modules from providing extra tasks. This isn't an alternative, this is a more native solution to the problem.

helmo’s picture

What about the '--no-dev' parameter for composer. One of my D8 test platforms (installed with drush dl) has a composer.lock file but was changed by the updated verify task by downloading things like phpunit.

How to handle servers that don't have composer installed? The first error line in a task log might be obvious for us, but it would not be nice to confront our users with that after the next aegir release. Should we start installing it via debian/aegir3-provision.postinst and not it in the release notes?

Jon Pugh’s picture

I did add a @TODO about how to alter the composer install command, perhaps we do default to '--no-dev' and make the entire command a drush option so it could be overridden globally or in the platform?

I will add a check for the executable. If you could add it to the Debian package that would be great.

ergonlogic’s picture

Re. "small-core", here's a good description. The most relevant definition is:

The real historical problem that has made the decision to add or remove components in [Aegir] core difficult is the dichotomy between [Aegir] as a framework and [Aegir] as a product. Framework advocates want a small, agile core focused on robust APIs that can advance at a faster pace. We’ve traditionally called this “small core.” Product advocates are focused on [Aegir] being a complete product for website building; with sensible defaults, out-of-the-box [website management] tools, and a plethora of features.

This was the fundamental idea behind the "golden contrib" initiative.

FWIW, I think adding Drush Make directly to core was a mistake; specifically because it is now becoming irrelevant, and we can't clean turn it off.

Jon Pugh’s picture

I can appreciate the philosophical discussion of what should or not be included in this project in the future and the "next generation".

However I am more concerned with the present.

Our users right now are suffering from being unable to deploy composer-managed-drupal on Aegir without going to the command line or by using devshop or hosting_composer.

I have had code in DevShop to do this for probably more than 1 year. It's about time we put this in.

  • Jon Pugh committed bc6326d on 2937147-composer-install
    Issue #2937147 by Jon Pugh: Add Composer support directly to Provision...
Jon Pugh’s picture

Based on advice from Helmo I wrapped the entire thing in a

if (drush_get_option('provision_composer_install_platforms', TRUE))

Now users can "turn it off" by setting that in their drushrc.php file.

memtkmcc’s picture

This feature is so badly needed, that I'm eager to support whatever hack in Aegir core is needed to make it happen. *Now*.

Let's leave code elegance and related subtleties for another occasion (and branch) once we will have more time to do these things accordingly to the rules of the programming art.

That is my opinion/vote, at least.

GTD!

Jon Pugh’s picture

helmo’s picture

Last week I started coding a bit in the Debian postinst scripts to install composer. However now that it's included in both stretch and xenial it seems better to just defer to those packages.

Unfortunately it's not in Jessie, so adding it to 'requires' would break there. ... something we can do in 4.x but as Jessie is supported until 2020 breaking it in 3.x now is not nice.

2937147-composer-install now has a commit I'm testing for this

helmo’s picture

Status: Needs review » Reviewed & tested by the community

Seems to work ok... I just added a shell_exec('which composer') to be sure we're not failing when an existing installation just lacks the composer binary.

I created #2938992: Enhance composer support in Aegir core to handle the new option in hosting_composer.

Jon Pugh’s picture

Thanks helmo...

I think we might need to move the shell_exec()... I'm all about preventing existing installs from breaking, but for users expecting composer install, this will cause a silent failure if composer is not executable.

Let's change it to give a warning if composer is not executable, and tell the user how to disable the composer install (and the warning) by setting the drush option?

  • Jon Pugh committed 82cf956 on 2937147-composer-install
    Issue #2937147: Add Composer support directly to Provision.
    - If check...
Jon Pugh’s picture

Ok, now what happens is:

if drush variable allows it, and shell exec composer fails, we show a warning:

The composer exectuable was not found. Install composer using instructions located at https://getcomposer.org/doc/00-intro.md#globally or, if you wish to disable composer install, set "provision_composer_install_platforms" to FALSE in the ~/.drush/drushrc.php file.

That way users who do want composer to run, will be told why it won't run.

Jon Pugh’s picture

Status: Reviewed & tested by the community » Fixed
FileSize
22.4 KB

screenshot of warning message for missing composer.

How does it look to you helmo? If you set to RTBC again I can merge right now.

Jon Pugh’s picture

Status: Fixed » Needs review

  • Jon Pugh committed d8cb5e3 on 2937147-composer-install
    Issue #2937147: Add Composer support directly to Provision.
    - Cleanup...
Jon Pugh’s picture

Changes pushed, New patch attached.

  • Jon Pugh committed 986760a on 2937147-composer-install
    Issue #2937147: Add Composer support directly to Provision.
    - Typo.
    
Jon Pugh’s picture

YAP

helmo’s picture

Status: Needs review » Reviewed & tested by the community

Great ... https://gitlab.com/aegir/provision/pipelines/16791163 is running a final test ... lets merge if that passes.

Jon Pugh’s picture

Gitlab test passed. Running PR on github to test against the behat tests: https://github.com/aegir-project/provision/pull/27

Jon Pugh’s picture

  • Jon Pugh committed 33ab262 on 7.x-3.x
    Issue #2937147: Add Composer support directly to Provision
    
  • Jon Pugh committed 82cf956 on 7.x-3.x
    Issue #2937147: Add Composer support directly to Provision.
    - If check...
  • Jon Pugh committed 93f3407 on 7.x-3.x
    Issue #2937147 by Jon Pugh: Add Composer support directly to Provision:...
  • Jon Pugh committed 986760a on 7.x-3.x
    Issue #2937147: Add Composer support directly to Provision.
    - Typo.
    
  • Jon Pugh committed bc6326d on 7.x-3.x
    Issue #2937147 by Jon Pugh: Add Composer support directly to Provision...
  • Jon Pugh committed d8cb5e3 on 7.x-3.x
    Issue #2937147: Add Composer support directly to Provision.
    - Cleanup...
Jon Pugh’s picture

Status: Reviewed & tested by the community » Fixed
Jon Pugh’s picture

Thanks everyone!

This is a big feature, it's really powerful when combined with hosting_git.module. I'm going into the docs now to write up how to use composer and git for Drupal on Aegir!

colan’s picture

For follow-up work necessary to provide the Deployment Strategies functionality, see #2938992: Enhance composer support in Aegir core.

Jon Pugh’s picture

Added documentation on using Composer in Aegir to the Platforms page:

https://github.com/aegir-project/documentation/blob/3.x/docs/usage/platf...

Jon Pugh’s picture

We should make a big deal out of supporting this setup now on the next release. People really want to build sites this way and haven't been able to do that using Aegir!

ahillio’s picture

Wow, having followed #2736801 and now this issue, I am so delighted to see this! BIG THANKS Jon and everyone else!!!

mengi’s picture

I agree, this is a big deal. Thanks to all who worked on this!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.