A concrete example comes from D8.4 #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x.

Drush is important: "drush pm-update" to apply security updates; often to run cron, take backups; 1000s of contrib modules shipping with drush.inc; aliases and batching. In comment #31 of the above issue @xjm states "Per the DA, 60% of Drupal core installations are from tarball, and therefore not managed by Composer". It is these installations that are at risk of drush failing.

Moving to composer would solve the problem, but it seems likely that many of these users have no experience of composer and would find it difficult, see #2845379: Provide optional composer integration but don't force users to understand how to use composer. Furthermore, there are existing sites that have been handed over to a customer and it would be preferable not to break them.

The heart of the problem is that using a global drush creates dependency problems, and these are on the verge of becoming unmanageable. The solution is to use a site-local drush. The composer method already does this, which is why it works.

#2909032: Support drush for sites not built from composer solves the immediate problem
#2242947: Integrate Symfony Console component to natively support command line operations entirely side-steps the dependency issue and would be a great tool. However it doesn't address existing sites needing to run existing drush commands (including 1000s of contrib modules with drush.inc).

Comments

AdamPS created an issue. See original summary.

cilefen’s picture

Status: Needs work » Active

There is no patch and no plan so this is active status. What kind of alteration to Drupal core would help with this?

Chi’s picture

I think this should be in Drush issue queue on GitHub.

AdamPS’s picture

@Chi I understand it from reading #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x, the most likely way for it to fail is a change to the dependencies of Drupal. Hence the task for Drupal is to evaluate any planned dependency changes and check they don't break compatibility with drush. It would be good to get confirmation of this from a drush expert.

Chi’s picture

the most likely way for it to fail is a change to the dependencies of Drupal

Right. And this was already done in #2712647: Update Symfony components to ~3.2. There is nothing we can do about this in Drupal core unless we decide to downgrade Symfony to version 2 which I suppose is not acceptable.

AdamPS’s picture

Correct, the change to Symfony v3 has already been dealt with - right now drush 8 works with D8.4.

This issue is about safeguarding against future breakages of a similar nature.

moshe weitzman’s picture

Alas, it isn't "dealt with". The drush.phar that we recommend people use has Symfony2 in it. If people use that against a Drupal 8.4+ site, unpredictable failures can ensue when Drupal gets an older version of Symfony than it expects. Drush8 has to support older versions of Drupal 8 so I don't see how it can immediately upgrade to Symfony3. Also, 3rd party drush commands may be written for symfony2 api.

AdamPS’s picture

Title: Keep Drush 8 working with D8.5 onwards » D8.4 onwards not compatible with Drush 8
Version: 8.5.x-dev » 8.4.x-dev
Category: Task » Bug report
Priority: Major » Critical
Issue summary: View changes

@moshe weitzman Many thanks for the correction.

I have updated the title to clarify that there are still issues with D8.4. I have increased the priority to critical. It seems likely that with D8.4 many users will find that drush stops working. I have set this to a bug report because vital things that used to work are about to stop working. I suspect this might be controversial and someone will alter it and I'm certainly not going to fight about it. Personally I guess that the level of frustration that is about to arise amongst the less technical/developer part of the community will justify these high settings.

Proposed actions for D8.4

  • Create further documentation to warn users and explain their available courses of action, roughly speaking: take the jump to drush9+composer or limp on with drush8 and careful workarounds.
  • Ask the drush team could provide a second phar for drush 8 that includes Symfony 3, i.e. in the short term maintain one drush 8 phar for <=D8.3 and one for >=D8.4). This would solve the problem for servers where all sites use the same Drupal version. It could also work for users who were willing to remember to type drush7 or drush8 depending on which site they were managing.

The documentation for converting a site to composer is likely to baffle most non-developers. It is far away from being a clearly defined set of steps that someone could follow. Using composer for Drupal seems to be roughly alpha quality at present - fine for developers who like to live on the bleeding edge, but not a polished stable solution integrated into Drupal.

amateescu’s picture

This was probably discussed already in another place, but why can't Drush use the vendor libraries supplied by Drupal (8)? Compatibility with Drupal 7 comes to mind, but can't we overcome that somehow by only loading Drush's vendor libraries if they can't be found in Drupal core?

AdamPS’s picture

(Drush experts please correct me)
@amateescu The solution that works well is a site-local drush that shares dependencies with Drupal. The versions of all components must be compatible. This is almost always achieved with composer, which updates drupal and drush together.

Symfony will change between major versions, and the code in drush needs to change to match. So drush cannot easily work with whatever version of dependencies that it happens to find. @greg.1.anderson has kindly handled that on a one-off basis creating a version of drush 8 that works with either Symfony.

"drush pm-upgrade" is typically used in a strategy where there is a global drush. Users who install drush from the drush phar which a specific version of Symfony built in. There are also complications because drush has to operate before it has found a Drupal site, and indeed a few commands don't even need one (crucially site-install!). Furthermore drush commands may end up operating on a remote server.

I think the drush team has thought about this carefully and the composer method is the only thing that works reliably in the long-term. In the short-term we have a problem that composer is not fully integrated as an out-of-the-box solution.

catch’s picture

@moshe

If people use that against a Drupal 8.4+ site, unpredictable failures can ensue when Drupal gets an older version of Symfony than it expects.

Have you tried drush with 8.4+ and experienced unpredictable failures? When we worked on the Symfony 3 core update we tested extensively with various versions of drush, and didn't actually hit any failures at all. There was an issue at one point but it was resolved a while before the update landed. So unless something's regressed, while there's a theoretical possibility of unpredictable breakage in practice it should be fine. Symfony 2.8 and 3 are API-compatible, and we're not using any new APIs in Symfony 3 yet at all.

greg.1.anderson’s picture

@catch Please see The Trouble with Two Autoloaders. When using a Drush phar (or other form of global installation), mixing the dependencies between Drush's vendor directory and Drupal's vendor directory can cause problems. Right now, the Drush 8 phar happens to work with Drupal 8.4.x, as you tested and observed. However, even patch version updates to Symfony 3 can cause failures. Unless you are using a site-local Drush, any update to Drupal core's dependencies (including non-Symfony dependency updates) can cause problems with the global Drush.

There was an issue at one point but it was resolved a while before the update landed.

This is because the Drush maintainers have to eat glass and walk across fire to keep the global installation working. Well, maybe not *quite* that bad, but there is definitely work that has to be done, and success is not guaranteed.

@greg.1.anderson has kindly handled that on a one-off basis creating a version of drush 8 that works with either Symfony.

I think it was Moshe who made the last fire walk to get Drush working with Symfony 3.

@amateescu The Drush 9 PR #2843 does as you suggests, and postpones dependency injection, and loads as few classes as possible, until after the Drupal root is identified. This alleviates most of the problems with having two autoloaders. This comes at a cost of maintaining a more restrictive "preflight" stage in the Drush bootstrap. (Code the few ever look or think about.)

Note that there is still no pm-* commands in Drush 9. Without that, it is questionable whether #2843 solves anything for non-Composer users. The Drush 8 version of that PR is #2787. That PR is on hold pending the completion of #2843. Folks who care about the future of the global Drush and the `pm-*` commands should participate in these issues. The current pm maintainer has not had much time to contribute to Drush for a while, so these commands are largely neglected, despite their wide use.

Finally, I would like to point out that this issue with Drush is the "canary in the coal mine" for Drupal's dependency updates / SemVer / major version release policy. In short, Drupal's plan is:

- Major releases e.g. Drupal 9 only remove deprecated APIs from the last Drupal 8 release.
- Drupal makes major updates to dependencies during minor Drupal updates, as allowed by SemVer.
- Drupal modules may sometimes directly call the APIs of Drupal's dependencies.

These things, considered and discussed separably, are all perfectly reasonable-sounding proposals. When taken collectively, though, it should be clear that at some point, sooner or later, something is going to have to give. The update from Symfony 2 to Symfony 3 was only possible because these two releases are so similar. How is Drupal ever going to update to Symfony 4? It is likely that some collection of Drupal modules will be broken with each (or at least some) Drupal 8 minor release. Composer has mechanisms to deal with this, but global Drush and pm-* commands -- and, for that matter, the Drupal module installation UI -- are not well-equipped to manage these sorts of changes. The take-away here is that Drupal should be conservative with dependency updates until Composer use is standardized. This will be manageable if enough effort is made in that direction. If we're all content with "happens to work", then we run the risk of running up against a wall where Drupal cannot update a dependency without breaking one of the assumptions in the update plan. We need to front-load Composer work before that happens.

mpdonadio’s picture

Is this going to affect DrupalCI at some point since it does a `drush site-install`? And core dev work where a vendor-local version isn't really feasible?

greg.1.anderson’s picture

@mpdonadio: It is speculative at this point to say exactly what affect this will have on various uses, as it depends on what changes are made in the APIs of Drupal's dependencies, and how those APIs are called by Drupal, Drush and the various modules. If the core causes are not addressed per #12, though, then yes, eventually DrupalCI and core dev work will be impacted.

We should also be clear that Drupal only updates its dependencies during minor releases; as long as this policy is maintained, then if Drupal 8.4.0 works with Drush 8, then we expect that it should continue to work for all of the subsequent 8.4.x releases. Things can break at any time during the development of 8.5.x, though, as dependencies change. And so this pattern will repeat for each Drupal minor release. Also, this is why Composer-managed Drupal sites should use webflow/drupal-core-strict.

The problem can be addressed from two sides:

1. Make Composer-managed Drupal sites better in core.
a. Support 'composer require' of Drupal modules in core from the Drupal admin UI
b. Support 'composer update' from the Drupal admin UI
c. Support 'composer create-project' of Drupal (refactored to be like drupal-composer/drupal-project)
d. Add a minimal Symfony CLI to core, with `site-install` and an extension mechanism to add CLI commands to modules.
e. Replace `drush make` as the mechanism for building distributions on drupal.org.

2. Make global Drush better
a. Contribute to #2843 and #2787
b. Help maintain pm-* code in Drush 8
c. Port and maintain pm-* code in Drush 9

(1) is the long term / lasting strategy; many of these topics are being discussed elsewhere, e.g. #2845379 & so on. If we can satisfy all of these needs by 8.5.0, then maybe we do not need to put much effort into (2). If it's not possible to "solve Composer" for all users in that timeframe, though, then more contributors to help maintain legacy functions would be helpful.

catch’s picture

Title: D8.4 onwards not compatible with Drush 8 » Drush and core compatibility is fragile
Category: Bug report » Task
Priority: Critical » Major

The update from Symfony 2 to Symfony 3 was only possible because these two releases are so similar. How is Drupal ever going to update to Symfony 4?

See #2899825: Release and support cycles for PHP dependencies for discussion.

Since there's no identified bug here, I'm downgrading this to major and retitling.

If we want to talk about dependency updates in general, let's please do that on the issue especially opened for discussing dependency updates.

AdamPS’s picture

Thanks everyone for your time.
@catch I have done some testing and I am worried that there is still a bug, albeit one with an easy workaround (which is to ignore the error and everything is fine after).

At a guess, drush running against D8.3 has already loaded Symfony 2 and it goes wrong because the new D8.4 core has Symfony 3.

Is it possible that in the testing done so far drupal core was upgraded using git, and then testing done running drush commands after?

Error: Call to undefined method Symfony\Component\HttpKernel\Event\PostResponseEvent::setDispatcher() in                                [error]
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch() (line 94 of
/data/www/testa9.albanyweb.co.uk/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php) #0
/data/www/testa9.albanyweb.co.uk/web/vendor/symfony/http-kernel/HttpKernel.php(88):
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch('kernel.terminat...',
Object(Symfony\Component\HttpKernel\Event\PostResponseEvent))
#1 /data/www/testa9.albanyweb.co.uk/web/vendor/stack/builder/src/Stack/StackedHttpKernel.php(32):
Symfony\Component\HttpKernel\HttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request),
Object(Symfony\Component\HttpFoundation\Response))
#2 /data/www/testa9.albanyweb.co.uk/web/core/lib/Drupal/Core/DrupalKernel.php(634):
Stack\StackedHttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Symfony\Component\HttpFoundation\Response))
#3 phar:///home/adam/ttt/drush.phar/lib/Drush/Boot/DrupalBoot8.php(198):
Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Symfony\Component\HttpFoundation\Response))
#4 phar:///home/adam/ttt/drush.phar/includes/preflight.inc(75): Drush\Boot\DrupalBoot8->terminate()
#5 phar:///home/adam/ttt/drush.phar/includes/startup.inc(458): drush_main()

#6 phar:///home/adam/ttt/drush.phar/includes/startup.inc(365): drush_run_main(false, '/', 'Phar detected. ...')
#7 phar:///home/adam/ttt/drush.phar/drush(114): drush_startup(Array)
#8 /home/adam/ttt/drush.phar(10): require('phar:///home/ad...')
#9 {main}.

greg.1.anderson’s picture

AdamPS’s picture

@greg.1.anderson Thanks for the reference. So it's a known issue - or at least it's known to a few people. However I'm not clear what the proposed resolution is.

I can certainly believe there is no easy fix so we can potentially edit the release note to warn users they will see the error and can ignore it. No doubt some users won't read the warning, and a question will appear on Drupal Answers, but they should figure it out before too long.

However as far as I can tell this edit has not yet been done (see code). Shall I raise an issue for that?

Mixologic’s picture

re #13 Drupalci only uses drush site-install for drupal 7 testing.

alisonjo2786’s picture

So, I love drush, I don't want any drush anything to stop working, now or ever, I was absolutely crestfallen when I found out what's coming down the pike :( That said...

What do y'all think is more realistic/practicable, in time for Drupal 8.5, i.e. six months from now?

  1. Re-working the existing, or creating comparable new, drush commands for pm-* functionalities.
  2. Implement some kind of composer + Drupal admin UI thing.

(Oh and, is that the correct deadline?-- Drupal 8.5 is the concern at this point, right? Or are we concerned about 8.4? I'll admit I got a bit lost.)

webchick’s picture

8.4.0 comes out October 4 (aka in 4 weeks), so if there are compatibility problems between it and Drush, that needs pretty urgent sorting out. :\

8.5.0 won't come out until March 7, so we have some time to work on a more long-term solution for that one.

Source: https://www.drupal.org/core/release-cycle-overview

cilefen’s picture

Once one configures a site to be managed by composer, the difference is a follows:

drush pm-update webform becomes composer update drupal/webform && drush updatedb

That's it. If your hosting doesn't execute composer install --no-dev when deploying to production, that should be done before committing, pushing, and deploying.

cilefen’s picture

Ha - views is a bad example. Duh. I'll change that! -done

alisonjo2786’s picture

Thank you, @cilefen -- we're actually meaning to use this issue to focus on things related to drush + core compatibility, though, rather than on CLI dependency management overall, if that makes sense. (That said, definitely a helpful reference, thank you!)

@webchick Thanks for the calendar explanation! Yeah I think there are concerns about drush + Drupal 8.4, but I'm shaky on the severity -- like, I think the pm-* commands do still "work" in a "use at your own risk" way, but, that's actually a big part of how these strings of related threads became extra active in the last few weeks -- Drupal 8.4 + Symfony upgrade necessitated upgrading to (not yet stable) drush 9, and drush 9 doesn't have pm-* commands anymore -- ergo, all the site-builder-y hubbub related to drush 8.4+ (or 8.5+, depending on how you look at it).

We have somewhat of a reprieve, bc the awesome drush team made another Fire Walk (see above), but drush 8.x + Drupal 8.4 are still iffy, or use-at-your-own-risk-y, I think? (I'll let the others get more specific here...)

cilefen’s picture

I was responding to #21.

alisonjo2786’s picture

Ah, I see. To the bullet #1 in comment #21, or?

Re-working the existing, or creating comparable new, drush commands for pm-* functionalities.

(The goal would still be to have *drush* commands for this/these tasks, tho -- or am I still misunderstanding you?)

AdamPS’s picture

Version: 8.4.x-dev » 8.5.x-dev
Issue summary: View changes

I have created a separate issue for the outstanding D8.4 problem: #2907224: Upgrade to D8.4 using drush generates an error message. The upgrade appears to fail with an error, but in fact seems to have worked. Next step is to decide on a resolution - perhaps live with and document.

This issue should remain for the D8.5+ task to manage drush compatibility of future releases.

AdamPS’s picture

@catch thanks I have added a comment to #2899825: Release and support cycles for PHP dependencies referencing back here.

Can anyone see a way to add some automatic tests of drush compatibility? I guess it would involve using the latest drush 8 phar to install the current stable D8; test a couple of drush commands; upgrade to the current dev D8; test drush commands again.

webchick’s picture

An alternative (though possibly contentious as far as implementation details go) is adding some of these "core" Drush commands to core... #2242947: Integrate Symfony Console component to natively support command line operations

AdamPS’s picture

@webchick thanks for following this issue. From a user's perspective #31 seems attractive - it makes Drupal more self-contained and solves the current awkwardness that Drupal cannot self-upgrade core without help from an external tool (except by means of manual untar and complicated copying).

Please can I invite you to post your comment on #2845379: Provide optional composer integration but don't force users to understand how to use composer or perhaps #2477789: Use composer to build sites?

I feel that this issue is best reserved for the matter of keeping drush working whilst an alternative is developed.

webchick’s picture

Hm, no, I think that comment belongs squarely to this issue. If the fragility problem is caused by Drush needing to add another autoloader that conflicts with core's autoloader, then one approach to solving that (I would presume?) is to move the "80%" commands, such as pm-update and friends, into core in order to eliminate that requirement. I couldn't say whether that option would be any easier than either option 1 or 2 from #14, however.

ressa’s picture

+1 for moving most used Drush commands into core. Drush and Symfony would move in lockstep code-wise, and tests can be run, to make sure that Drush works with future Drupal core releases.

alisonjo2786’s picture

+1 -- I mean it sounds daunting and I have no idea what's involved, but if it's doable...

@AdamPS thanks for adding #2907224, excellent call!!

greg.1.anderson’s picture

For the sake of clarity, a recap on the Drush + Drupal 8 situation:

- Currently not officially supported, but "happens to work" (informally sanctioned)
- Likely to break with future updates to Drupal dependencies
- Drupal dependencies are not expected to change until 8.5.x.
- pm-update from 8.3.x to 8.4.0 results in a strange error message, but update appears to work

Regarding #32, "put an MVP cli in core" should have been an option #3 in #14. I do think this is the easier option from a technical standpoint. I'm happy to help sprint on this in Vienna if this is the direction folks want to go.

diqidoq’s picture

Like @catch, I can confirm that D 8.4.rc1 and Drush works so far here. But I am admittedly a lil' bit off balanced about the "news" regarding Drush's future torned here in certain comments. o.Ô

Thanks for the sum-up Greg.

Just for the record:

composer create-project drupal-composer/drupal-project:8.x-dev my_site_name_dir --stability dev --no-interaction
takes more than 10! minutes ... on a very new i7 quad-core CPU with 32 GB Ram and fan noise increasing ...

drush dl drupal --select --destination=xxx site --drupal-project-rename=xyz
5 seconds ... and not even a little peep ... ...

Chi’s picture

takes more than 10! minutes ... on a very new i7 quad-core CPU with 32 GB Ram and fan noise increasing ...

Composer is doing a huge number of HTTP requests. I am using Composer parallel install plugin. It makes the process much faster but it is still slow in comparison with drush dl.
https://github.com/hirak/prestissimo

AdamPS’s picture

@greg.1.anderson thanks for the summary in #35
I wonder if we should try to be a bit careful when talking about level of support. As has already been outlined in this thread, drush+Drupal 8 is a vital combination. Many current users who use it for many tasks including to download security updates, and they don't necessarily have an easy alternative. These are people who have chosen Drupal, sometimes as the way they earn their livelihood, and they trust the Drupal community to provide them with a product they can continue to use.

I accept your point that this situation is delicate, and this issue is aimed a) fixing things for the future and b) informing Drupal core committers that they need to be careful in the meantime.

I don't think it's necessarily helpful to tell the wider community that their Drupal website is not officially supported and only happens to work. It does work for D8.4 now, and as you say dependencies aren't expected to change, so we don't have to be so negative. Please can we try to find some different words? For example it's supported for D8.4 but won't necessarily be available in D8.5.

AdamPS’s picture

#35 "put an MVP cli in core" sounds logical and clear and it would be great to have. In case it turns out to be a bit tricky, here's one more idea to throw into the mix. This is a variant that I think would be trivial to implement, although I guess could be considered something of a hack.

We could add drush to the Drupal tar/zip download images.

It should be no more than "composer require drush" and check in the resulting composer.json. Then I guess there would be a change to global drush when it's searching for a site-local drush to check in /vendor/drush as well as /../vendor/drush. To cater for users who are worried that drush is a security risk, we could add a config option that governs if "drush shipped as part of Drupal" is enabled, and let it be disabled by default .

One last thought: if on this thread we solve the drush+drupal dependency that's great, but there is still the question (albeit less of an urgent ticking bomb) of how to load the dependencies of contrib modules (in particular see the address module). Currently composer is the main candidate for solving that problem too. So I don't want to be negative, but I'm just putting forward the point that the changes here don't kill off the need for a dependency manager in Drupal. I.e. this new "strategy 3" would kill off #14 strategy 2" but "#14 strategy 1" might still be on the table as the long-term lasting requirement.

ressa’s picture

@greg.1.anderson:

"put an MVP cli in core" should have been an option #3 in #14. I do think this is the easier option from a technical standpoint. I'm happy to help sprint on this in Vienna if this is the direction folks want to go.

That sounds amazing, I particularly take note of the sentence "this is the easier option". I really hope it happens. Regarding the hack-solution proposed by AdamPS, I fear this will just land us in trouble down the line, since it is not a full integration. But I'll let Greg be the judge of that :-)

By the way, I recently re-read Greg's chapter on Drush in The Definitive Guide to Drupal 7. It's essential reading for anyone serious about Drupal, and still valid for Drupal 8.

greg.1.anderson’s picture

So, yes, the observations above are correct.

  • We have no active pm-* maintainer, so the question of how to manage any needed changes in this code should it be necessary.
  • Modules such as address and others (more and more over time) obligate site builders to switch to Composer-managed sites as they bring in modules that require it.
  • If you have a Composer-managed site, you don't need the pm-* code.
  • Maintaining pm-* and the global Drush is a lot of work.
  • Eventually, at an unspecified point in the future, everyone will need to use Composer anyway, at which point the pm-* commands become irrelevant.

This is related to the global Drush insofar as folks who do not use Composer must use the global Drush. While the global Drush is technically still usable by Composer users, and some folks still prefer to use it this way, the utility of the global Drush is all the same far far higher to non-Composer users. This is what I was referring to elsewhere when I mentioned the consequences of *not* adopting Composer in Drupal.

In terms of the word used to describe the supported configurations of drush, they are nicely described on the Drush installation instructions, which is to say that you need Drush 9 to use Drupal 8.4.x and later. It would be really nice if, come 4 October, we had either a stable version of Drush 9, or supported Drush 8 with Drupal 8.4.0 (or both). However, increasing the support level of software does not happen with words, it happens with work, and there is a cost to that.

alisonjo2786’s picture

Hopefully not too OT -- @greg.1.anderson, #41:

Eventually, at an unspecified point in the future, everyone will need to use Composer anyway, at which point the pm-* commands become irrelevant.

  • I think that the folks who hope to continue using drush for those pm-* things, meant for that to be the focus of this thread, but I could be wrong -- i.e. those are the key things that could/will be lost, so those are the things that ppl are worried about losing, b/c they're going to be lost.
  • "everyone will need to use Composer anyway" -- everyone who doe CLI things, i.e. everyone except for Drupal Admin GUI people, or are you saying that the overall trajectory is for Drupal Admin GUI not to be usable for module updates, etc.?

EDIT: related issues... #2352637: Remove the UI for installing/updating modules from update module if it is not fixed in time for release, and, "User Interface Changes" list bullets over here: Use composer to build sites [#2477789] | Drupal.org

greg.1.anderson’s picture

are you saying that the overall trajectory is for Drupal Admin GUI not to be usable for module updates, etc.

We must have a way to use Composer from the Drupal Admin GUI before we can hope that every user might adopt it. This is not the only objection to Composer that folks have raised, but it is a key blocker.

alisonjo2786’s picture

#43 Thank you, gotcha, that's what I thought, just wanted to make sure :)

( #2845379: Do not force users to understand how to use Composer )

alisonjo2786’s picture

...you need Drush 9 to use Drupal 8.4.x and later. It would be really nice if, come 4 October, we had either a stable version of Drush 9, or supported Drush 8 with Drupal 8.4.0 (or both)

But, aside from the errors that don't seem to have tangible consequences, Drush 8.1.12+ does work with Drupal 8.4.x. But you're saying it isn't **supported** for Drupal 8.4.x, is that accurate?

I think one of the biggest /immediate/ concerns is that on October 4, pm-* won't work for people, or that there will be mixed messages about whether it works, because, for the drushers of the world, not having pm-* equals being forced to switch to composer, or, people thinking that's what's happened, which I think will be alarming and hubbub-causing, to say the least :-/ So maybe our immediate focus needs to be on that other issue that @AdamPS kindly created (but I haven't looked at since when he initially created it): #2907224: Upgrade to D8.4 using drush generates an error message

  • Are there other fixes needed to be able to say "Drush 8.x works fine with Drupal 8.4.x" / What's in the way of "supported Drush 8 with Drupal 8.4.0"?
greg.1.anderson’s picture

Well, we would need to finish the Drush PRs mentioned in #12, and we would need a maintainer for the pm-* commands. The list may be longer; this is up to @moshe weitzman.

cilefen’s picture

I think one of the biggest /immediate/ concerns is that on October 4, pm-* won't work for people.

#2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x ("Current Status") documents that the .phar file works.

As written this issue has multiple goals, some that require communication, and others that require coding. Some of that is Drupal core code changes, and others would be drush changes. That means this issue should be converted to a what we call a meta issue. It should in the issue summary, link to actionable, appropriately scoped issues that together solve the problem. One of those could be a drush issue called "Recruit a maintainer for the pm- commands", and one of the concerned community members could champion that effort.

As for the current goals, we can consider "Increase awareness that there is a problem, especially amongst core committers." completed.

Thank you all!

jmolivas’s picture

Related to #14

a. Support 'composer require' of Drupal modules in core from the Drupal admin UI
b. Support 'composer update' from the Drupal admin UI

Not sure if any of you here is aware of Composercat a Graphical interface for Composer (PHP).
https://www.getcomposercat.com/

The latest version supports autocomplete for Drupal projects:
Composercat's autocomplete now works with Drupal's Composer repositories (alongside Packagist). This feature is only available for Drupal projects. All you need to do is configure it in repositories or create a project from a Drupal template package.

c. Support 'composer create-project' of Drupal (refactored to be like drupal-composer/drupal-project)
This looks like the way to go and provide a `drupal/core` package to easy updated as `drupal-composer/drupal-project` does.

d. Add a minimal Symfony CLI to core, with `site-install` and an extension mechanism to add CLI commands to modules.
Let's not forget drush is not the only CLI for Drupal 8 we also have Drupal Console.

Talking about DrupalConsole we are keeping module:download, theme:download but deprecating any download functionality and making those commands wrappers for composer.

Latest DrupalConsole Launcher works for 8.3 and 8.4 since we only provide a minimum about of commands and then load DrupalConsole for site vendor directory once a valid Drupal site is found.

e. Replace `drush make` as the mechanism for building distributions on drupal.org.
Totally agree this should switch to composer instead.

AdamPS’s picture

I raised the original issue and I am grateful for the attention everyone has given it. Unfortunately I won't have further time to spend on Drupal.org issue queues for a few weeks.

I was encouraged when @catch downgraded this issue in #15 on the basis that "there is no known bug".

However from #41 I hear "yes but it's not supported". It seems controversial for a minor Drupal release to bring with it a loss of support of a major function (pm-upgrade) with a lack of entirely satisfactory alternatives. Does anyone know if there there been any advance warning of this? I don't mean to criticize individuals or drush, but just to point out that the consequence of a bunch of changes interrelating is potentially the sudden unexpected loss of something important.

Does that mean we should reopen an urgent 8.4.x urgent task I wonder? It seems that there is no maintainer and no support for the scenario that apparently as many as 60% of D8 users rely on for security upgrades.

Might the hack I suggested in #39 help in D8.4 as an emergency scenario until something better can be developed? It seems to solve the "two-class loader" problem with minimal code changes.

cilefen’s picture

Does anyone know if there there been any advance warning of this?

Yes. That was #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x.

How is that Composer is not entirely satisfactory? I'm seriously asking.

diqidoq’s picture

@cilefen:

Yes. That was #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x

Thanks for the linked issue.

How is that Composer is not entirely satisfactory? I'm seriously asking.

It is! Looking at it from its own point of view. But from Drupals point of view ... Maybe I misundersrand your question but, trying to answer it: because of Composer being still young, I think. And because of #36 and #37 and #14. And because of


composer require drupal/bootstrap:? drupal/coffee:? drupal/devel:? drupal/redirect:?
vs
drush dl bootstrap coffee devel redirect --select
or
composer create-project drupal-composer/drupal-project:8.x-dev $sitename --stability dev --no-interaction
vs
drush dl drupal --dev --drupal-project-rename=$sitename
... Where the first of the second comparision (composer) still creates latest official release and ignores the dev state. And because of the still low adaption rate of Composer in the Drupal community. Which has reasons. But it feels like some are in denial of this facts.

Another spot I would like to put focus on is: Composer reminds me a little bit of PECL and its first hype. Where is it now? We cannot build a complete admin UI update system leaning on a tool, which we not have control over nor know its future yet. The same partial discussion we had already regarding community chat where many good points were made against tools we cannot control. Symfony is one thing and big enough, but we shouldn't dismantle essential Drupal parts and should rather check for their strengths and if/how they can coexist.

Making Composer a strong requirement for Drupal sounds very risky to me, TBH. But as I sad, maybe I do not see the spots in future others see here and maybe I missed some discussions about it (for sure).

greg.1.anderson’s picture

#48.d: I didn't forget about Drupal Console; that's why I said "Symfony CLI" (maybe "Symfony Console" would have been clearer). I don't think that we should add all of Drush, as suggested in #39. As for the `site-install` command, I didn't mean to slight `site:install` by using the Drush name. It would make sense to use the Symfony Console / Drupal Console form for the command names. I don't care if the implementation for site install comes for Drush or Drupal Console (if we did move forward with this idea); either way, it would come out about the same. I think that the module cli command plugin mechanism in Drush 9 and Drupal Console are about the same, so I think we agree on approach here. Providing something like this in Drupal Core would be pretty easy, and would be an overall improvement to our current situation.

diqidoq’s picture

@ #52 One of the main reasons to use any command line tool vs admin UI is the faster way to achieve things for the advanced Drupal site builder and developer. Composer (as is now) for Drupal is exactly the opposite. It feels rather like another complexity layer in between, with quite long commands which take ages to process. Can somebody enlight me why - despite of the widespread adaption of Composer and the hope that it brings more user to Drupal who are compfortable with Composer - we have such a rush to woop everything over to Composer? There is nothing wrong with it. I use it all the day. But not for Drupal. And in the moment I do not see reasons for doing it except that newer Drupal versions try to force me to.

So on the end, I am not allone (as I heard) with creating bash aliases to shorten the commands again to give back some Drush command logic when playing around with newer Drupal/Composer test projects.

Mile23’s picture

Re: @webchick in #30: Here's a reroll: #2242947-88: Integrate Symfony Console component to natively support command line operations

And yes, this scenario is exactly why I was pushing for that before the 8.0.0 release.

greg.1.anderson’s picture

Hey @mile23, I did not realize that #2242947 already has a module command loader. That is great.

The MVP for that issue should be defined so that it can be finished and merged.

Mile23’s picture

@greg.1.anderson: Thanks. Could you help spec an MVP in that issue? That's kind of the main hold-up: What people beside me think is needed.

greg.1.anderson’s picture

@mile23: Happy to help out, but core committer opinions are weightier. ;) Let's see what can be done at Vienna.

jbitdrop’s picture

#36 / #51: Yeah, that's what I thought :/ ... but I think it's unavoidable.

AdamPS’s picture

How is that Composer is not entirely satisfactory? I'm seriously asking.

I think the reasons roughly fall into 3 categories

  1. Back-compatibility, as per https://www.drupal.org/core/deprecation. I work as a site installer for simple sites. I have delivered sites to customers and they are automatically upgrading using cron/drush - this will break and the customers will call me. I have scripts/systems that work and co-workers who understand how to use the current mechanisms. Moving to composer potentially gives me plenty of days of unpaid work.
  2. Readiness, especially #2845379: Provide optional composer integration but don't force users to understand how to use composer. There is still a lot of work to be done before the composer-based approach reaches parity of the current tarball approach for ease of use, features, and stability. It feels like an alpha right now because APIs are not agreed/stable.
  3. Suitability: memory limit (composer can fail even with 2GB); shared hosting where FTP is only option; auto-installers (e.g. Softaculous)

Dozens of others issues also, some of them collected into #2002304: [META] Improve Drupal's use of Composer.

My proposal as per https://www.drupal.org/node/2477789#comment-12258369 is that the composer enthusiasts work on (2) and in the meantime we keep the tarball installer working.

AdamPS’s picture

Title: Drush and core compatibility is fragile » [META] Drush and core compatibility is fragile
Issue summary: View changes

OK I've had a go to make this issue [META].

I am a little hazy on #2242947: Integrate Symfony Console component to natively support command line operations so please correct me if needs be. From reading the issue, it's looks like a separate thing from drush, with different syntax and APIs. No doubt be a great tool that people can gradually adopt. However there are existing sites that have been shipped to customers relying on drush. For example a server may have a crontab file that calls "drush @mysite cron". It seems that if drush stops working as per this issue, then the site owner will need the developer to change this to something else based on symfony.

More generally, drush is so widespread and powerful including: drush commands from contrib modules and custom; command hooks; drushrc.php; aliases and groups.

As @Mile23 suggested, in a ideal world, this might have been done for D8.0. However now that we are on D8.3, there are already sites shipped relying on drush and contrib modules shipping with drush.inc files.

Hence I have raised a new issue #2909032: Support drush for sites not built from composer based on #39.

diqidoq’s picture

To bring an example of how the fast Composer implementation makes me worry is a possible inconsitency (which may be a useful issue report for Drush later): Since a Composer started Drupal project's folder structure differs from a Drush created Drupal project's folder (Composer creates another ./web/Drupal_root per default), the "if-is-Drupal-root" check of Drush will cause a false in the superior of Composers ./web folder. But you can't install modules via Composer if not in the superior folder of ./web, which is the projects main folder if installed via Composer. Otherwise you will get a

Could not find package drupal/package at any version for your minimum-stability (stable). Check the package spelling or your minimum-stability.

And you cannot disable modules via Drush if not in ./web since this is the actually Drupal root folder from Drush's perspective holding the ./core and ./moduldes folders etc.

From my understanding reading the last issues regarding Drush and Composer including this one here, there will be a certain coexistence period in which I assume the terminal based administration with both tools should work, even if installing new modules will be part of Composer if Composer has created the project.

In this case we need a solution for Drush to "know" about the Composer folder structure to not fail if not in ./web for any still useful Drush command. Otherwise you will get a ...

Command pm-uninstall needs a higher bootstrap level to run - you will need to invoke drush from a more functional Drupal environment to run this command.

... when you try to pmu color tour -y in the superior folder of ./web.

But Composer can't replace many of the Drush usefulness yet nor in near future. So...

Hence I have raised a new issue #2909032: "Support drush for sites not built from composer" based on #39.

... would actually lead to an additional issue called:

"Support Drush for sites made with Composer to be able to still use Drush for tasks Composer can't."

Another question is: if this should be reported on Drush Github. If then this is an isolated Drush issue, isolated from Drupal core Drush/Composer integration issues to discuss ... which is kind of wrong IMHO.

moshe weitzman’s picture

Drush9 has already addressed the issue you describe where its hard to run commands from project root.

diqidoq’s picture

@moshe weitzan, ha! thanks!

Yeah, the reason why I came back here tonight was actually to edit my comment with a finding of a similar issue here: https://github.com/drupalcommerce/project-base/issues/13

Are you referring to this one or is this another druplicate ;-) ?

Mile23’s picture

Since a Composer started Drupal project's folder structure differs from a Drush created Drupal project's folder (Composer creates another ./web/Drupal_root per default)

That's incorrect. That's not the default behavior for Drupal under Composer.

If you use drupal-project to create a Drupal site, it will do that. drupal-project is a third-party project developed to address shortcomings in the Drupal 8 file system like having PHP In the docroot. And other stuff. :-)

It's like using an installation profile.

Instead of using drupal-project you can do something like this:

$ composer create-project drupal/drupal d8 @stable --no-dev

This will create a d8/ directory with Drupal 8.3 in it just like if you downloaded the tarball. CD to the directory and do this:

$ cd d8
$ composer require --dev drush/drush
[ magic happens ]
$ ./vendor/bin/drush site-install --etc....

Now you don't have to play catch-up with either Drush or drupal-project. When it's time to update Drupal releases, you follow the instructions in core/UPDATE.txt once it's amended: #2867757: Improve instructions for updating composer.json and /vendor

diqidoq’s picture

@Mile23, thanks for chiming in.

That's incorrect. That's not the default behavior for Drupal under Composer.

Well, then, we should elaborate on the meaning of "Default behaviour". Mine is: Following users to their issues, from an issue description which can not be incorrect, if described correctly from the suggested way in the docs, IMHO. Following the instruction of Drupal.org method A will exactly lead to the described behaviour. I do not want to sound nitpicky (I know I do), but while knowing about your sugggestion, this is not documented as the Default way here. Maybe "Default" is misunderstandable here. You refer to "Default" by not using drupal-composer but drupal/drupal, but this should be marked as "Default" then where the docs describe the scenario.

In this case:

Using drupal-composer/drupal-project (Option A)
$ composer create-project drupal-composer/drupal-project:8.x-dev my_site_name_dir --stability dev --no-interaction

... will lead to the following folder structure:
my_site_name_dir > web > core,modules,etc

For Drush the user will require to be in my_site_name_dir > web as working directory.
For Composer the user will requite to be in my_site_name_dir as working directory.

And this already begins with trying to install Drupal (db) after getting it with Composer.

I have asked 4 of my employees to install Drupal 8 with Composer following self googled provided instructions from Drupal.org to reproduce this issue. And I would cautiously say that an issue report of chx is also a good idea to take under consideration. ;-) By the correctly stated difference between your command and the one of the instruction I refer to: Which one is the "Default" then? It's maybe yours. But for those who require to look in the docs it's not.

And finally we should not forget what the intension of my example was.

ressa’s picture

For everyone following this issue, amateescu is experimenting with an approach, where a service on drupal.org handles all the Composer dependency resolution, and it looks very promising: #2910136: Experiment: package PHP libraries in a single Phar file

bojanz’s picture

Category: Task » Support request
Status: Active » Fixed

This issue seems to have no actionable ideas not covered by other issues, so I'm closing it.
That way ressa doesn't need to cross-post announcements.

Side note:
Composer 1.5 allows you to run its commands from web/
Drush 9 allows you to run commands from the project root.
That problem is solved from both directions.

ressa’s picture

Yes, sorry about that, I went a little overboard.

webchick’s picture

Status: Fixed » Closed (duplicate)

I guess this is probably a more apt status.