CLI entry point in Drupal Core

+1

I've built a POC a few years ago. That proves how easy to create a CLI endpoint for commands that only working on fully booted Drupal site. That's an example of Pareto principal. Just a few commands (like drush site-install) that need to support non-booted environment, brought so much complexity to Drush.

One thing I'd like see in this implementation is supporting command discovery in vendor directory. So you would be able bundle console commands to a Composer package (not Drupal module).

+1 sounds like solution to long lasting set of issues, it could be extendable via dex drush

PS looking how to extend it for xhprof but SF runtime backend is one more strong reason

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Overall, I like the proposal. Drush can already accept Symfony Console commands in its command list, so if the command loader was available as a public method of a discoverable class, then all dex-provided commands from modules could also be available via drush commandname, for existing Drush users, as desired.

Regarding the Drupal core cli, though, I think that the command name should remain drupal, and the work here should be limited to providing a mechanism whereby modules can add new commands to this existing script.

I think "Anything supporting a non-booted site." should be in-scope so that we can implement this as part of the existing drupal core CLI. Having an intermediary step where we have dex and drupal and then switch back to drupal later on would seem like a lot of wasted time updating docs, training materials, tests, etc..

If we can have a resolution for both existing sites and new sites, then it would be easier to adopt to using this rather than a new command.

How many commands in Drush core can work on non-booted Drupal installation?

I found just three.

• site:install
• archive:dump
• archive:restore

I believe those commands deserve own console endpoint.

Comment removed.

I think "Anything supporting a non-booted site." should be in-scope

I think the IS and other comments here make a compelling argument against this.

@acbramley, I disagree based on my previous comment about multiple commands being more confusing so solving the problem of bootstrap/non-bootstrap is important now rather than later.

The issue summary says:

Perhaps a future all-encompassing Drush replacement would utilise the drupal namespace and cleanly replace dex.

It is not necessary to build an entire Drush replacement to unify the dex and drupal front controllers. We have a paved path for doing this in Drush that wouldn't be too difficult to use here. It's harder to remove things that you don't want than it is to add something new, so I do not find it to be a compelling argument that we should introduce a new script only to immediately deprecate it.

I think we could get this merged if we unified on the drupal script name. I personally feel like we have too many different command names that are trying to do the same thing, and expanding that collection further is not moving in the right direction.

For commands like site:instlall, what if those were just Compsoer plugins? We already recommend the composer create-project command as the first command you run, what if the next command was just composer drupal-install.

Then you have a installed site, and move onto using drupal (or whatever the name is).

For the record, I prefer just drupal as the command name.

I don't know too much about the existing drupal command, but it seems to not require a booted site, so if the non-booted parts of that were just moved to Composer commands, that would free up the drupal command to become one which can be used for a booted site.

drupal at core/scripts could then be deprecated, and eventually replaced, while drupal at core/bin is what we are building here.

Edit: to be clear, I'm not saying we need to replace drupal and provide a site-install command in this issue. What I'm saying is that, if we agree that using Composer for those commands is the most sensible approach, then we basically free ourselves of the problem, and this can proceed knowing that we have a solution.

I can't help but feel like some of the strong opinions against combining dex and drupal might be motivated by an impression that taking that step would pull in a lot of complexity from the Drush bootstrap logic in order to support both bootstrapped and non-bootstrapped commands. In actuality, though, this can be achieved with a little over a dozen lines of code by overriding the Application::find() method. Drush has been doing this successfully for years, since Drush 9, and it has worked well across multiple versions of Symfony. The rest of Drush's preflight and bootstrap can be left behind. The key part look like this (in the derived Application class):

  public function find($name): Command {
    try {
      $command = parent::find($name);
      if ($command instanceof ListCommand || $command instanceof HelpCommand) {
        $this->bootstrap();
      }
      return $command;
    } catch (CommandNotFoundException $e) {
      if (!$this->bootstrap()) {
        throw $e;
      }
      return parent::find($name);
    }
  }

I have created a branch that demonstrates the above technique in action. Bootstrapped and non-bootstrapped commands live side-by-side in the drupal script. This fulfills the vision from the IS: to eventually replace dex with a unified drupal command. The rest of the code in the branch needed to make this work, beyond the fourteen lines shown above, is pretty much the same code that is in the current MR, with some refactoring. I have left the original dex script in the branch, so that it can be tried alongside the enhanced drupal script. Some enhancements from dex, e.g. the wrapper in vendor/bin, have been carried over from the original MR and applied to the drupal script. I also included an example command attached to the system module; I think this should be removed before merging to core, but it is useful to have around during development.

I think that the work done in the original MR here is really excellent. The advent of attribute-based discovery, and conformance to the latest Symfony conventions around command definition gives us a really clean interface to allow Drupal modules to add cli commands to a lightweight front controller in core. I think that we have a really good chance at getting this merged, if the community can reach an agreement on whether the script that is used to call Drupal module commands should be called dex or drupal. I hope that the minimum additions I've shown here are sufficient to convince folks that we should integrate with the existing drupal script now rather than later.

Another reason not to use Dex is it is a very common acronym in the web3 (blockchain) space (meaning Decentralized Exchange).

I have some exciting updates here.

I pushed a commit to Greg's MR in this issue. It now handles command aliases properly. I think Greg has shown how we can simply support commands which don't bootstrap. This is vital for updatedb, cache:rebuild, and a few more. And it costs us very little code with good benefit. I suggest we focus on this MR going forward. It needs a reroll against main - https://www.drupal.org/docs/develop/git/using-gitlab-to-contribute-to-dr.... Help wanted.

I also refactored all Drush commands to be vanilla Symfony Console commands. That means that Drush commands can be easily brought into Drupal core as a followup. I think now is the time for that. Please see my blog post for more on that. Lets work together to make it happen.

Awesome work. So glad to hear this is progressing.

Besides reviews, I'm happy help with linting cleanup and merge conflicts. However the MR needs to be rebased on main and I don't have permission to do that.

I've edited the MR to be against main, rebasing needs to be done locally though since there are conflicts.

Hmm, both rebasing and merging locally produce a ton of conflicts.

briefly resetting it to make it easier to see what changes are required, the merge conflicts aren't too bad.

We probably want to update the issue summary to clarify the different approach and scope in merge request 8419 from the original merge request. I think I had/have been hesitant to do that, but there does seem to be more consensus.

I spoke to soon, rebasing was hellish. I went with a simpler approach - applying the diff directly to main and fixing conflicts manually.

I've created yet another MR with these changes as I didn't want to force push over the commit history in 8419

If someone could double check my work, then we can close out the other MRs and continue work in https://git.drupalcode.org/project/drupal/-/merge_requests/14902

I ran interdiff between MR 14902 and MR 8419, @acbramley.

one failed chunk was the change in composer/Metapackage/CoreRecommended/composer.json, and the equivalent change with the correct symfony release is in the new MR.

The only changes were in composer.json and composer.lock, which are fine. I have attached an interdiff.

I would create a small CLI app in Drupal core with the ability to discover commands from installed modules. There's no need to migrate all Drush commands – just a small subset of essential ones like drush cr, drush config:*, etc. This way, Drush would become a slim extension for the Drupal CLI, especially for local development.

Issue summary still needs update, but added a comparison with current drupal command for those that came here via Moshe promoting it heavily ;) Please correct if I was wrong :)

Thanks a lot @mradcliffe, based on #36 it sounds like we can hide all the other branches and proceed with MR14902?

Hello I have been following this and related issues for a while with a lot of interest.

The new MR looks already very nice.

There is just one thing I would strongly like to suggest: change vendor/bin/dex to vendor/bin/drupal and Dex in constants and class names to something else.

Drupal console which "occupied" the name is officially sunset: https://github.com/hechoendrupal/drupal-console/commit/5db610d8b7b44ecba...
The reason is because Drush was updated in the mean time to do the same thing, and now with all the work here and in the related issues with the symfony runtime it seems drupal core is at this point too where commands authored in a contrib module run the same from drush or from the drupal core cli. This is fantastic!

So I would vote in favour of renaming it to just drupal. Do we need to open another issue to discuss the name or can someone with more authority on these things weigh in? I would be happy to assist in adapting the MR :D

I updated the table that @gábor hojtsy to add a column representing MR 8419/14902 that was initially forked by @greg.1.anderson as I think there is some confusion.

Also I am not entirely sure what File Size means in the table when in reference to patch since there is no patch for the existing column but it has a patch size? Can you clarify what you meant by that row, gábor?

The Core Committers informally discussed Moshe's high-level proposal in Slack. There is agreement that a system-wide CLI would be a good thing for Drupal. So structurally, the idea has a place.

A few ideas were shared, but none were discussed or explored in depth. Nothing here reflects a formal decision. Think of these more as initial reactions from various Core Committers. I tried to summarize the discussion as best as I could.

1. Some asked what the maintenance cost would be for core? How many hours per week do the Drush maintainers log in maintenance and feature development?
2. There was discussion about the risk of an unfinished transition. Some were concerned we might repeat past patterns: start strong, ship a partial solution, and then lose urgency to complete the work once Drush becomes a dependency.
3. Instead of depending on Drush first, some wondered if it would be better to identify the most-used Drush commands and implementing those directly in core, rather than making Drush a temporary dependency.
4. There was also some talk about having to manage additional dependencies. Drush brings its own dependency tree, which may not all be needed for a core CLI.

In short: there is openness to the "what" (a core CLI and to gradually covering common Drush use cases). The discussion was mostly about the "how". A possible next step would be to outline a few implementation options and their tradeoffs?

Do you think such a big endeavor deserves community initiative?

It might be safer to open a new issue for the naming just so it doesn't cause lots of traffic on this issue, then we can bring the results back here.

Thank you to the core committers for considering my high level proposal and to Dries for summarizing that discussion. I'll first reply to each point.

1. Maintenance for Drush averages about 4 hours per week. It is less in recent years as the set of commands has matured. A lot of the time is "self-induced". That is, the maintainers are choosing to switch the format of site aliases or switch to Console commands.
2. This is a risk with any ambitious project. Should we not attempt ambitious projects? See my final paragraph for more thoughts.
3. If this is what the product and core committer teams want to do, I'm all for it. See my final paragraph for more thoughts.
4. This would be a temporary situation n my proposal, and we can eliminate it if we decide not to depend on Drush

The biggest risk for Drupal is that this issue stalls for a few more years and and our users continue without a built-in CLI. The best way to avoid that outcome IMO is to launch a community initiative for CLI in core. That team will then work through details around how the CLI gets staged into core (using an outline like Dries proposed). Like I said, I am not particular about whether we depend on Drush temporarily or not. If we don't depend on Drush, its important that the initiative have one or two core committers attached because there will be a lot of MRs each with code+tests.

I went ahead and pulled the trigger and updated the issue summary to use the standard issue summary template, and tried to summarize a proposed resolution regardless of the MR. I included the core committer discussion and tried to summarize @moshe weitzman’s response into the Remaining tasks section. And added some further tasks/questions to answer.

I copied the issue summary from 2024-06-24 by dpi into a markdown document and generated an accessible PDF, which is attached to the issue now to make it easy to read the original proposal.

+1 to community initiative for implementing drush to core CLI commands.

Do we think that an initiative plan is necessary before this issue is RTBC?

Regarding #45.2 I don't think Dries' #45.2 is an accurate summary of the discussion.

The 'risk' of an unfinished transition specifically relates to bringing drush in as a composer dependency of core and getting stuck in a half-way situation for an extended period of time. This doesn't really relate to bringing specific commands in.

So it's really part of point #3 which I can try to expand on my own concerns about.

drush 14.x has a dev dependency on core 11.x, but it would be going into 'main' (and possibly 11.x too).

It also supports Symfony 6 and 7, whereas main only supports Symfony 8 now.

So we'd need to keep the various dependencies in sync, adding support in drush before we can require things in core. Those kinds of changes have to happen in drush eventually now, but it doesn't matter if things don't line up for a few weeks at a time, a composer dependency would make them all hard blockers.

But also because drush commands have various dependencies on core APIs, sometimes we'd make changes in core that could break drush too and those would happen in core first. So we'd end up with hard-dependencies in the other direction as well. Again it's fine if main breaks drush for a few weeks at the moment, but it wouldn't be if we started having test coverage for it and similar.

drush also has to support multiple versions of core, but a command in core only has to support the version it's in (e.g. an command in 11.x doesn't need to work with 12.x or 10.x).

This isn't the first time this kind of concern has come up, we've never figured out how to bring in 'Drupal code' as a composer dependency of core, it's always ended up either moving things out of core or fully moving them in because that results in a one-way dependency.

So for me it would be easier to start bringing individual commands over. There are some very obvious commands (updb, config:import, cr) that everyone uses constantly. I don't think anyone had any specific concerns about doing that at all.

If all the commands that aren't in core stay in drush for the forseeable future, and we don't need to decide where any commands that eventually don't make it in need to live for a quite a while, that's more of a question about what drush looks like at the very end of the process which can be figured out later.

Thanks @catch for elaborating on the "risk of unfinished transition" issue. That makes a lot of sense. How about we skip the Drush dependency and do the following:

1. Announce a community initiative or similarly "staff up" a team for this
2. Complete this issue (a barebones CLI in core)
3. Complete many issues introducing key commands. Some will come from Drush, some need rewriting from scratch.

I pushed some fixes and renamed the program and classes to drupal`from dex. The test module remains with dex in the name for now.

We have 1 test failure in testComposerLockHash. Also phpcs needs work.

The merge request needed a rebase again so I rebased it.

w/r/t the failing \Drupal\Tests\ComposerIntegrationTest::testComposerLockHash:
@dww suggested that we run composer update nothing, and that worked! The test passes after running that because bin/dex was still in bin:

diff --git a/composer.lock b/composer.lock
index e984dfdbcf6..989288136ea 100644
--- a/composer.lock
+++ b/composer.lock
@@ -368,7 +368,7 @@
             "dist": {
                 "type": "path",
                 "url": "core",
-                "reference": "8bd47715a85ebca9a9e31c0d162f7b5a58e93f1d"
+                "reference": "df840a689a3eca0b6139302c75e7bdf7ac89ecc4"
             },
             "require": {
                 "asm89/stack-cors": "^2.3",
@@ -455,7 +455,6 @@
                 "drupal/core-version": "self.version"
             },
             "bin": [
-                "bin/dex",
                 "scripts/drupal"
             ],
             "type": "drupal-core",

Taking dex out of bin hashes it correctly and the test passes.
Edit: Oops, I did it like regular markdown, fixed.

For documentation, I think that we should add a new section to core/core.api.php

1. A new dock block section.

/**
 * @defgroup cli Drupal Command-Line Interface
 * @{
 *    Some documentation describing how to implement a command?
 * @}

2. Add an @link under additional topics at the top of the file.

Adding meta issue.

parent is wrong)

Thanks, @andypost. Too many open tabs. :(

I resolved the package_manager test failures, but this is still Needs work because there are pending review items to fix from @longwave and myself. As well as documentation and merge request clean up (some class rename and example command todo).

Renamed dex_test module to console_test and adjusted class names. Updated one final reference to `dex` in a core.services.yml comment.

I have addressed the MR feedback that I knew how to address. I would appreciate if someone else could take a pass at the remaining items.

I resolved some of the items.

There seems to be a random FunctionalJavascript fail that is not related to any changes:

Drupal\Tests\contextual\FunctionalJavascript\DuplicateContextualLinksTest::testSameContextualLinks
Error: Call to a member function isVisible() on null

core/modules/contextual/tests/src/FunctionalJavascript/DuplicateContextualLinksTest.php:58

Re #72: It should have been fixed upstream with #3583187: [random test failure] DuplicateContextualLinksTest::testSameContextualLinks. Try rebasing.

Thank you, godotislate. I rebased the merge request.

I added the Needs change record tag because we should document that the preferred method of running drupal script is now vendor/bin/drupal, and also we should include instructions for DDEV ddev exec vendor/bin/drupal since it is the recommended local environment.

Documentation mentioning core/scripts/<drupal. I think that the ones that are related to spinning up a demo site using a direct clone of the Drupal project are less of a priority than ones that mention using composer create-project.

• User Guide: 3.4
  • Note: eojthebrave is currently refactoring the user guide into GitLab pages as part of #3470837: Decide on the future of the user guide
• How to download and apply recipes
  • There are GitLab pages documentation that may need to be updated as well
• Using Composer to manage Drupal
• Drupal QuickStart command
• Updating Drupal core via Composer
• Contribution tools page
• Starterkit theme
• Drupal theme folder structure
• Creating sub-themes
• Drupal for Developers

And we should probably update INSTALL.txt too in the merge request?

Note that `vendor/bin` is in DDEV's path, so `ddev exec drupal` works

Also note that `drupal` is IMO a really poor old thing because it only does sqlite3 and is so very limited. I'd hesitate to even mention it.

Also note that `drupal` is IMO a really poor old thing because it only does sqlite3 and is so very limited. I'd hesitate to even mention it.

Not for long. :-)

If you're working on `drupal` (yay!) could you give it a better name? That is really a horrible name that does not say anything about its actual usage or intent. Maybe `dcli` or something?

Re #78: I haven't followed the thread in full, but there's been a lot of discussion about the name. If there hasn't been consensus already about what the name should be, then I think it'd make sense to take the suggestions in #40/#44 to open a new issue to talk about the name so that efforts here can continue to go forward.

I updated INSTALL.txt instructions for quick-start.

I manually tested with composer repositories setup to pull from a symlinked local repo. Ran vendor/bin/drupal from outside the webroot, and it installed quickstart. That is less characters to type to demo drupal.

Regarding #78, We came to the consensus not to change the name for this merge request as it would break BC with existing core usage, and there was a strong preference in previous issues to keep the drupal CLI drupal so +1 to #40/#44 for continued discussion there.

Naming issue created separately as a child of this ticket.

I removed BootableCommandTrait because it is duplicative of DrupalApplication->bootstrap(). The 3 commands that used this trait (recipe, recipe:info, content:export) are now “regular” services discovered by ConsoleCompilerPass. Those commands may now use Autowiring, emit Console events, etc.

I added a logger.console service which listens for log messages that happen anywhere in the request and emits them to stderr. This is tested in ConsoleTest and RecipeCommandTest. This works for any command that bootstraps - the few others need their own solution because Drupal logger channels and such are not setup yet.

I resolved a couple of review items and updated symfony/runtime to 8.0.8.

Also BootableCommandTrait may have been used in contrib. or custom modules so we may not be able to remove it outright without breaking backwards-compatibility. We should check BC policy.

I created a draft change record to document changes including the BootableCommandTrait deprecation so that can get an @see reference to it as well.

I noted that this is currently @internal and experimental, and provided guidelines for module namespacing.

So are we deprecating core/scripts/drupal to be removed in the future? The CR implies yes, but there's no indication in the MR that I can see of this.

Thank you for the review, @aaronmchale. No, we are not deprecating the script. I tried to change the wording in the CR to mention that it is preferred to use the vendor location, but not required. The actual script still lives in core/scripts/drupal, but composer will move it to an application’s bin-dir, which for most Drupal sites is vendor/bin/drupal.

I moved the testing/evaluation instructions underneath Problem/Motivation.

The merge request has 3 open threads with review items mainly so I am going to change the status to Needs review to continue open discussion threads on the merge request.

I added more manual testing instructions.

Setting back to Needs work.

I tested a similar setup (and a second time the exact same instructions that were recently added) and found the same bug.

@kingdutch pointed me to this issue. In #3313404: Use symfony/runtime for less bespoke bootstrap/compatibility with varied runtime environments, we alerady signed off on adding symfony/runtime as an additional core dependency for that issue.

The plan was that once the issue was RTBC, the dependency would be added to core separately in #3552922: [PP-1]Add symfony/runtime dependency to core (so that we don't end up carrying the dependency around for multiple versions before core actually requires it). Then, that issue would be rebased (and result in a smaller MR) once the dependency was committed. Let's use that same approach here.

Disclosure per AI contribution policy: used Claude Opus 4.7 for planning and mechanical execution (renames, reference updates, BC shim). Reviewed, tested locally, and fully my responsibility. Co-Authored-By lines in each commit.

I updated testing instructions based on the merge request changes to the command name.

I addressed @xjm’s review comments about the comments in ConsoleTest though there is probably one comment in testCommandLoader that may need work after further review.

This leaves cache rebuild. I think that if that is the final review item after manual testing, then we should remove it for later to get this back to Needs review/RTBC.

Did another review pass but this is looking great!

Setting to Needs review despite the work needed for cache rebuild command to review current merge request open threads to find anything actionable.

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Thanks, NRQB. I removed the execute permissions from core/scripts/dr* because these are added when composer installs dr into the bin directory. I ran the package_manager and recipe standard test locally as a smoke test.

Added some review comments, but this is looking (and working) great.

Bumping to major as this seems important to add, and removing "Dex" from the title as we aren't using that name now.

I think we need to figure out a way to hide those deprecation notices, I hacked this into 11.4 and there are so many that the output scrolls off the top of my terminal!

twig_tweak provides some Command implementations in contrib. To test this I applied this MR against 11.4, downgraded symfony/runtime to 7.4, and installed twig_tweak. Nothing happened at first, but then I added this to twig_tweak.services.yml:

  Drupal\twig_tweak\Command\DebugCommand:
    arguments: ['@twig']
    tags:
      - { name: 'console.command' }

and the command just works!

$ ddev exec vendor/bin/dr twig-tweak:debug

Functions
---------

 * active_theme()
 * active_theme_path()
 * add_component_context(component_id)
 * attach_library(library)
 * attribute()
 * block()
...

This is really neat!

So great to see this moving along!

Just a note to the world that `vendor/bin` is in the PATH inside DDEV's web container, so you don't actually need `ddev exec vendor/bin/dr`, just `ddev exec dr` should do. And of course we'll want a built-in `dr` command in DDEV when this gets merged (or released?).

@rfay this doesn't work on my Drupal core development setup at least, although I did set it up a long time ago and have messed with it in various ways

$ ddev exec dr twig-tweak:debug
bash: line 1: dr: command not found

I'll try it out today @longwave. The assumption I made is that it's in /var/www/html/vendor/bin in the project's web container, maybe that's not the case in your setup.

Ah yeah for various reasons I have my webroot in a subdirectory drupal so the full path is /var/www/html/drupal/vendor/bin/dr

Per Slack, @longwave wants to get this in before 11.4.0. Tagging accordingly. Got us back down to 6 open threads. Not quite ready to put this back to NR, but it's really close (to RTBC, in fact).

Note: the "Warning" in the latest pipelines is from #3588024: PHP 8.6 - Functions mb_regex_encoding() and mb_ereg() are deprecated. Once that lands we can rebase and get green pipelines again.

Meanwhile, we're down to only 4 open MR threads. They've all been addressed by myself and/or @mradcliffe. Mostly just waiting for @longwave to resolve them if he's happy with our responses. There's also one thread from @xjm about an unclear comment that's been clarified, but I haven't wanted to resolve that without their approval.

So I'm going to be bold and move us back to NR. Would be great to get more eyes on this, and see if there's anything else needed before RTBC.

Thanks!
-Derek

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

needs-review-queue-bot has lost its mind. 😂 That file is already 0644 in the issue fork branch. And, there's an MR thread open by @longwave to request we move them to 0755, too. 😅 I'm going to tag this in the hopes that the bot stops messing with this issue.

Thanks,
-Derek