Add Alpha level Experimental Package Manager module

The 3 failures on March 8:

• Drupal\Tests\package_manager\Kernel\PathExcluder\NodeModulesExcluderTest → legitimate failure → fixed by my commit just now, the test already passed! 😊
• Drupal\Tests\ckeditor5\FunctionalJavascript\MediaTest::testViewMode() → random failure

Next: porting all commits that have happened in the contrib module in the past ~60 hours, which includes the reverting of the absurd PCRE limit work-around thanks to #3346628: FixtureManipulator should only use Composer commands, rather than manipulating JSON directly and the removal of the runtime dependency on composer/composer, which is why the diffstat for the last commit is a whopping 33 files changed, 660 insertions(+), 1381 deletions(-)! 🤓🥳

We have only a few alpha blockers left at #3319030: Drupal 10 Core Roadmap for Automatic Updates, and #3321905: Add colinodell/psr-test-logger to core's dev dependencies + #3331078: Add php-tuf/composer-stager to core dependencies and governance — for experimental Automatic Updates & Project Browser modules are sibling core issue that will add the 2 new dependencies that the Package Manager module needs!

Ted is working on an issue summary update! 🚀

#3321905: Add colinodell/psr-test-logger to core's dev dependencies landed 👍

#3331078: Add php-tuf/composer-stager to core dependencies and governance — for experimental Automatic Updates & Project Browser modules still needs to land 🤞

General questions about this bit:

The codebase must be writable by the web server. Because this is intrinsic to the purpose of this module and the two modules that depend on it, this requirement is unlikely to change in the future. Although this will prevent Package Manager from working on some hosting environments, some users may use Package Manager in local or cloud environments where the file system is writable, even if their production environment is not. Project Browser is the most obvious example of this, because installing modules is common when building a site and is often done locally. In Automatic Updates, updating manually via the UI (rather than automatically during cron) may be done by some users in development environments, which is beneficial even if Automatic Updates cannot be used directly in the production environment.

With automatic updates, it's my understanding that unattended updates will be done on cron - this means potentially a different user to the webserver running cron with access to the filesystem, and webserver write access not being entirely required for that situation.

Can automatic updates be configured like that - unattended updates via non-web cron only?

If so, would it be feasible at some point to have queued installs and updates - i.e. you select what you want, it goes into a queue, cron runs the queue (this might need to be something like key/value expirable instead of a real queue so we could show pending status), package_manager installs your stuff, it shows up later. It would require a frequent server-side cron run but pretty sure cpanel and similar can do that. This might make things available to a slightly wider number of users although a fair bit of UX to overcome.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

#13:

This might make things available to a slightly wider number of users although a fair bit of UX to overcome.

Absolutely!

Is this a nice-to-have or a must-have?

Ideally we'd have (approximate) numbers for this. We probably can't get that. But maybe we have some experience with it? Do you think it's more like 1, 10 or 50%?

#14:

Additionally we could even do the staging of the install/update into the stage directory at the time the user does the selection.

I was first gonna disagree, but no, this is totally right, because the creation of the stage does not require the ability to write to the codebase, only to the filesystem (similar to just uploading files). That'd definitely make the UX concerns @catch raised less severe, because feedback would be happening together with user actions, instead of at a later time.

I would think this functionality would be more important in Automatic Updates than in Project Browser because this would allow more site to select security updates.

IMHO what @catch described would only provide an acceptable UX for Automatic Updates, not for Project Browser. Project Browser IMHO requires the ability to see the results of what you're doing at the time you're doing it, otherwise it'd be a nightmarish UX.

The requirement that the web server can write its own files is quite a bummer. We spent considerable effort ~15 years ago in Update Manager to let it work on sites that the web server can write to, and on ones that are more securely configured. For all the effort we're pouring into making a slick, new, "more secure" version of all this plumbing -- to have to tell everyone "but open yourself up to a bunch of wider escalation vulnerabilities by making sure your webserver can write to its own source code" seems like a major step backwards. 😢

I guess we don't really know the actual numbers of how many sites are more securely configured or not. Certainly the UX is more slick and easy if everything is writable. And if folks are really security conscious, they're probably not going to use any of this. So, to help with the long tail of sites not upgrading, most of them are probably on server-can-write configurations and it doesn't really matter. But it's unfortunate we can't play similar tricks in Package Manger that we are in Update Manager to have "sudo" step if the server can't directly write to its own files...

This is a bit of an unsettling sentence:

And if folks are really security conscious, they're probably not going to use any of this.

That makes it sound to me like Automatic Updates will be (more or less) just a toy? In my mind, using Automatic Updates should offer the highest level of security, and not require an insecure set up.

If it does ... well then it kind of defeats the whole purpose of installing it in the first place, as I see it. Or maybe you can't have one (Automatic Updates) without the other (insecure set up)?

Thanks for the detailed feedback, @tedbow! A few quick thoughts for now:

1. Definitely do NOT want Drupal to ever save such credentials for later use. That would 💯 defeat the purpose of defense-in-depth here. 😂
2. I'm not necessarily proposing we need to recreate the authorize.php experience, or re-use any of that code. I'd be shocked if the "API" (such as it is) that we came up with 15 years ago would actually work for these new use cases.
3. If the run-a-command-via-cron-as-the-"super"-user approach is easy enough to include, that'd be great. For sites that care enough about security to have the files owned by a separate user, they can probably figure out how to RTFM enough to setup such a cron job. We probably don't need a UI-based version of this at all anymore. But if it's relatively easy to figure out how such a UI would work and fit in, and build all the plumbing (and counter-tops) now such that we could bolt such a sink into the house at a later time, great.
4. I'm not totally convinced that this is a no-go for Project Browser. The UI will already not be "instantaneous" -- we've gotta download things from the Internet, after all. There's already going to be a "waiting while we download (and have composer fetch all the dependencies) step. If "waiting for the download to start" takes 0-60 seconds (until the next cron job fires), that's not going to be the end of the world. I think it'd be worth building such plumbing into the PB workflow, too, and not immediately rule-out defense-in-depth with the explanation that "users will expect immediate results"... It's already not going to be immediate, the UI will have to handle that gracefully...

Gotta run, more later (I hope).

Thanks again!
-Derek

Ideally we'd have (approximate) numbers for this. We probably can't get that. But maybe we have some experience with it? Do you think it's more like 1, 10 or 50%?

I don't have data to back this up, but here's my hypothesis. From small/low-traffic sites to large/high-traffic sites, I think we have the following hosting categories:

• Generic shared hosting
• Specialty shared hosting
• Single server VM, VPS, or dedicated hosting
• Multi-server

I think the first category is the largest in terms of number of sites, and I think the vast majority of sites in this category already have a PHP-writable codebase, because the shared hosting provider gives them a single user account that's the same for ssh, sftp, and php.

For the second category, an example of specialty shared hosting might be a university with an IT department managing a pool of servers, but where each department or team can get its own website. Such an IT department might be savvy enough to provision each internal website owner with 2 user accounts: one for ssh/sftp and a different one for php.

For people running on VMs (e.g., Digital Ocean droplets), VPS, or dedicated, Drupal's docs should recommend that they set things up so that the codebase isn't writable by the web server. Even if this is a smallish percentage of total Drupal sites by number, it's an important audience, and it would be good to not steer this audience towards the less secure setup of having a webserver-writable codebase. I think this is the audience we should be thinking about for #13.

For high availability or high traffic sites that require balancing load across multiple web servers, the automatic updates module proposed for Drupal core can't be used on its own. The site would need its own deployment process to make sure the code on all web nodes is kept in sync. However, the site could setup a staging server that runs automatic updates and then triggers the site's deploy-to-prod process. Depending on the other security checks involved in that process, it might or might not be important for that staging server to have its codebase be non-writable by the web server.

@effulgentsia: That's a helpful way to categorize things in #22, thanks.

FWIW, I've been using DreamHost as my "Generic shared hosting" solution for many years. It's definitely a big player in that space. One of the key features that drew me to it in the first place was that it does let you create different users for different things, and although it takes a little extra work, you can (and I do) setup sites there so that the files are owned by a different user than httpd runs as. I'm sure I'm in the minority on that approach, but pointing out that even some of the "Generic shared hosting" solutions can still be targets for making this work (in some way) on more secure-in-depth sites.

Yup. You can ssh into a host and use crontab directly. I never mess with cpanel. So yes, in principle, this would all work there with the separate cron job…

Is this a nice-to-have or a must-have?

#3351895: Add Drush command to allow running cron updates via console and by a separate user, for defense-in-depth might be must-have (although I'm not sure why running regular Drupal cron via drush and switching off automated cron isn't enough to cover that?). I don't think any of the rest of this is. However it might be necessary to get x% extra sites to adopt automatic updates.

IMHO what @catch described would only provide an acceptable UX for Automatic Updates, not for Project Browser. Project Browser IMHO requires the ability to see the results of what you're doing at the time you're doing it, otherwise it'd be a nightmarish UX.

If I install something via the UI via Ubuntu's package manager, or via google play on my phone, I usually hit 'install', start doing something else, and then somewhere between 15 seconds to 3 minutes later get a notification that the app was installed. If I'm installing over 4g sometimes it has to give up once or twice and restart and takes a lot longer. You do get the 'percentage complete' stuff but as we all know that's a lie anyway. The fact that it happens in the background is a major feature in fact. This is also the case for manually triggered updates on those environments too. So it's a pretty standard pattern that installs and updates get queued, and eventually they get done, and then you can check back and/or get a notification that it happened. It's also IMO a better UX than 'tough, you can't use this unless you change your file permissions' and it's also a much better UX than when installs and updates used to block all other operations back in the day.

@effulgentsia:

I think this is the audience we should be thinking about for #13.

Yes this is also broadly what I have in mind - people who aren't on call to update their website on Wednesday afternoons, but can follow 'advanced' instructions step by step to get their server/Drupal configured decently. Or just people who might or might not use automatic updates, won't mind configuration it to work on their system (like adding a cron job and adding something to $settings or similar), but won't want to specifically configure their entire system to work with automatic updates.

#22: insightful analysis, thanks @effulgentsia! 😊

#26:

although I'm not sure why running regular Drupal cron via drush and switching off automated cron isn't enough to cover that?)

Agreed. But I suspect @dww's reasoning is that drush should not be required to be installed on the web server, Drupal core alone should be sufficient? @dww, can you elaborate? (I'd really rather not be guessing!)

The fact that it happens in the background is a major feature in fact. This is also the case for manually triggered updates on those environments too.

Fair! But a key difference is that Drupal does not have UI concepts in place for keeping the user informed of these kinds of things… there's never been a concept of "background tasks" and keeping the user informed of that. Although one could argue that that is what cron does, and then a Drupal status message would be the equivalent of a notification? (Easily missed/ignored though 😅)

Is that what you had in mind?

We might want to split this bit off to its own issue, but keeping going here just for now given it's currently pretty linear.

Although one could argue that that is what cron does, and then a Drupal status message would be the equivalent of a notification? (Easily missed/ignored though 😅)

Yes something like that. If we don't use queue itself, but instead key/value or similar, then we could have a constant message telling you the update/install is pending (maybe only on the project browser/automatic updates UI) and an extra message telling you it's done. We could keep something in session referencing the pending operation(s) then check status based on that, so that the logic only needs to run for the user who triggered it + maybe for everyone on the project_browser/automatic_updates UI pages in the case of multi-admin installs.

If you miss/ignore the notification, does package_manager (or project browser/automatic updates) maintain logs for what it's done? It seems useful especially for automatic updates to be able to easily see that x module was updated from x.y.y to x.y.z release (on the site rather than an e-mail), and then that could also show installs. Since updates might trigger installs too when dependencies are added, any such log probably needs to show both anyway.

I don’t exactly care about drush cron vs a dedicated script, and I’ve never raised any such concern.

I’m raising the flags that designing Package Manager to not handle the non-writable file system case is a mistake (that we can still correct) and that saying “but even if we did, it’ll never work for Project Browser because users expect immediate feedback” is misguided since nothing about Project Browser will actually be “immediate”, anyway. I’m advocating (like I have for many years) that all these update-your-site tools should work when a site is configured “correctly” 😉 to not let httpd write to its own files.

I posed the question about the Project Browser UI having a "waiting for download to start" phase, and got confirmation that:

1. The PB UI already has a progress spinner / waiting for stage X.
2. Adding another phase would be a small incremental change to the existing UI, not a whole new kind of problem for them to solve.

So I believe there's no reason to claim PB can't work with defense-in-depth, too. It's already not instantaneous, and another phase isn't a game changer for their UI. But supporting the non-writable filesystem case would be a game changer for that (subset?) of users who configure their sites "correctly". 😅

Slack transcript attached (with permission). https://drupal.slack.com/archives/C01UHB4QG12/p1681848852629629

Thanks,
-Derek

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

MockPathLocator: This class allows tests to have a directory other than the real Drupal codebase used as the "live" directory. This lets us freely alter the "live" directory without altering the codebase that is running the test.

Without reviewing all the test coverage in detail, the first question that comes to mind here is 'why not use build tests for this?', see for example ComponentsIsolatedBuildTest - was this attempted? Is there an issue I can look at or documentation for why not? I see a couple of build tests, but they aren't using MockPathLocator which seems to be all for kernel tests.

Re #40: Package Manager has unique needs -- we need to move files around and simulate a "fake" Drupal site, but we also need to be able to test the APIs directly. Build tests are too "far" from the APIs to be useful for this; kernel tests do not, by default, deal with real files in a filesystem.

So, rather than try to get a build test to boot up a kernel in a fake site that isn't bootable, we ended up adapting kernel tests so that they can use Package Manager's APIs with a small simulacrum of a Drupal site based on real files. That's what almost everything in PackageManagerKernelTestBase and package_manager_bypass are there to facilitate. It's simply an easier lift than doing it the other way around.

So, rather than try to get a build test to boot up a kernel in a fake site that isn't bootable, we ended up adapting kernel tests so that they can use Package Manager's APIs with a small simulacrum of a Drupal site based on real files. That's what almost everything in PackageManagerKernelTestBase and package_manager_bypass are there to facilitate. It's simply an easier lift than doing it the other way around.

This sounds like a good explanation - I'll try to take a closer look soon to get my head around it a bit more.

It's very hard to tell just from the issue summary what the remaining work is here.

#3319030: Drupal 10 Core Roadmap for Automatic Updates is now linked (it wasn't previously), but specific issues which will result in changes to package manager prior to an alpha commit, or core bugs that are blocking PM should ideally be tracked directly in this issue - even after reading the roadmap issue, it's not clear to me which ones block PM, which ones block AU, which core bugs are just bugs that affect the modules, or which will unblock changes AU to PM that are still necessary etc.

So this seemed to stalled, is there any help needed? Don't have the role to make any of the calls though.

@catch wonder if you have an answer or know who to ask regarding #46?

@effulgentsia just closed #3385644: [policy, no patch] Consider whether to keep Package Manager and Automatic Updates in a separate repo/package than core in order to facilitate releasing updates to the updater as 'works as designed' so I think we can call that done now.

So does that mean this should continue? Definitely needs rebasing.

I think the best next step to move this issue forward is to get reviews of the code in https://github.com/php-tuf/composer-stager. @TravisCarden just recently (2 weeks ago) released 2.0.0-beta4. It's only in beta status because it hasn't been reviewed much by people other than those working on Package Manager. As far as we know though, it's feature complete and can be marked RC or stable once it's adequately reviewed. The most noteworthy recent change is that per #3416542: [policy] Require rsync for automatic updates in Drupal core and punt other syncers to contrib we removed the PHP filesyncer so now only works if people have rsync installed. This makes the package easier to review and easier to maintain. If there's a significant population who's on systems that don't already have rsync installed and can't install it, then a separate GitHub repo/package could be created (if someone were willing to maintain it) containing alternate file syncers (e.g., one that uses the Windows robocopy command for people on Windows), but those would not be "part of core" (i.e., maintained by Drupal core committers). The Drupal issue for discussing stuff related to Composer Stager is #3331078: Add php-tuf/composer-stager to core dependencies and governance — for experimental Automatic Updates & Project Browser modules. Issues and PRs could also be filed in the GitHub repo.

Beyond the above, @tedbow: it would be good to get an updated MR of Package Manager into this issue.

https://github.com/php-tuf/composer-stager/issues/350 got completed so now Composer Stager is compatible with both Symfony 6 and 7, so I think the MR in this issue should include it so that tests run properly.

This needs another rebase - conflicts in composer/Metapackage/CoreRecommended/composer.json

Did a not at all comprehensive first pass review to try to get things moving here.