[policy, no patch] How much of The Update Framework integration is needed for alpha-level review/commit of Package Manager?

The signing and verification toolchain and d.o-side support for core and contrib release signing is required prior to Package Manager (and therefore the modules dependent on it) being beta experimental in core.

It is not an Alpha requirement of any of the 3 related modules

@xjm and I discussed this, and this is incorrect. TUF blocks alpha for all 3 modules, because a security review blocks alpha, and TUF is needed for a proper security review.

Being new to this initiative since November, this definitely was a shocking surprise 😳

Looking forward:

1. Is it then an accurate conclusion that 100% of package_manager, automatic_updates and project_browser are blocked on #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged?
2. Are there other d.o infrastructure issues that are blockers?

I have not been following this closely, but I guess now I'm following it a bit closer because this has come up.

A thing I am not clear on is this, which is a copy and paste of Ted's last comment from that slack thread (assume it's fine to quote it since it was linked):

@xjm
I had a question about the scope of the full review and when that could be done.
Will the full security review include all the code that generates the TUF artifacts as part of the packaging pipeline on the drupal.org server side, including https://gitlab.com/rugged/rugged? I also assume there would at least be some non-public configuration(maybe code) outside of Rugged.
Or will the full review be limited to TUF artifacts that the drupal.org server side provides and the client code that ships with drupal (either directly or as a composer dependency) and interacts with those artifacts?
Wondering because this question probably determines when we can consider TUF server side portion done as far the AutoUpdates system being in place where it can be ready for a review

It's my understanding that rugged is deployed to d.o staging, that there is one known rugged blocker for php-tuf (hashed bins). Once hashed bins is available (in the client and server), then it should be theoretically possible for php-tuf, package_manager, and d.o staging to work together as a proof of concept.

However there are various other rugged blockers (queue crashes, terabytes of storage duplication) which won't affect php-tuf but do make it not deployable to d.o production yet. And it might not be possible to sign everything on d.o staging due to the scalability/storage blockers or at least not without as-yet-unimplemented workarounds.

That means, assuming the above is accurate now, and stays accurate, that there is a window between php-tuf and the code that depends on it being theoretically alpha-complete, and TUF being deployed to Drupal.org production.

It seems like a code review of php-tuf and friends could happen at that point - at least, that would be an option for when reviewers could start reviewing if not finishing.

But even if review can start then, it probably wouldn't make sense to review rugged until it's about ready to deploy to d.o production. Also it might not be possible to sign all packages on d.o staging, meaning user testing would be hampered (assuming we can even point end-user composer.json to d.o staging in the first place) and probably pen testing too. But we're not reviewing rugged for core inclusion, it's context for a php-tuf review as far as core is concerned if not the entire system.

For me it would make sense to at least inform people that different aspects are possible to review when they are, then people can either review early and often as different elements are available, or wait, depending on availability and how they want to approach things. But there would need to be a clear outline somewhere of what needs reviewing, what can be reviewed, what can't etc. at the moment I have no idea where to find this information and asking 2-3 different people gives 2-3 different answers.

Getting this many in-depth reviews from this many people for this many moving parts is going to take a while, so the easier it is to know what to review and when, the better.

I'm retitling the issue to reflect a more nuanced question.

#9 highlights some of the things that would need to happen for a full security review of Package Manager. My concern with that is if we require all of that for initial, alpha-level, commit, then we're effectively putting Project Browser reviews on hold for months, potentially making the difference between whether Project Browser can be released in 10.2 vs. 10.3. Unless we think Project Browser reviews can happen without Package Manager being in core, but I'm not sure if we have experience with prior core additions being successfully reviewed ahead of their dependencies landing.

I wonder if it would make sense for an alpha-level review (and commit?) of Package Manager to include TUF integration, but against the staging TUF repo, without hashed bins, with only some (10? 100?) modules signed, and without solving the other Rugged issues that still need to be solved before it can be deployed to production.

The downsides of the above would be:

• If you enable Package Manager (and when it's ready, Project Browser) in their alpha state (which only core testers can do, since alpha modules aren't released in tagged releases), you'll only be able to install modules that are in that subset of modules that are signed on staging.
• The security review performed for Package Manager alpha would lack the ability to review Rugged's hashed bins implementation and the other Rugged issues that aren't done yet. So, additional security review will be needed after those issues are done. That would potentially add work for security reviewers, and risk some security bugs not getting caught due to the reviews being separated.

The upside would be:

• Project Browser (and possibly Automatic Updates?) reviews wouldn't be blocked on finishing Rugged.
• As a result of ^, PB and AU would be more likely to ship in 10.2 vs 10.3.

Hm, for some reason, I seem unable to change the title of this issue. I think the title should be:

[policy, no patch] How much of The Update Framework integration is needed for alpha-level review/commit of Package Manager?

I wonder if it would make sense for an alpha-level review (and commit?) of Package Manager to include TUF integration, but against the staging TUF repo, without hashed bins,

I don't think this is feasible, from comments by @phenaproxima in slack somewhere, the problem is that the metadata file without hashed bins is something like 200mb, which will crash when you try to parse the JSON. Also the client needs to support hashed bins for when it's available, and there is other stuff to do in package_manager before it's ready to review anyway, so hashed bins at the moment is unlikely to block review for long or at all, and package_manager can't work without it.

There's some misunderstandings here still so I'm just going to summarize some more from the ongoing Slack discussion.

TUF blocks Package Manager alpha for the following reasons:

1. TUF is a significant part of the Package Manager MVP and will be a nontrivial amount of code, so Package Manager doesn't make sense as an alpha module without it.
    

2. For the same reason, doing security review without TUF implemented doesn't make sense; Package Manager is fundamentally an insecure design without it and we know that, so there's no reason to do the needed security review for alpha without that code being implemented.
    

3. The contrib AU module currently mitigates the lack of TUF signing with:

   • Attended-only updates
   • Requiring HTTPS

   We can't require HTTPS for core, even for alpha, because not all hosts support it and so the module could put a site on such hosts in a state where nothing can be installed or updated. This is a serious data integrity and upgrade path problem, which is one of the few reasons release management will get involved in alpha requirements and say "no" to something.

However, we were willing to allow a Project Manager beta that just provided composer commands (and did not install anything) which therefore would not be blocked on Package Manager. We offered that option to the initiative team, but from their perspective it wouldn't actually be helpful to them due to the extra overhead etc. For PB, they are comfortable targeting 10.2 once Rugged/TUF is implemented and deployed.

A similar approach is not available for Autoupdates, because the whole point of getting it into core is to review and test the tool that writes updated Drupal core code to the filesystem, and so there's no functionality to make an alpha of if you take away Package Manager support completely.

The decision to require functional TUF integration code prior to any core commit has now been re-discussed and re-made in:

1. The original initiative core conversation at DrupalCon Nashville (for the abstract concept of package signing and validation)
2. August 2018 (for what would become the csig equivalent in the 1.x version of the module)
3. August 2020 (for TUF, which we'd decided to adopt by Feb. 2020)
4. June 2022
5. November 2022
6. February 2023
7. March 2023

Edit: I would appreciate it if folks responding here would not try to re-litigate workarounds to any of the above points, and treat the decision as made, in the interest of maintainer health and bandwidth if nothing else.

Edit 2: Apologies that my initial comment mentioned beta and not alpha. Did not mean to add to the confusion. I thought the sense of urgency was coming from trying to get a beta done in time for 10.1, which is not achievable. There are no release deadlines associated with alphas.

@effulgentsia and I spoke more about this issue.

We both agree that a working TUF implementation is a requirement prior to a core commit of autoupdates.

The main point we disagree on is whether TUF is also a requirement for an alpha of Project Browser, and whether the tradeoffs for the review process are a net loss or gain overall.

@effulgentsia believes what Project Browser does is sufficiently similar to the current module download functionality, and that we could safely require HTTPS instead of signing and verification as a temporary alpha requirement.

I on the other hand am concerned about the drawbacks of adding such an incomplete Package Manager to core in terms of the review process, user testing, and security, and I'm also concerned about the complications and possible critical disruptions to alpha testers from adding a temporary HTTPS requirement. I furthermore think it adds overhead and risk to the review process without improving the release timeline for the modules.

However, we both have the same goal: Start code reviews and user testing for all three of Package Manager, Automatic Updates, and Project Browser as soon as possible, and smooth the path to core inclusion.

The compromise we agreed on is that we should begin core maintainer reviews of the Project Browser contributed project as soon as possible. We also should expand efforts for user testing of composer-stager, Automatic Updates, and Project Browser using the contributed projects. Finally, we can potentially open a core merge request for Project Browser sooner rather than later, even though it may not be committed until the working TUF implementation is added.

To this end, I think we should create a core issue for maintainer code review of the Project Browser contributed project (tagged for release manager, framework manager, and JavaScript review). We should also plan user testing sprints at upcoming events, and have some communications promoting user testing of the modules.

First round of review! This is looking great already 😊

Wrong browser tab 🙈 … and so consider this a serendipitous "bump" 😅

Since #18, #3351190: [policy, no patch] Should Package Manager require Composer HTTPS? has had further discussion and the corresponding issue for the Package Manager module (#3351247: Harden our HTTPS requirement) landed 5 days ago.

That means this issue indeed needs review. Restoring that state.

Hashed bins appears to be implemented in rugged as of a week ago, that leaves one outstanding ruggeed issue listed on #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged, which as far as I know is more of a nice to have or possibly production blocker. I think that means TUF on d.o staging is unblocked now.

That's all within just over a month since this ticket was opened, so to me it doesn't look like TUF is a long term blocker, it looks like the last few bits are being ironed out in testing and getting fixed promptly enough - at least enough to run on staging so everything can be reviewed.

#3316617: Add a validator to check that PHP-TUF's Composer integration is present and configured correctly has landed. That means we're now just hard-blocked on d.o infrastructure at #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged to actually enable what #3316617 introduced, and we have #3358504: Require PHP-TUF's Composer integration plugin ready and waiting for that 👍

While TUF isn't deployed on d.o packagist in production yet, it's mostly deployed on staging, production blockers are well documented and in progress, the security audit has started and is going well - so for me all the things I was concerned about in March have been met by now (or months ago, but I also wanted to get #3351190: [policy, no patch] Should Package Manager require Composer HTTPS? resolved properly too). i.e. it should be possible to holistically review package manager with PHP TUF together, and it'll be valid for when it's all deployed and working together because major changes aren't still in the pipeline. I don't know the situation of the external audit for PHP-TUF and friends but sounds like that also might start sooner than later.

I would also like to see us commit PM/PB/AU as early as possible in the 10.3.x/11.x cycle so that they get a good round of alpha/beta testing before the next minor release.

Additionally, because we'll have the 10.3.x and 11.x branches open at the same time (which we have not had an equivalent situation since 9.5.x and 10.0.x), if we remove/disable guardrails, it should be possible to test updating from 10.3.x to 11.x and/or from 11.x alpha releases to later 11.x alpha releases, without actually having to have automatic updates in a tagged stable release as beta. This would make a core commit as alpha a lot more useful than it would have been when it would only have been alpha in one branch.

Are there any remaining concerns here? Or if not it would be useful to sort out the actual alpha reviews a bit more systematically.

@tedbow

Regarding "alpha/beta" right now we restrict target version to RCs and above but that could easily be removed with a test module that beta testers could use.
The bigger problem is "alpha" testing. Since alpha experimental modules are not in releases I think this will be much more difficult. The module is not meant to be used with dev versions so that would problem with starting from dev version at the alpha commit.

No it should be fine from that point of view too, this is what I meant with:

if we remove/disable guardrails, it should be possible to test updating from 10.3.x to 11.x and/or from 11.x alpha releases to later 11.x alpha releases

We release alphas (actual tagged releases with release nodes) of the new major version well in advance of a beta release - we did this for 9.0.0 and also 10.0.0. This is usually starts once changes like Symfony major updates and similar are in, whatever we decide is a meaningful target for testing, in case contrib modules want to track compatibility against alphas rather than a moving dev target. The alphas are getting less urgent from that perspective the further along things like rector get, but we'll still do them, and then that provides actual tagged 11.x releases to update to/from with automatic updates.

I also think we could do that for 10.3.x too if it helps with AU testing in a way that 11.x alphas don't (like trying an actual major update?), but didn't fully think through the consequences of that - obviously we don't want an actual site getting confused and trying to run on a 10.3 alpha released in January in production, which I don't think they'd ever do with 11.x.

Having both branches in tandem, definitely with early 11.x alpha releases and possibly early 10.3.x alpha releases is the best possible timing for an alpha commit of AU for this reason. Maybe we could do alpha releases of 11.1 and 10.4 but it's not built into the release cycle the same way.

Discussed with @Gábor Hojtsy and we agreed that from our perspective TUF should be at maximum a stable blocker for both Automatic Updates and Package Manager. We acknowledge the integral role of TUF in enhancing security for automated updates, especially when updates are done at scale. However, not having TUF should not prevent us from getting the Automatic Updates in front of the core maintainers, core developers, and early testers. We believe we should do this as soon as possible because now the module is being built in its own silo.

We understand the need for thorough testing before enabling Automatic Updates by default, including testing updates with TUF. However, tying its development to alpha/beta milestones seems counterproductive. We can see that there are benefits to being able to do a holistic security review but it sounds like a lot of this is already possible before the TUF has been deployed on d.o packagist in production per #24.

Concerning the adoption of Automatic Updates, it's clear that a majority of the user base requires a high level of confidence before enabling such features in production. While the contrib module already offers automatic updates, its limited adoption hints at the community's cautious stance on this. In our research, we found that users prefer to wait until Automatic Updates has proven its reliability over time before starting to use it themself.

If it's not a beta blocker, then how will sites running it as beta be TUF-enabled prior to the module being marked stable, so that we don't end up with a rump of sites that never opt-in?

I think package manager can technically run any composer command, so it is theoretically possible to update to a version of package manager that opts you in to TUF for d.o (which might then require a second update to a version of package manager that enforces TUF for d.o - no idea if this would be possible in a one-stage update).

Such a mechanism doesn't exist at the moment, and while it would be useful to be able to opt-in other packagist endpoints if/when they add TUF support (like the main one if that ever happens), it's adding additional stable-blocking technical debt to the module that wouldn't be necessary if it just starts off with TUF support from the outset.

To expand on #29, I think dropping this as a beta requirement adds a lot of additional work, requirements, timeline, and risk.

Let's take a site that installs 10.3.0, and enables automatic updates (experimental) without TUF support.

For automatic updates to be stable, we want the d.o endpoint to be TUF-enabled, so we now have to add a mechanism to TUF-enable this site to core, currently that feature doesn't exist, and it's not on the roadmap, as far as I know it's only brought up once or twice by me as a possibility in various issues. This is what I'm talking about in #29.

Let's say this mechanism wasn't added as a beta requirement, so we still need to add it during beta.

To test the mechanism, we need to take a non-TUF site on 10.3.0, and update to 10.3.1, and make sure it's TUF-enabled after the update.

But now, we also need to check that once that site is newly TUF-enabled, that it can self-update again (i.e. that TUF-enabling didn't break anything like corrupting composer.json), so that requires another patch release, 10.3.2.

We probably also want to add validation that d.o is TUF protected (so it can't be un-protected), but can we even add that validation until we have no 10.3.0 sites left? I don't know the answer to this, but that might be yet another patch release to add that validation - but we can't mandate that sites update 10.3.0, 10.3.1, 10.3.2 etc., they could skip from 10.3.0 to 10.3.2 so that also needs to work and be tested.

Some of the above might be wrong, I am guessing because I haven't thought about it in that much detail. But if it's right, then it's adding two, maybe three releases with complicated upgrade paths we otherwise wouldn't before AU can be stable, or alternatively leaving a rump of sites that would never get TUF-enabled unless they manually run composer commands (which would defeat the object of having package manager in the first place and also penalise beta testers).

All of this is adding a transition from non-TUF to TUF that simply would not exist if the beta-stability module includes TUF from the outset - so we wouldn't need to do any of that work.

Now the work above wouldn't be wasted, because it could be used for packagist.org down the line if they ever add TUF support, but I don't see how that would be stable blocking if we're not trying to change the TUF-status of any endpoints before stable. Only #28 would make all this extra work and testing required.

I agree that the complexity introduced by not having TUF enabled by beta Automatic Updates is not ideal. Thanks for explaining in detail the complexity involved with that. However, this risk would be only valid if we didn't have TUF-enabled with the Automatic Updates in the next release. Not having Automatic Updates in the next release seems even worse because it would mean that we would have to wait at least until 2025 for stable Automatic Updates. Potentially even until June 2025, if we follow our current policy where experimental module statuses are only changed in minor releases.

It sounds like we could try to minimize the risk related to enabling TUF after it has been shipped by introducing a beta blocker to evaluate feasibility of #30 if TUF has not been enabled. We could decide on the approach closer to the release depending on the status of TUF. If #30 isn't feasible, then Automatic Updates could not reach beta until TUF has been enabled.

The ideal scenario is still that by 10.3.0 we have TUF enabled Automatic Updates. Unless #30 introduces too much complexity, having Automatic Updates without TUF would be still better than not having Automatic Updates at all. This would enable us to start testing TUF whenever it's done, potentially well before 10.4.0. This way Automatic Updates could potentially even reach stable in 10.4.0, and be enabled by default.

Thanks for explaining in detail the complexity involved with that.

That took time that I could have spent doing literally anything else. Meanwhile #3253158: Add Alpha level Experimental Update Manager module hasn't been updated for two months. Why are people constantly bringing up different versions of this and not instead spending that time getting the module ready to commit? How many hours have been spent talking about it in the past year?

However, this risk would be only valid if we didn't have TUF-enabled with the Automatic Updates in the next release.

No not really. Beta-stability modules are expected to provide upgrade paths, so #30 comes into play as soon as package manager/automatic updates is marked beta. There is no need for it to be marked beta before a 10.3.x/11.x beta release because there's no real advantage to it being more stable than the core release it's in (not that we shouldn't if it's actually ready, but there are major advantages like upgrade paths and providing bc if it's not ready and marked anyway). The only way to avoid the extra work is to mark it beta after TUF is enabled/required because then there's no upgrade path to deal with at all. This means waiting for TUF reduces work significantly right up to the very last possible beta deadline for 10.3.

The proposal only 'works' if you were to get to the very last beta deadline for 10.3/11.x, know that TUF is not on production, and then mark PM/AU beta at that point, knowing that you'll then have to do all the TUF work and the extra upgrade path during the 10.4/11.1 cycle. But you're then putting that additional, complex, upgrade path in the way of every single beta tester of automatic updates into patch releases of Drupal core, the number of which will increase over time the longer the window is between it being available and that update being added, increasing the chance of buggy updates for something that is supposed to make them seamless.

So we would need #3394983: Automatically configure TUF Composer settings after the module is installed for new sites, and then a hook_update_N() or hook_post_update_NAME() to call the same code for sites with AU installed already. Because it'd include an update handler, it wouldn't be able to be run unattended though I assume, but those can still be done semi-manually with the AU UI?

In general I think it's fine to not require TUF for AU alpha per #24 although I would like to understand @xjm's concerns more. Bringing the beta requirement into this issue is just making what could have been a much easier decision more difficult - why not have opened a new issue for this instead of completely changing the scope of this one?

Why are people constantly bringing up different versions of this and not instead spending that time getting the module ready to commit? How many hours have been spent talking about it in the past year?

I believe this conversation has kept going because there hasn't been a decision on this. From my end, the conversation has mostly ended because I have been tired of having the discussion, but not because I believed there was an actual decision.

Bringing the beta requirement into this issue is just making what could have been a much easier decision more difficult - why not have opened a new issue for this instead of completely changing the scope of this one?

I wanted to bring the beta issue here, because I thought it was better to be transparent about our thoughts regarding the beta requirements before making a decision regarding the alpha requirements. Bringing it up later could have come up as a surprise, which could have made it look like we're trying to get rid of the requirement altogether by removing it from the requirements whenever we're deciding on requirements for the current milestone at hand.

I'm happy to see that by bringing this up, we surfaced some blockers to removing TUF from the beta requirements that seems like could be addressed. I'm fine with moving that discussion to a follow-up issue.

Bringing it up later

In its own issue doesn't imply later.

We should probably update the issue summary since it specifically asks the following questions at the moment:

Is TUF integration an Alpha, Beta, or Stable requirement for core inclusion?
Is TUF integration requirement the same for Package Manager, Project Browser, and Automatic Updates?

@hestenet gave an update on timelines today.

The code delivery To support root key management should be end of January - but we need to actually implement and test the root key ritual in Feb.

[A production deployment of TUF] could happen by end of Feb, but more likely, March.

This brings up two things for me:

We definitely can't rely on a production TUF until very end of February or into March, if we want to merge automatic updates as alpha before then (which I think we do), we need to make a proper decision here on alpha requirements. It would have been nice if a production deployment happened and this entire issue became moot, but we're not there yet.

I'm wondering given the DA has TUF on a dev endpoint, would it be possible (and if so, desirable) for us to point an alpha package manager to the dev endpoint? That would completely break the hard dependency between the production deployment and an alpha commit with TUF support - we'd be able to test package manager with TUF etc.

But, it might involve work both to implement it, and to back it out again when the production endpoint is available. But assuming that all happens in alpha we could skip any upgrade path for that bit I think.

I think this method would allow people reviewing the module to test full TUF protection against the dev endpoint in the Alpha stage but then have most testers not have to worry about the unreliability of the TUF dev endpoint(though they obviously would not get TUF protection).

In principle this seems like a very good workaround that would allow end-to-end testing with TUF, without it actually being deployed on d.o production yet, and more flexible than #39.

I'm wondering given the DA has TUF on a dev endpoint, would it be possible (and if so, desirable) for us to point an alpha package manager to the dev endpoint?

Right now this is hooked up to staging.devdrupal.org, which does not have an active packing pipeline. All updates are done by somewhat-manually pulling releases down from production and setting everything off. This currently only includes the 150 most-used projects. So probably not useful to use it at al beyond development.

Updating the proposal based on the latest discussions on the issue. I'm removing the "needs product manager review", but I'm adding the "needs framework manager" review.

We need a follow-up for #40. Including that in the proposed resolution too.

So the proposal in #40 is effectively using a module as a feature flag to enable TUF for testing purposes. I think this is a good idea; currently we seem to be a bit deadlocked because we only have the TUF dev endpoint and we want to test it, but to migrate confidently to the production endpoint we ideally need some testing to be completed by real world users. So if this can be enabled and disabled at will by testers, this seems like the best of both worlds.

PhpTufValidator is present in the current Package Manager module it just is not tagged with event_subscriber so its logic is not enforced.

In 10.2.0 we enabled autoconfiguration for event subscribers, so the tag is no longer necessary - so just double checking that the above statement is still true, as IIRC you are using other "new" features like autowiring in AU already: see https://www.drupal.org/node/3357408

I've tweaked the issue summary a bit to outline what's described in #40 a bit more, hopefully correctly. I'm happy to remove the RM review tag from this issue once we've got the follow-up open.

I think the proposal here will allow a sufficiently realistic set-up for people to test this prior to alpha commit (whether it's the test module or because we can do it some other way), so removing the 'needs framework manager review' tag. Will leave it to another release manager to remove the RM review tag so I'm not doing double duty.

My view from #45 hasn't changed, I don't see an easier way of moving this forward and starting to get test results from real users. Removing the release manager tag as @catch has also weighed in here and there have been no comments or objections from the other release managers.

I think this still needs the followup for beta discussions requested by @Lauriii in #36 and could do with an issue summary update/review before it can be marked RTBC.

Opened the follow-up: #3421925: [policy, no patch] Decide if a production level TUF is a requirement for beta-level commit of Package Manager.

I think we can mark this fixed, still need to figure out what actually happens with it depending on what's easiest/ready in time but the overall proposal that we enable people to test with TUF without it necessarily being on d.o production seems like a good position compared to where everything is.

Changing to latest version when this was closed.