[Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged

Reviewing the known issues in Rugged:

• #120 Add a new "monitor" worker is fixed!
• #114 Revert to upstream Composer plugin is still open
• #99 Implement support for hashed bins will be necessary for the scale of Drupal.org packages; so data transfer is limited to a reasonable amount, and that JSON can be reliably parsed
• Currently containers are built locally and pushed, https://rugged.works/how-to/images/. This would be better if it could be driven by GitLab CI automatically. Likely needs an issue opened in the rugged project

Adding to the issue summary:

Plan for signing core components

• Templates like drupal/core-recommended are hosted on Packagist.org, so they can be installed with one step after getting Composer. Core component subtree splits are hosted in the same way to simplify core packaging
• https://packagist.org/packages.json changes frequently, so signing on behalf of Packagist.org will need some way to either always have the current version signed, or proxy that's able to serve what we have signed

Do we have any idea about an ETA?

I'm asking because in #3349368: [policy, no patch] How much of The Update Framework integration is needed for alpha-level review/commit of Package Manager? it was just revealed that this is hard-blocking package_manager, automatic_updates and project_browser getting added to Drupal core.

Adding link to https://gitlab.com/rugged/rugged/-/issues/74

I've revised the summary with updates for the Rugged tickets.

Looks like hashed bins is also done https://gitlab.com/rugged/rugged/-/issues/99

This is now ready: https://packages.drupal.org/8/metadata/

Before calling it done, we need:

• #3352216: Securely sign Drupal core packages, even though they are hosted on GitHub/packagist directly
• securesystemslib includes non-compliant `keyid_hash_algorithms` property when generating key IDs https://gitlab.com/rugged/rugged/-/issues/192
• Reset processing targets batch on boothttps://gitlab.com/rugged/rugged/-/issues/191
• Nice to have, not required - Clean up more completely when targets containing empty directories are processed https://gitlab.com/rugged/rugged/-/issues/149
• Verify root key rotation process

Added 2 more child issues:

• #3460020: Add host key verification for sending targets to rugged for signing this is a security hardening. We are not sending anything private to be signed by rugged, but we should still verify where we are sending it
• #3201223: Deprecate composer 1 will reduce the error rate for the client, especially if our rugged stack has an outage

Does this mean that package signing is practically all rolled out?

This is now ready: https://packages.drupal.org/8/metadata/

Yes, I think we can call this done.

I hope there’s more testing with #3477553: [PP-1] Manually test TUF-enabled Composer projects before this is made generally available out of the box with Drupal core. But that and the rugged followups are all being tracked in their own issues.