| hestenet (he/him) |
cc @phenaproxima and @tedbow |
| tedbow |
Per Drupal 10 Core Roadmap for Automatic UpdatesWe currently have 2 alpha blockers though will have to move 1 more from Beta to Alpha after getting clarification the The Update Framework requirement |
| drumm |
I might have missed it in the threads, how’s support for hashed bins in the client? |
| tedbow |
@drumm we don’t support it. because we weren’t aware till recently drupal.org would be using it. @phenaproxima is going to implement it |
| drumm |
Cool, and I’d like to see either #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged as an explicit dependency in the core issue |
| drumm |
** #3343490: Deploy rugged for TUF signing to production |
| dww |
Seems like #3331078: Add php-tuf/composer-stager to core dependencies — for experimental Automatic Updates & Project Browser modules is the next major blocker to all core module issues, and seems pretty thorny from the outside. :sweat_smile: |
| tedbow |
@dww which part is thorny? |
| dww |
The alpha-ness. |
| dww |
:chicken: :egg: : Core doesn't want alpha dependencies. PHP-Tuf wants to be alpha until Core is happy using it. |
| tedbow |
@dww oh yeah that. seems like maybe if we get some reviews on it in the Alpa version(before the core commit) then we could make a non-alpha version. Just would suck to make a 2.0.0 release and then get a review that asked for BC breaking change |
| phenaproxima |
If that happened, we could either 1) make changes in a BC way, or 2) just do a 3.0.0 version. |
| phenaproxima |
Version numbers are cheap. |
| tedbow |
thats true |
| xjm |
An alpha-version dependency is totally acceptable in the dev branch of HEAD. Our minimum stability prefers stable, but it's possible to override that, and we do this every six months to adopt Symfony betas during the minor development process. An AU alpha that depends on a (signed) Package Manager alpha that depends on a composer-stager alpha is 100% cool.The only tricky part is we might want to remove said alpha dependency before tagging core alphas, like we do with experimental modules. Just an extra step for RMs so that production core doesn't have an unused dep. |
| hestenet (he/him) |
@drumm will provide updates here |
| hestenet (he/him) |
Parent issue: #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged |
| drumm |
I’m working on making #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged accurate for what we know about what’s left |
| drumm |
The basic integration seems to be working well, the client is working with what the proof-of-concept server is doing #3341719: Add a drush command to send targets to rugged |
| tedbow |
What version of the TUF protocol does rugged support? I get a 404 on this docs link here https://gitlab.com/rugged/rugged |
| hestenet (he/him) |
Ah - try this link instead - the short description is updated, but the readme is not, it looksl ike: https://rugged.works/ |
| hestenet (he/him) |
And the work was done to support the 'latest' spec which is still 1.x: https://theupdateframework.github.io/specification/latest/ |
| tedbow |
couple questions.is the goal of Rugged to support the latest version of the spec 1.0.31 by launch? I assume we want the server and client to synced on thisIs the Spec agnostic about TAP 15 and other TAP’s? Since the Spec doesn’t seem to have any release notes its hard to tel |
| tedbow |
https://github.com/theupdateframework/specification/releases |
| hestenet (he/him) |
The most 'change record' they seem to provide is a diff, yeah: https://github.com/theupdateframework/specification/compare/master..draft |
| hestenet (he/him) |
@ergonlogic would need to speak to which specific release we were on when we completed the bulk of the work - so that we can sync up to the client on that yes. I don't think it's the Sept release 1.0.31I will go ask in the TUF channel about how TAPs are supposed to work together with the spec. |
| tedbow |
so from https://github.com/theupdateframework/taps/blob/master/tap15.mdThis TAP is not backwards compatible, and will need to be included in a major version release of the specification.This makes it seems like it cannot be in 1.x of the spec |
| drumm |
I also updated #3343490: Deploy rugged for TUF signing to production with some details for launching, so I don’t forget what I set up. There are still some “draw the rest of the owl” steps like “Set up Rugged in production” |
| hestenet (he/him) |
@tedbow - gotcha - in that case it seems most likely we want to implement 'classic' hashed bins? (cc @ergonlogic ) |
| hestenet (he/him) |
Still going to post in TUF channel to clarify tho |
| tedbow |
yep. thanks |
| ergonlogic |
yep, "classic" it is |
| ergonlogic |
I just pushed a fix for the docs link in the README, btw (edited) |
| ergonlogic |
We're tracking the spec version supported in python-tuf, which is specified in tuf/api/metadata.py (https://github.com/theupdateframework/python-tuf/blob/v1.1.0/tuf/api/met...). Currently 1.0.29 |
| xjm |
I don't know if this is the right thread or if it should be in the hashed bin support thread. but I saw someone quoting someone else saying something about having storage troubles for 80,000 projects. We should only be signing things that have D10+ tagged releases. /cc @catch @hestenet (he/him) @ergonlogic Not dev tarballs and not anything not compatible with D10, because TUF support will never be backported from D10. |
| xjm |
Oh and @drumm ^ |
| tedbow |
@xjm I would want to confirm that would work with @phenaproxima who wrote the Composer integration plugin. Right now it is written so you need to declare that https://packages.drupal.org/8 is a TUF enabled endpoint and it expects all packages from that endpoint to be TUF enabled. We would at the very least have to have some way to customize that to allow packages to opt out based on the drupal/core requirement that it has.I think the problem would be if I have a Drupal 10 site and run composer require d7_only_module because beforehand I don’t know that it is not compatible with d10(assuming the package is not so declarative in its name:wink:). Would the TUF Composer Integration Plugin fail before it even had enough metadata to know the drupal/core requirement? If so the error message would probably be something like “TUF signing error” which doesn’t tell the user what the real problem is. |
| phenaproxima |
@xjm When you say “TUF support will never be backported”…for the record, the Composer plugin and PHP-TUF have absolutely zero opinion about Drupal. |
| tedbow |
@phenaproxima The point I was trying to make above is that even if we could add some customization somewhere I am not sure with how Composer downloads work if we would even have the requirements early enough to opt out specific packages |
| drumm |
Right, we’re signing all of https://packages.drupal.org/8 for all the reasons above. One small clarification, replace composer require d7_only_module with D8-only in the example - /8 is things compatible with 8 & later. |
| drumm |
The math for that is 100s of MB already, which is a lot to download and parse for any Composer operation. Even if we cut corners, the size only goes up. |
| phenaproxima |
This, I think, is why we need hash bins. |
| drumm |
And whatever there is or isn’t client support for at launch effectively constrains what we can add in the future. No bin support on the client means we couldn’t add them later on the server. If the client implements bin support without the server, even though following the spec has served us very well so far, it could be broken without being caught. |
| phenaproxima |
I mean, we’re gonna do what the spec says |
| drumm |
Yeah, its gone great so far |
| xjm |
@tedbow @phenaproxima So the concern is if that some 8.x-only project is not signed and they try to composer require it, they might get that it failed due to a lack of signature rather than the "right" error message of it failed due to the constraint conflict?Does the composer operation that kicks off stager have the opportunity to validate that first?Just trying to look for a way around storing terabytes of unnecessary data that will never be used. (edited) |
| phenaproxima |
Why would we be storing terabytes? |
| drumm |
This issue is the largest number I’ve seen, ~half a terabyte #3349408: Decide on & implement targets management for Rugged instance. That’s purely server-side, and is looking straightforward to work around. |
| phenaproxima |
@drumm Yeah, I definitely do not think we should be storing the actual targets. Once Rugged signs them, they can be thrown out |
| phenaproxima |
I think you only need them again to re-sign (edited) |
| drumm |
I’ll probably do that with find targets older than 1h | xargs rm I just want to take a little time to read through the rugged code and confirm the other options aren’t worthwhile, and get myself more familiar with rugged internals |
| phenaproxima |
Makes sense. I personally know nothing about Rugged’s internals |
| drumm |
I should know more. And the engineering to do partial signing of https://packages.drupal.org/8 to save maybe half the computer resources, for now, isn’t really worth it if it was practical. |
| xjm |
Thanks all; third parties were making it sound as though the storage and resourcing was a huge problem, and quoting numbers of projects that could not possibly have release compatible with core TUF integration. (@phenaproxima That's what I mean won't be backported.) So I just wanted to make sure it was clear that there's no sense signing stuff without releases or stuff that won't ever be D10-compatible with TUF. The D8+ makes a sensible cutoff if that's what's being used, although I'm surprised that would still be 80,000 things. |
| catch |
From another thread somewhere, it's 80,000 releases, which if there's 4,000 projects that's an average of 20 releases per project |
| drumm |
Just double checking - yep that’s accurate. 82,757 releases are compatible with Drupal 8 or later |
| ergonlogic |
I can have Rugged remove those files after signing. I didn't because I figured we'd be implementing consistent snapshots sooner or later, at which point we'll need those files, just renamed .FILENAME.EXT. |
| drumm |
That’d be ideal. And oh man that would be many terabytes of storage if we had consistent snapshots. |
| ergonlogic |
right, so let's bear that in mind before seeking to implement it :wink: |
| hestenet (he/him) |
@ergonlogic and I are getting a supplementary agreement in place to get this support implemented as quickly as we can. |
| hestenet (he/him) |
https://gitlab.com/rugged/rugged/-/issues/99 |
| hestenet (he/him) |
Re: client side of Hashed Bin support @phenaproxima - we're looking at implementing the server side according to the new 'succinct hashed bins' suggestion from: https://gitlab.com/rugged/rugged/-/issues/99#note_1092688071 |
| hestenet (he/him) |
However, we could implement 'classic' hashed beens from the spec instead, if you have a strong preference for what you implement in the client. |
| phenaproxima |
@hestenet (he/him) I think the final decision is @tedbow’s. However, if the succinct thing is just a suggestion still, and not part of the spec, it might make sense for us to implement the thing that is part of the spec right now. |
| hestenet (he/him) |
It is linked as 'accepted' here: https://github.com/theupdateframework/taps#accepted |
| phenaproxima |
Oh, fair. If it’s gonna be added to the spec, that’s fine. Still, final call is Ted’s |
| hestenet (he/him) |
'k 'k |
| tedbow |
It a while since I have looked at this. Isn’t there 1.x and 2.x version of the TUF spec? Does TAP 15 get implemented in both?what version is Rugged currently implementing? |
| tedbow |
ok maybe was confused about this I thought there was a 2.x version of the spec but I don’t seen any 2.x tags here https://github.com/theupdateframework/specification/blob/v1.0.31/tuf-spe... (edited) |
| hestenet (he/him) |
Yeah, so far as I know it's just the 1.x |
| hestenet (he/him) |
Noting from the other threads that it looks like we need to implement 'classic' hashed bins:https://drupal.slack.com/archives/C7QJNEY3E/p1679420579129859?thread_ts=... |
| phenaproxima |
@hestenet (he/him) There is another way to do this, without using path hash prefixing. |
| phenaproxima |
(In fact, without needing to implement anything different.) |
| phenaproxima |
Right now, from what I can see, Rugged is signing every target with the same role (targets). If you start delegating, however, then you might be able to accomplish the same goal of not having everything signed by one role, and listed in one file. |
| phenaproxima |
cc @drumm |
| drumm |
I don’t know that rugged has any form of delegation built in. Regardless, I think I want something self-contained in rugged, like hashed bins as an option, or configuration to delegate according to patterns. Right now the communication of incoming targets is just the targets themselves on a mount, and the 2 main directories will have thousands of files each. There isn’t a spot for additional metadata like “delegate like this” |
| phenaproxima |
@hestenet (he/him) Ted and I discussed this. I think, if I focus on it, I can get “classic” hash bin support into PHP-TUF in two weeks. |
| phenaproxima |
The succinct stuff is a) not in the spec yet, and b) will, from what we hear, necessitate a new major spec version. We can graft it onto PHP-TUF later if we need to. |
| hestenet (he/him) |
Alright - so on our side, I'll make sure @ergonlogic and I are working to do 'classic' support on the server side as well. |
| ergonlogic |
I'm pretty sure that TAP 15 is planned for v1.2, not v2.x |
| ergonlogic |
It's in python-tuf v2.0 |
| hestenet (he/him) |
From this thread with Marina it sounds like it may be waiting.. https://cloud-native.slack.com/archives/C8NMD3QJ3/p1679420839874959 |
| ergonlogic |
okay, interesting. "Accepted" seems like a poor naming convention for what appears to amount to "under consideration" |
| ergonlogic |
presumably, python-tuf v2 will also allow for "classic" hashed bin delegation |
| ergonlogic |
the backward compatibility breakage there has mostly to do with method signature and naming changes, afaict |
| phenaproxima |
I think it still makes sense to implement the classic version for now, and graft on succinct support later as a performance improvement |
| drumm |
Would we then have to pick one or the other on the server at any given time, unless we do API versioning and maintain 2 TUF repos? Or can the two coexist? Either way, we’d have to maintain classic for a long time until everyone updates |
| phenaproxima |
Well, I am uncertain honestly |
| phenaproxima |
I haven’t seen any indication that path_hash_prefixes is deprecated or anything… |
| drumm |
We could use some telemetry on the TUF version supported and/or version of the client, so we know when we can discontinue or activate support for things. Maybe a user agent string addition. Something will change in the long run |
| hestenet (he/him) |
For reference, thread with TUF folks about the TAP 15 thinghttps://drupal.slack.com/files/T06GX3JTS/F04V12H3NS2 |
| catch |
A bit meta, but is this blocking all php-tuf/package_manager development, or is it more of a 'stable blocker' while other stuff can be happening? |
| tedbow |
on the package_manager side this does not block us doing other work in the Drupal module itself. But we would need to update our https://github.com/php-tuf/php-tuf client. We can add the hash-bin support as soon as we get agreement on which method we will use. We could support both methods but seems like a waste of effort as right now we don’t anybody else who is asking for this and we have limited resources. We would still need to test the integration of the client and server when both are done. |
| hestenet (he/him) |
^^ And as far as I'm concerned, I believe we're moving forward with the 'classic' method. per our last discussions, I just wanted to provide that conversation as context. |
| drumm |
@phenaproxima Composer’s user agent is like Composer/2.2.21 (Linux; 5.4.0-122-generic; PHP 8.0.19; streams; CI) can the PHP-TUF version be added to that? |
| phenaproxima |
@drumm I don’t know. I could research it. |
| phenaproxima |
@drumm If you could open an issue about that in https://github.com/php-tuf/composer-integration/, so I won’t forget |
| drumm |
WIll do, was about to do that and wanted to make sure it wasn’t already done |
| hestenet (he/him) |
This meeting:➤ Is for core developers, initiative contributors, the Drupal Association and anyone interested in the initiative.➤ Usually happens every other Tuesday at 1700 UTC.➤ Is done over chat.➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!➤ Has a public agenda anyone can add to➤ *Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a 👤 emoji. To take a comment or thread off the record, start with a 🚫 emoji. |
Comments
Comment #11
hestenet