Automatic Updates Initiative meeting on April 20th, 2021 [#3208054]

This meeting:➤ Is for core developers, initiative contributors, the Drupal Association and anyone interested in the initiative.➤ Usually happens every other Tuesday at 1700 UTC.➤ Is done over chat.➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!➤ Has a public agenda anyone can add to: #3208054: Automatic Updates Initiative meeting on April 26th, 2021➤ Transcript will be exported and posted to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

dts	Hi! David from Pantheon.
dstol	Dave from his patio
tedbow	Ted, at Acquia, from my home office in Ithaca, NY
pwolanin (he/him)	HI, Peter from BioRAFt
hestenet (he/him)	Tim, checking in between session monitoring for the Higher Ed summit.
rachel_norfolk	Rachel, UK, need to ask a question… :slightly_smiling_face:
afournier	Anthony, Drupal dev, looking to pick up a new issue. I’ll follow up in a thread.
Darren	Darren, occasional Drupal contributor. Looking to finish adding support for file transfer backends.
TravisCarden	:wave: Travis Carden from Acquia. I'm the current owner of the Composer Stager component of autoupdates. I'm just here to look pretty. :wink:
Matt Newby	Matt from Charles River Labs. Technical lead for our site, and quite interested in learning about how autoupdates can satisfy both the security needs for zero-day vulnerabilities as well as allowing sites to remain stable — unattended and untested updates on my site make me exceptionally nervous!
xjm	:wave: xjm, core release manager, joining late after a post-DrupalCon-exhaustion nap
eiriksm	eirik from norway, reading through to keep up to date the day after :slightly_smiling_face:

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion.

dstol	Seems like a bit of a date mix up in #3208054: Automatic Updates Initiative meeting on April 26th, 2021
dstol	But maybe it was something to accommodate DrupalCon
tedbow	well looks like we had our last slack meeting on the 6th. so I think today is correct
dstol	that node is saying the 26th
dstol	not a big deal, I was just slightly lost
hestenet (he/him)	Topic: Debrief on DrupalCon initiative day
tedbow	@dstol yeah but I saying it should be today. since our last meeting was 2 weeks ago
hestenet (he/him)	Topic: Upcoming key dates for the initiative --- (remaining 9.x releases 10.x date)
dstol	@tedbow I live and die by the calendar notifications
tedbow	that seems a bit extreme :joy:
dstol	it is, I just assume it’s usually the most definitive source of meeting correctness
pwolanin (he/him)	do we have a roadmap or plan for TUF signing on d.o infra?
tedbow	this maybe is question for @hestenet (he/him) or @drumm?
Darren	I'd like people to know we won't be asking them to allow Drupal to overwrite its own code once support for file transfer backends is done: #3159719: Prompt for temporary access when file system is not writable
drumm	No formal plan for the server side yet. We have a couple documents around, but haven’t had a chance to organize into an implementation plan, I could use some help on building it out.
drumm	Nevermind, I see the thread.
hestenet (he/him)	Topic: Interested in learning about how autoupdates can satisfy both the security needs for zero-day vulnerabilities as well as allowing sites to remain stable — unattended and untested updates on a site seem risky. per @Matt Newby
dts	@Darren Do you know how that will fit with Composer doing the writes? Is this possible to use as a Composer plugin?
Darren	It might require a different type of plugin. These are the methods of the FileTransfer class: https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21FileTrans...
dts	@Darren It might be possible to turn the Drupal work here into a library that Composed could also use.
Darren	@dts I would definitely be interested in that.

2️⃣ Debrief on DrupalCon initiative day

tedbow	we made some good progress on php-tuf. PR merged from @webchick @afournier and @pwolanin (he/him)@pwolanin (he/him) had a great find with a problem on how we are handling keys! :tada:
tedbow	also PR from @dts
tedbow	still have some “good first issue” issues https://github.com/php-tuf/php-tuf/issues?q=is%3Aissue+is%3Aopen+label%3... you working 1 and want it assigned to you let me know
pwolanin (he/him)	responded to Ted on https://github.com/php-tuf/php-tuf/pull/174 could use some other feedback and decision if we want to wait for added fixtures to add a test.
rachel_norfolk	Talking of debriefs…. I was thinking about inviting the initiative to record another video interview, like we did before DrupalCon. It was fun and informative for people — would you be interested??
tedbow	@rachel_norfolk good idea, I would be interested
rachel_norfolk	excellent. Let us all get that essential catch-up on sleep and we will do it!
xjm	My own debrief: I think our plan for the contrib day with a focused autoupdates session with the initiative team and interested experienced contributors paired with bugsmash worked really well
xjm	If we do this again I am going to insist we start preparing content earlier for all talks. Gábor also suggested that the words "technical overview" in the core convo title might have scared people off, so maybe in the future we could make it "Detailed overview and how you can help" instead (or something)

3️⃣ Upcoming key dates for the initiative --- (remaining 9.x releases 10.x date)

hestenet (he/him)	Alpha period for 9.2 opens the week of May 3rd! Only a few weeks away.
hestenet (he/him)	Alpha period for 9.3 will begin Oct 25th
hestenet (he/him)	And then D10 alpha is likely to start around May of 2022, if we are on track for a June launch.
xjm	This means it'd be super nice to wrap up the last bits of #3041885: Display relevant Security Advisories data for Drupal, get it RTBC, and get it committed so that we ship code in 9.2.x. (The readiness checks, stub module, update module improvements, etc. definitely will not be beta by then.) (edited)

4️⃣ do we have a roadmap or plan for TUF signing on d.o infra?

dts	I was originally going to work on a prototype on the hack days, but I ended up working on test fixtures for some issues. I feel like this remains among the less-defined parts of the plan. (edited)
hestenet (he/him)	We have been priority strapped, particularly in the run-up to DrupalCon so we need to regroup on the timing for the D.O infra side.
hestenet (he/him)	It would be helpful to know - when does this become a blocker for other components of the initiative?
pwolanin (he/him)	I think the plan may be more of a blocker than the deployment? We need to know if we need to e.g. handle delegated keys
hestenet (he/him)	Gotcha - we may need a little bit of a dedicated regroup on that. We do also have a notes doc some where with an initial sketch of some ideas (maybe you have that handy @dts?)
hestenet (he/him)	Neil and I are helping out with the Higher Ed summit today, so we might have to swing back around asynchronously.
dts	I'm joining from my phone today, so it's a little challenging for me to scour through docs to find it. I think it's pretty dated now, anyway.
dts	It wasn't very Composer/Packaging aware.
drumm	There has been some blocking already, @phenaproxima (he/him) is making assumptions about the server-side implementations already.
pwolanin (he/him)	I was pondering if one would want to create a “role” per signed package? Or maybe per source? like drupal/* packages vs. symfony/* ?
drumm	https://docs.google.com/document/d/1GdxB8m1yEIj0z3J_INpQZqMEOoMLiHlrJjVk... has some notes
dts	We definitely need to define how the client will hint to the server what it's looking for. I'd say that's the least defined integration point.
phenaproxima (he/him)	We definitely need to define how the client will hint to the server what it’s looking forNot quite sure I understand what you mean by this, @dts?
dts	@phenaproxima (he/him) A TUF repository cannot usually know what file a client seeks unless it both already has it signed and serves it. Since any lazy signing design would require the server knowing what the client wants to add it, we would need to augment TUF's design to add a hint (like via a query parameter) for what file the client is looking for.
phenaproxima (he/him)	@dts Well, the way the plugin works right now, it identifies TUF targets by vendor/package_name/normalized_version. Very deterministic
phenaproxima (he/him)	So you could maybe do something like ?target=drupal/token/1.9.0.0, which is an extremely clear way to identify a specific file to sign.
dts	Sure, but what would the TUF repo server look for to know that it's missing a package that the client wants?
dts	Oh, yes, exactly.
dts	That's all I mean: a hint like that.
phenaproxima (he/him)	Although, to be fair, even that might be necessary.
phenaproxima (he/him)	Why, you ask? Because when compiling info on available packages, ComposerRepository will make requests for things like /drupal/token.json, which contains everything you need to find a file to sign.
dts	Alternatively, we could provide an out of band (as in non-TUF repo request) way to request that the lazy repo add a package that the client couldn't find. (edited)
phenaproxima (he/him)	I think it’ll be hard for us to intercept a 404 like that
phenaproxima (he/him)	(In the plugin, I mean.)
dts	>Why, you ask? Because when compiling info on available packages, ComposerRepository will make requests for things like /drupal/token.json, which contains everything you need to find a file to sign.At least by the strict TUF spec, a repo shouldn't see a request for a target file unless the whole signed path is in place to that file.
phenaproxima (he/him)	I don’t think I understand…

5️⃣ FYI: we won’t be asking them to allow Drupal to overwrite its own code once support for file transfer backends is done: #3159719: Prompt for temporary access when file system is not writable

pwolanin (he/him)	I’m not sure how much this matters - I think the 80%+ use case is people on shared hosting where it needs to write the code from php
tedbow	the above issue is for the contrib module. So up to those maintainers what they want to support.For core this would be question for product or framework managers I think
pwolanin (he/him)	feels like a place where maybe core should have some way to override or replace the default mechanism in contrib?
pwolanin (he/him)	but… how will that work through composer?
tedbow	yeah the updater will extensible. if you someone wanted to write a Update plugin that solved it should be possible but yeah not sure how that work. the automatic_updates contrib module doesn’t use composer
tedbow	@Darren raised so wanted to make sure he saw this thread
Darren	Composer by definition overwrites Drupal code. I'm only worried about vulnerabilities in Drupal being used to overwrite its code.
tedbow	for drupal 9 core Drupal will be calling composer commands to do the automatic updating
Darren	Where is the code for that?
Darren	Allowing Drupal to handle unattended updates means an attacker who finds a vulnerability before it is patched could compromise the site and prevent it from receiving future updates. If that is the only way a site can receive unattended updates, the benefits might still outweigh the risks, but the risks need to be well documented.
Darren	File transfer backends are only for attended updates.
tedbow	@Darren that is code to be written but plan is here #2940731: Automatic Updates Initiative overview and roadmap
Darren	I will continue working on the contrib module code so it is available when the core code is written.
tedbow	@Darren it will definitely be helpful to look at. thanks
tedbow	I hope to get a proto up soon. this is rough issue #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow#comment-14026403but I don’t think this functionality to replace the existing updating will be added till later. will try to update the issue summary for this by tomorrow
tedbow	but it does update via composer
Darren	Thanks!
xjm	@Darren Also note that the TUF spec identifies a number of potential attacks like the one you describe, either preventing or at least detecting them so that (if we surface the errors properly) the site owner would become aware of the attack on the updater. And the staged composer update and package signing means that even if they manage to compromise the updater, an unsigned or bad update will also be detected and rolled back. Of course this doesn't help with the scenario where the exploit in question is an RCE and the exploit was leaked and the server gets owned before update, but for those situations, the goal is to automatically update the site before attackers can reverse-engineer the exploit (edited)
xjm	@tedbow This thread also makes me think that we should consider allowing a specific schedule to check for updates multiple times during the known, scheduled core security windows. One of our historical vulns was exploited within 7h, which is less than some sites' cron increment. We know when core itself or contrib will ship updates so the update module can be hardcoded for that even. (edited)
xjm	It still would need to check in between in case of an off-schedule release, but we can make sure Wednesdays get checked once at the end of the window or even twice during the window as a special case
xjm	If a site were patched by RCE with malicious code in the Drupal codebase, the signature and checksum of that codebase would not match and the readiness check would raise an error also, I think.
xjm	@tedbow The contrib module has a readiness check for the signature/checksum of projects, right?
xjm	@Darren In the TUF spec I believe it is specifically referred to as a "freeze attack" (when you try to trick the site into thinking it's already on the latest release and/or that no update is available)
xjm	@Darren Highly recommended reading if you have the time: https://theupdateframework.github.io/specification/latest/

6️⃣ Interested in learning about how autoupdates can satisfy both the security needs for zero-day vulnerabilities as well as allowing sites to remain stable — unattended and untested updates on a site seem risky. per @Matt Newby

hestenet (he/him)	This is definitely on top of folks' minds, for sure. I think perhaps @phenaproxima (he/him) and @TravisCarden between them can speak to some of this problem space.
phenaproxima (he/him)	I’m not sure I know to answer this
phenaproxima (he/him)	Personally I’d loop @xjm in on this one
xjm	@Matt Newby So one recommendation if you are concerned about stability is to use the next-oldest minor. (For example, 9.0.x right now instead of 9.1.x.) This means you will have to do minor updates every six months on our schedule, but it only receives security and critical data integrity commits every few months rather than bugfix releases once a month. Nothing else in between, so there are fewer patch updates to run and fewer chances of a regression in between. Many people use this strategy for core.Contrib is a different case, of course, because generally it doesn't have core's strict policies around allowed changes, nor the multiple branches to choose from.For these reason, site owners will be allowed to cosnfigure things like "only core should be automatically updated" or "only highly critical/zero-day issues should require an update". The readiness checks will help insure ahead of time that your site is in a state to receive these updates. (edited)
xjm	Like 9.0.x has had maybe 5 commits in the last four months, versus hundreds in 9.1.x. So statistically 9.0.x has a far lower chance of any regressions
xjm	@Matt Newby The staged updater also will theoretically be able to be wired into a site- or host-specific workflow, so that you could automate running a site QA suite before deploying the update in your own normal process.
Warped	Wondering if a contrib module is using a dev version and a stable version is released, does that add consideration for updating when it might not otherwise be updated. Are only tagged releases valid? Would that mean a module that gets commits, but hasn't had a release in a long time won't get updated at all? Even if the site is using the dev version.
xjm	@Warped Running a dev version will by default raise a readiness check error
xjm	@Warped A good contrib author will issue a stable release prior to having a security fix go out. This is also an area where we have an opportunity to educate contrib maintainers about good release management practices.
xjm	@Warped In reality though enough sites run dev versions that there may end up being configuration allowing site owners to update from dev releases. The risk is that we don't know at all how recent the dev release is, so it could be HEAD or it could be from 5 years ago.
Warped	Agreed. Maybe can make it more public for people to try to contact the maintainer if a module is stagnant, and maybe there is the opportunity to co-maintain or take over a project.
Warped	Making a needs-some-love list of modules that have many open/rtbc issues, old commits w/o a release, etc, might bring to view modules people hadn't noticed were lagging.
Warped	And being at the top of that list might motivate some maintainers to move it along.:stuck_out_tongue_winking_eye:
xjm	Honestly contrib maintainers could even use the core practice of tagging a security release off the last tag instead of HEAD and merging the changes, so that the security release is less likely to disrupt anything. There's nothing special about that process; it's just git. In that case, the dev tarball will also get rebuilt with the fix upon release, so the person running HEAD could choose to update to the latest dev tarball instead. It would not be the default behavior though because we strongly encourage tagged releases.
xjm	@tedbow @hestenet (he/him) All the points in this thread are probably worth capturing on our roadmap somewhere so that we don't forget about them when developing the core updater

The hour is up, but the meeting will continue asynchronously - feel free to add your own additional topic threads at this point if you’re coming in late. :thankful:

tedbow

thanks for running the meeting @dstol!

Participants:

dts, dstol, tedbow, pwolanin (he/him), hestenet, rachel_norfolk, afournier, Darren, TravisCarden, Matt Newby, xjm, eiriksm, drumm, phenaproxima (he/him), Warped