Auto Updates Meeting 2018-09-06 5:00PM UTC

Agenda

• Reviewing progress out of MWDS.
  • We drafted a roadmap: https://docs.google.com/document/d/1D848kBxuBB5XFLLfSrNinKbVGxS6weqmVndx...
  • Were there action items? If so what's their status?
  • One action item was "meet with the Composer initiative to compare our roadmap/needs to theirs" (who?)
• Discussing any plans for Drupal Europe
• Communications/Expectations management around the aims of the initiative

Minutes

Reviewing progress out of MWDS:

- Roadmap drafted here: https://docs.google.com/document/d/1D848kBxuBB5XFLLfSrNinKbVGxS6weqmVndx...
- What are our next action items?

mixologic [42 minutes ago]
Can we enable commenting on that roadmap?

xjm [42 minutes ago]
Yep will do, sorry

xjm [42 minutes ago]
Fixed -- reload to comment :slightly_smiling_face:

xjm [42 minutes ago]
Would love review/feedback on the purple roadmap before adding it to the public roadmap issue

hestenet [41 minutes ago]
It sound like we have two separate threads of development:

A. Updates Manager improvements to be backported to D7

A-Prime: Allowing core to be updated by Update Manager

B. The Automatic Core Updates for Composer based projects (edited)

xjm [41 minutes ago]
A. is really just a couple patches

hestenet [41 minutes ago]
@mbaynton Do I have it right that you’re taking the lead on A?

Whereas @David Strauss and @pwolanin are leading on B?

David Strauss [40 minutes ago]
Yes, I'd like help defining where I can be most productive as a contributor.

David Strauss [40 minutes ago]
And B is the area I have the most interest in.

xjm [40 minutes ago]
Ah, I think we are meaning different things.
> Add support for manually updating core in Update Manager. Allow core to be updated via the administrative UI In the same way that contributed projects can be updated through the interface. May include some Drupal.org infrastructure enhancements to make updates efficient.
is the thing @mbaynton is working on (edited)

hestenet [40 minutes ago]
Ah. gotcha. We’ll call that ‘A-Prime’ :wink: (edited)

mbaynton [39 minutes ago]
@hestenet I think there's more than two separate threads of development, if I'm navigating the document correctly. Theres a whole other thing happening starting on page 2, this is where I'm currently working.

hestenet [37 minutes ago]
Okay - I’m adding a section for “Who” under each item.

So we can self-select who is interested in carrying what.

xjm [35 minutes ago]
@hestenet I shared the doc with you so you can edit it

hestenet [34 minutes ago]
Thanks

hestenet [31 minutes ago]
Okay @here I’ve put some initial names by each section based on what I remember people saying from MWDS.

hestenet [31 minutes ago]
What seems to be missing primarily is folks to claim the first block of Update Manager work.

hestenet [29 minutes ago]
@David Strauss Since your interest is primarily in the Automatic Updates for composer based stuff… I assume that’s where you want to focus.

Do you still feel like working on a prototype example for any Composer/Symfony-ish project? Or is thinking through signature/verification stuff something you’d want to think about first?

David Strauss [28 minutes ago]
I think I also agreed to help with signature verification work for part A.

hestenet [28 minutes ago]
Okay. That sounds good.

David Strauss [27 minutes ago]
To answer your question more directly, of the Part B stuff, I'd prefer to focus on prototypes around packaging Comp/Symf projects over signing.

hestenet [27 minutes ago]
’k ’k

David Strauss [27 minutes ago]
I love work on signing, but I'm familiar enough with it that I'm not worried about accidentally creating a prototype that wouldn't support it being integrated later.

hestenet [25 minutes ago]
Do we have someone familiar enough with Update Manager to work on our precursor issues there? Or do we need to recruit someone?

xjm [17 minutes ago]
Ted Bowman has learned a lot about update manager in the past couple months, but it depends on whether we'd be able to get him allocated to it.

xjm [16 minutes ago]
drumm would be a good reviewer for any update manager changes with another lead dev working on it

Who will be at Drupal Europe, and does someone want to organize a BoF?

hestenet [25 minutes ago]
Myself, mixologic, and drumm will be there.

xjm [24 minutes ago]
I will be there. I think a BoF is a good idea. I wouldn't be able to lead it though. :slightly_smiling_face:

xjm [23 minutes ago]
We could ask Gábor to add it to the "Hard Problems" list for help scheduling so it doesn't overlap other BoFs/meetings

hestenet [23 minutes ago]
I like that idea. Though if it’s just the four of us we may not have the knowledge we need in the room.

(I suppose we can cancel it though, if that happens)

xjm [21 minutes ago]
If we publish it in the BoF list we might also get a flood of interested people. It might turn into "describe the roadmap" to folks.

hestenet [20 minutes ago]
True

So I've got working code at this point that does new and useful things around updating core from a web browser

, but it uses something that is not a full release .tgz archive to do it. This has infrastructure implications; basically an additional file would need to be generated and hosted as part of new releases. So that's something I'm looking to talk to DA people about...

hestenet [24 minutes ago]
I remember overhearing something about this. @mixologic or @drumm are probably the ones to talk to, yes.

mbaynton [23 minutes ago]
If we ever hope to do auto updates of security patches I think we'd really want to use the same things, as well.

mbaynton [23 minutes ago]
...In Drupal 7, I should clarify. The plans for 8 are a differentn world.

drumm [22 minutes ago]
What sort of file?

hestenet [17 minutes ago]
@mbaynton It might be good to schedule a time to demo your code for Neil and Ryan sometime after Drupal Europe.

mixologic [10:22 AM]
`but it uses something that is not a full release .tgz archive to do it.` ?
is it metadata?

mbaynton [10:24 AM]
@mixologic you run a script that compares the release n-1 tree to the current release, the script puts files that have changed plus some metadata like files that were deleted into a .zip; that's the thing that my code knows how to consume.

mixologic [10:24 AM]
so a delta?

mbaynton [10:24 AM]
Yeah
This is imo way better when you're limited to updating in-browser because of the massively smaller amount of work that needs to be done and fewer files that need to be jimmied around, and fewer moving parts to break

xjm [10:25 AM]
That gets tricky with D8 -- releases aren't always in numerical order
like we have 8.5.7 released after 8.6.0, or 8.4.7 and 8.5.3 or whatever released on the same day and with the same patch (edited)

drumm [10:27 AM]
I’d guess 8.6.0 might not have a delta; and 8.5.7 would only have the delta from 8.5.6

David Strauss [10:30 AM]
+1 on shipping deltas for Part A. It will vastly reduce overhead on the client and/or the disruptiveness of applying the updates.

mbaynton [10:30 AM]
There's a huge number of possible ways to handle those issues, in that you can generate and host all the update packages you want to take you from any interesting cobminations of version a to version b, and you can also actually put deltas for several versions into the .zip. I think that's more of a "what do we want to do" question than a technical hurdle.
honestly my concern at the moment is whether there's support for doing it at all, just in that my continued work is dependent on buy-in to publish some of these things in some form

drumm [10:35 AM]
Are there any common standards for that sort of file format, so we are more likely to avoid pitfalls in unexpected edge cases?

mixologic [10:38 AM]
patch files are the standard right?

mbaynton [10:38 AM]
This is an obvious question I should have investigated more thoroughly 2 years ago when I developed that part, but in terms of formats that there was already-proven php implementations for I didn't find any.
applying patches in particular actually has abysmal native php support last I looked, anyway

mixologic [10:39 AM]
But I guess we'd still need some internal metadata that describes "this should be applied to version X to get version Y".

mbaynton [10:40 AM]
Obviously I can't promise I've thought of every possible edge case, but I can say that the format and accompanying code that consumes it has test coverage for lots of crazy shenanigans like `my/file/here` was a whole directory tree in release n-1 and now it's a file

mixologic [10:41 AM]
all of this sounds like things we (the DA) need to file under API changes for "updates 2.0".
@mbaynton is your code expecting to be able to download a particular delta file, or can it hit an api, present the pertinent data, and get back the proper delta file?

mbaynton [8 minutes ago]
hmm so I'm not immediately finding a url that lets me reference a line in git.drupal.org, but the sandbox project is here: https://www.drupal.org/sandbox/mbaynton/2997009. The module itself is like probably < 100 lines and includes one where it passes a url to the delta file into the main update engine. So yeah you could easily change that to get the url from wherever.
Drupal.org
Web Install Kit
This experimental module integrates the Curator Web Install Kit with Drupal to enable in-browser updates of Drupal core. Demo video: https://youtu.be/r3H-KzfV5sk It's published here so you can try it yourself -- please try it out & evaluate it by installing it like a normal module. See core issue 2995202 for discussion and a high-level overview of how it works. Limitations:
Sep 3rd at 9:46 AM

mbaynton [6 minutes ago]
If these get DA-hosted, that's the part that would change.

drumm [6 minutes ago]
We might want a packaged artifact that can be checksummed/etc. Also I’m guessing don’t want to generate those on the fly, even if cached.

mbaynton [5 minutes ago]
yep sounds right to me

mixologic [4 minutes ago]
Yes, and Im thinking that rather than the site that needs an update picking the right file location based on what it thinks it needs, it passes its metadata to drupal.org, and d.o. is responsible for returning the proper artifact.

mixologic [4 minutes ago]
That artifact can totally be generated in advance.

mbaynton [1 minute ago]
one dumb way to accomplish something like that is what I did for the example, I just made a bunch of them for "I currently am version x, take me to latest" (which I trust core to get right) https://www.getcurator.org/cpkg/drupal7/

mbaynton [< 1 minute ago]
ps, deflate is disabled on those so they're way bigger than they need to be. It's a demo.

Communications/Expectations management around the aims of the initiative.

A lot of people are interested in this initiative - how do we communicate the roadmap publicly and set expectations for what this will accomplish in its various phases?

mixologic [10:52 AM]
^ that was me wondering about what, exactly, is in scope and not.

When the public hears "automatic updates" it can encompass so much, and perhaps even be things we've descoped.
i.e. D7 vs D8, critical security, vs security, vs feature updates, Core only, or Core + contrib etc.

xjm [2 minutes ago]
We should consider submitting a core conversation for Seattle. And at some point I think Dries will probably want to do a blog post referencing our roadmap and summarizing it at a high-level.

xjm [1 minute ago]
We also have a page on the strategic initiative section of D.o but it is mostly a stub. We should use that space. :-) (edited)

xjm [< 1 minute ago]
We can also link our meeting minutes queue there too, so people can check out what has been discussed recently even if they miss a meeting