Drupal 10 roadmap

#3319030: Drupal 10 Core Roadmap for Automatic Updates

Initiative Overview

The goal of the Automatic Updates Initiative is to provide safe, secure automatic updates for Drupal sites. This is an essential strategic initiative for Drupal 9 core for two reasons:

  1. Manually updating sites is difficult, time-consuming, and expensive, and the difficulty of updating Drupal is repeatedly raised as a top concern for using Drupal.

  2. Delays between the release of a security advisory and the deployment of the corresponding Drupal update result in a time window where sites are vulnerable to exploit. Many sites do not apply updates for 2-4 weeks after the release of a security advisory, and site owners who want to ensure the security of their site need to have site maintainers available during the 1200 EST security release window (which is in the middle of the night for much of the Eastern hemisphere).

Both these factors increase Drupal's cost of ownership and decrease user confidence in the platform.

How to get involved

Join the #autoupdates channel of the Drupal community Slack.

Meetings are held on the every other Tuesday at 1:00 PM EST in the #autoupdates channel. Minutes from past meetings and agendas for upcoming meetings can be found in the Automatic Updates project queue.

Work is currently going on in 4 projects:

  1. The 8.x-2.x branch of the Automatic Updates Contrib project. This work will be port to Drupal core directly. Issues to be worked on can be found in the issue queue for 8.x-2.x
  2. The Github PHP-TUF library: this will ensure updates are correctly signed
  3. A Composer plugin to integrate PHP-TUF signing with Composer. This depends on drupal.org's packaging pipeline and Composer facade supporting PHP-TUF signing, which is being built under the supervision of the DA.
  4. The Github Composer Stager library: this library will enable Drupal core to stage updates before they are applied

Session recordings

Automatic Updates architectural spec/overview

Background: Personas

  • Deploy and Ignore: Once the site has the functionality needed, there's little maintenance or updating. Doesn't subscribe to PSAs.
  • Site owner who cares once a year: Deploy and ignore w/ an exception once a year
  • Diligent but with Simple Needs: Typically applies updates within a week of release, possibly longer for non-security updates. Follows up on PSAs by directly updating the live site.
  • The Sophisticate: Needs to apply at least one build step (for CSS, Composer, etc.). Runs QA in a pre-production environment. May deploy to a multi-head cluster.
  1. Validation, readiness checks and notifications

    The Automatic Updates module verifies that it is safe to apply an update to a site prior to downloading an update, and informs the site owner if updates cannot be installed automatically.

    The readiness check API is being developed in the 8.x-2.x branch of the contrib project. See #3230045: [Meta] Create validation listeners

    The readiness checks include (but are not limited to) the following:

    • Is the site on the latest release of core?
    • Does the site codebase match the official release? (I.e., no patches applied, no hacks to core, etc.
    • Does the site have outstanding database updates?
    • Is there available disk space?
    • Are file permissions configured appropriately?
    • [Still needed / not yet added] Is the memory available sufficient for updates to be installed? Issue: #3159920: Add a PHP Memory Readiness checker

    In addition to the readiness checks, we're also adding display of critical security Public Service Announcements on the Drupal site status report and administration pages: #3041885: Display relevant Security Advisories data for Drupal.

  2. Signing and verification

    Signing of Drupal releases will be handled according to The Update Framework (TUF) specification. drupal.org will provide signed versions for Drupal releases and vendor libraries that will be needed for updates.
    Core will use the PHP-TUF library which implements a client library. This library is currently being developed may be used by other PHP CMSes.

    The PHP-TUF client library will require Composer integration to verify the packages before they are installed via Composer. This is done by way of a plugin for Composer 2.1 or later: https://github.com/php-tuf/composer-integration. The plugin depends on TUF signing information being available for all requested packages, which is being added to drupal.org's packaging pipeline and Composer facade under the supervision of the Drupal Association.

  3. Staged update and rollback

    The update functionality in Drupal core will need attempt to run an update via Composer and be able to determine if the update succeeded. If the update was not successful the site should be left unaffected and in a working state. If the update was successful, the production site should switch to the new codebase with 0 or minimal maintenance mode window needed.

    An earlier proposal for this was #3093093: Replace single index.php with a frontend controller that supports A/B updating (automatic updates).

    However, we encountered some issues with that, and our current proposal is #3199171: Automatic updates: For MVP, instead of an a/b bootloader, implement a semi-atomic composer update with a brief maintenance-mode window.

  4. Update installation

    • Composer-managed sites

      Composer-managed sites are supported in the 8.x-2.x version of Automatic Updates contributed module. This 8.x-2.x version of the module does not support sites that are not managed by Composer.

      The 8.x-2.x version of the this module is being prepared for inclusion in Drupal 10 core.

      Per #2 and #3 above, https://github.com/php-tuf/composer-integration will provide the TUF integration and https://github.com/php-tuf/composer-stager will provide the staging and rollback support.

    • tarball-managed sites

      In-place automatic updates for tarball-based sites for Drupal 7 are provided in the 7.x-1.x version Automatic Updates contributed module (development funded by the European Commission).

      The 8.x-1.x version of this module which supported tarball-based sites for Drupal 8 is no longer supported.

      [More detail here]

Roadmap

[Complete!] Phase 1

Phase 1 included developing a Drupal 7 and 8 contributed module with readiness checks, security public service announcements, and in-place manual and automated updates for tarball-based sites.

[In progress] Phase 2 (Initial core requirements)

Phase 2 of the initiative is to add the initial requirements for automatic updates to Drupal 9 core. The initial core module will provide automatic updates for:

  • Core only
  • Composer-built sites only
  • Patch/security releases only
  • Updates for the installed minor version

Phase 2 requirements:

[Outline and issue list here, based on most recent meeting]

  1. Completed 🎉: Port PSA functionality to core system.module

    #3041885: Display relevant Security Advisories data for Drupal

  2. Port the 2.x branch of the contrib module to core

    This will add two new modules to core: Automatic Updates (auto_updates) and Package Manager (package_manager). Package Manager provides the underlying API to create and manage a staging area in which to modify the Drupal code base, and will be used by the Project Browser module as well. Automatic Updates will implement the readiness check API along with several basic checks, plus the functionality to update core (and only core) in the UI or during cron.

    Core will also receive new third-party runtime dependencies, specifically php-tuf/composer-stager.

    The modules are being added to core in #3253158: [PP-1] Add Alpha level Experimental Automatic Updates module.

  3. Composer support

[Not yet in scope] Phase 3 (Post-stable features)

The following features are not part of the initial core scope, but may be added in future releases once the core module is stable:

  • Automatic updates for contributed modules/themes

Initiative Team

Related initiatives

Comments

hestenet created an issue. See original summary.

hestenet’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
mherchel’s picture

I want to chime in that I think this is an 💯💯💯 excellent idea! This feature definitely won't be for large enterprise level sites, but for smaller sites that have writable file systems. It's important for us to still cater to small sites because that's where the community comes from. Every long term Drupaler I know (including myself) started using small sites. Initiatives like this will help immensely.

I had some discussions over twitter on this, and people have expressed expected concerns. I think these could be mitigated by having this automatic updates disabled by default. We also need an extremely clear policy on what updates will be pushed, and maybe even have that be configurable (all updates, security updates, armageddon-level updates, etc).

Yay!

hchonov’s picture

One of the problems we have to account for is that the updates doesn't run in a specific order, which might break the site if not tested previously. The problem arises mostly when having contrib modules installed. We've recently encountered such a problem where we were unable of updating between major releases and had to update step by step - from minor to minor and from the last minor update to the next major.

bramvandenbulcke’s picture

At the Driesnote in DrupalCon Vienna there was a slide showing around 60% of users still use manual update procedures (FTP'ing tarballs). This huge number shows there is a large, mostly silent, group we should be aware of.

I would say that this initiative is vital for Drupal. For normal users, for sure. But also to prevent negative press when there will be big security hole similar to Drupageddon.

Of course, there should be an option to disable it.

anthes’s picture

The “auto updates for Drupal” discussion is going on for years now (see https://www.drupal.org/node/2367319 ). I’m happy to see a new “wave” of discussion and communication, let’s bring it to the next level. Hackers are used to automate scripts to outsmart us. Why shouldn’t we learn from hackers so we can improve and finally use our automation skills to fend off these attacks?

@mherchel, I want to add my thoughts regarding your opinion, that update automation won't be for large enterprise level sites - I’m part of a team, which provides a Drupal update automation management solution, also for enterprise customers.
Not only a few agencies automate Drupal updates (at least partly) with custom scripts already, that are executed by Jenkins or other CI tools. They face a common challenge: maintaining the bunch of scripts and keep on track with development of DevOp changes, new libraries, tools and workflows.
The lack of control of the update process is what concerns most DevOp teams and developers, so here’s one point which needs to be optimized so enterprise user will be happy with automatic updates as well. So yes, using update automation as an enterprise level Drupal shop sounds like a bad idea, if the update will be applied automatically without processing a whole pipeline with quality assurance etc. But that should not be the reason to exclude enterprises from update process optimization and increased website security, but to especially address this issue and find solutions for it.

So yes, automatic updates shouldn't be a default feature, every Drupal site owner should be able to decide about the way his or her site should be managed.
I absolutely agree with @mherchel, that a clear policy on update handling is needed. It's unlikely that a one-size fits all solution will work - that’s why a solution must be able to handle the different update behaviors, individual update workflows and security requirements of a user and a website. So, @hchonov is totally right, that the update pipeline needs to be managed within a specific test and time schedule system as well.

I’m looking forward to see more people joining the update automation discussion and exchanging their thoughts, expectations and concerns.

Mile23’s picture

An obvious use-case not already linked here is #2367319: Implement automatic background updates for highly critical security issues

It's a bit of a chicken-egg type development issue because enabling auto-updates for only super-scary security issues could happen without whatever infrastructure requirements are needed for a more general-purpose updater.

bramvandenbulcke’s picture

We should talk after the 28th of March: https://www.drupal.org/psa-2018-001.

Mile23’s picture

I just did some very very rough spec for #2940733-8: Site Builder Tool/Project Browser initiative

It seems that whatever build tool is generated for that issue would be used to generate a round-trip updater, particularly if one of the requirements here is that the user might download an updated tarball from d.o, generated to their spec.

#2940733: Site Builder Tool/Project Browser initiative seems like it could happen in the short- to mid- term. It could even create a tool that is used by the testbot to build the system under test.

pwolanin’s picture

Issue summary: View changes
pwolanin’s picture

Issue summary: View changes
giorgio79’s picture

"Many organizations have change management policies that would not allow this."
Like whitehouse.org that jumped the gun when a cheaper alternative came along? (Hint WP)

"Many Drupal-optimized hosting providers would not support writing new code on the webserver."
Really? How do they support Wordpress then? Drupal is a shrinking market size, so they wont last long.

"If the main beneficiary is build-and-forget sites on low-end hosting, does that justify a huge effort?"
Wow... Which site does not benefit from easier maintenance, and lower total costs of time spent on site updates??

All these questions seem to have been raised by nonsensical security "experts". We have them at my company too. Frankly, according to them the most secure solution would be to not have a website. The web is insecure, so have a backup. Period.

phenaproxima’s picture

All these questions seem to have been raised by nonsensical security "experts". We have them at my company too. Frankly, according to them the most secure solution would be to not have a website. The web is insecure, so have a backup. Period.

I'm not a security expert, and I believe that automatic updates are worth doing, but this is attitude is incredibly dismissive and naive. Security doesn't simply come down to "have backups in case you get hacked". If code can be executed remotely, for instance, the entire server is at risk, and not just the web site(s) running on it. Hell, it's possible that the entire network could be at risk. Depending on what network that is, the results could be downright catastrophic. Although it had nothing to do with Drupal, the Equifax breach (for one) came down to the company not taking security as seriously as they should have.

It would be extraordinarily irresponsible, bordering on downright mean, to just shrug our shoulders at the complex security issues presented by automatic updates. I think it's worth addressing those issues, but we need to do that in concert with the security experts. As you pointed out, the unfortunate reality of the Internet is that it's a dangerous place, and security is inconvenient for developers, but tough beans; that is the nature of web development. It's not fair to dismiss the security team's concerns and expertise. It's easy to scoff at security until you get hacked, your clients are freaking the hell out at you (in many cases, they could even take legal action against you), and you are in over your head and don't know what to do. Security breaches are the 11th circle of hell, and the security team deserves credit for trying to keep us all out of it.

So, if we're going to have automatic updates, let's figure out how to do them as securely as we can. I'm sure we can do it.

giorgio79’s picture

Ok, lets rephrase the question. Should Drupal be concerned with military grade security? If yes, the military better cough up the budget and not bog down Drupal development with excessive sec requirements.

phenaproxima’s picture

Let's let the security team decide what level of precaution and security work would be appropriate for this initiative, rather than throw vague conjecture around.

dsnopek’s picture

I suspect there are just as many plain technical challenges to make this work as security challenges!

However, to put the security challenges into perspective - the Wordpress auto-updater had a serious security vulnerability that could have allowed an attacker to deploy arbitrary PHP code to all Wordpress sites that have the auto-updater enabled:

https://www.wordfence.com/blog/2016/11/hacking-27-web-via-wordpress-auto...

It never was exploited, because it was discovered by Wordpress developers who followed responsible disclosure: it was reported to the Wordpress security team and fixed before the issue was made public. So, they got lucky. :-) But Drupal definitely wants to avoid a similar situation.

colan’s picture

The way I see it, there are several ways to go about this:

  1. Build this functionality into some secure automatic back-end process. The front-end should not be able to write to the file system. We should probably use Airship's model as a template, given that they've already got this working.
  2. Don't do this in Drupal itself; let the hosting infrastructure handle it. In Aegir, for example, we've started talking about adding support for "distributions", which you can think of as trackers for Drupal source code repositories (upstream or custom). Whenever an update is available (determined via polling or webhooks), a new platform (Aegir-speak for a Drupal codebase) is automatically provisioned. After CI tests have all passed, sites running on older versions of that distribution can automatically get migrated to the new one. (Aside: We probably won't do too much more work on this for Aegir 3 as we're figuring out how to architect it for Aegir 5.)
  3. Both. Provide the automatic updates in the application, but let folks disable it if they'd rather handle it outside of Drupal.
imclean’s picture

@colan,

2. Don't do this in Drupal itself; let the hosting infrastructure handle it.

This sounds like a good approach. Pantheon also supports this via git using upstream repositories, both within Pantheon and custom. There's a little bit of work to get the process to work with composer but perhaps that's where the effort should go.

mbaynton’s picture

@pwolanin, you made some pretty big issue summary edits in #12 a few days ago that I think may need more discussion. Mostly I'm not sure about

Drupal itself would have to be substantially transformed in order to meet this goal.

Could you explain some more why you think this is necessary? Is your concern essentially about what badness might ensue if you attempt to update code that's used to perform the update itself?

Rob C’s picture

2 should prolly be a separate issue (or even github project, this can support more then just Drupal). Got a library i wrote 5 years ago that's called by a git hook on the remote to deploy new releases. Supports wp and drupal7/8 atm and uses composer / drush make to build a new codebase during a release. If we split these issues we can dive deep into this concept and let the good people here fix auto updates via the UI.

SKAUGHT’s picture

2. Don't do this in Drupal itself; let the hosting infrastructure handle it.

If there were 'how to upstream' instructions, with the same idea more common now of 'using composer to build' that would be good for those who at least keep project repo's; would be nice.

But really, Core does need to support it as functionality. We can't assume 'we know how most sites run' on any server.

colan’s picture

#22:

2 should prolly be a separate issue (or even github project, this can support more then just Drupal).

It's funny you mention that, because the whole point of Aegir 5 is that it will be able to host anything, not just Drupal. Instead of Drush, the back-end will be Ansible to start, and possibly Kubernetes later. This is enough of an aside though; sorry about that.

#23:

But really, Core does need to support it as functionality. We can't assume 'we know how most sites run' on any server.

Right. Given that there are no fixed hosting infrastructures for Drupal at the moment, maybe we should reserve this issue for things that Core can do. Other hosting solutions (e.g. Aegir) can try to come up with solutions in their own projects. It's just that it would be nice if hosted applications didn't need to worry about this, but we're not there yet.

groovedork’s picture

"2. Don't do this in Drupal itself; let the hosting infrastructure handle it."

The point of this project is to get people on shared hosting some security. I am one of those people. I don't even know what Aegir is, and I don't believe I should have to.

There was an idea (excuse my noobishness if I explain it wrong) to let the drupal.org server do the composer part. It would then tell the installation what files to download. Outsource it. That sounded like an interesting compromise: still have composer run the show, but not needing to have it on your own server.

Mile23’s picture

@groovedork: That's #2940733: Site Builder Tool/Project Browser initiative where the basic idea is to let D.O build your codebase for you based on some specs.

effulgentsia’s picture

Drupal itself would have to be substantially transformed in order to meet this goal. Likely this makes automatic updates infeasible before Drupal 9. Some possible requirements:

For additional background on these proposed requirements, check out the proposal from @pwolanin and @David Strauss at DrupalCon Nashville.

I disagree that their proposal is not feasible for a Drupal 8 minor. I think it's worth exploring whether it can be done without breaking semver, and with reasonable backwards compatibility.

hestenet’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
hestenet’s picture

Finally got settled after DrupalCon and updated the issue summary with notes from both the Automatic Updates Architecture core conversation (https://events.drupal.org/nashville2018/sessions/drupal-core-auto-update...) and the 'Hard conversation' conversation about this topic.

There seems to be broad consensus that this and several other similar initiatives are dependent on the same precursor task:
"Improving Core's relationship with Composer"

To be more specific, the initial tasks (which may have issues, or may still need issues created) seem to be:

  • Move the vendor directory to where it needs to be for clean and consistent auto-updating, even in tarballs and non-composer installations. (Vendor and docroot should be in the root of the tarball)
  • Determine how to manage assets (scaffolding, a/b subfolders in the /files directory?)
  • Ensure that the overall filestructure supports this initiative, but is also compatible with related efforts(sitebuilder tool/project browser, etc)
anavarre’s picture

@hestenet could you please make the GDoc readable by everybody? Currently it requires asking for permissions to read it.

Mile23’s picture

There seems to be broad consensus that this and several other similar initiatives are dependent on the same precursor task:
"Improving Core's relationship with Composer"

The Composer initiative is here: #2958021: Proposal: Composer Support in Core initiative

colan’s picture

So should we postpone this one then?

Mixologic’s picture

I dont think so. I think the direction this is heading is 'create an updater for a composer based application' -> so something can be POC'ed and work can begin on it, with the aim that eventually core will be 'a composer based application' and will be able to leverage the updater. The other initiative will work in conjuction with this one, but doesnt block progress on this.

David_Rothstein’s picture

I disagree that their proposal is not feasible for a Drupal 8 minor. I think it's worth exploring whether it can be done without breaking semver, and with reasonable backwards compatibility.

Along those lines, I think it's also important to think about what the minimum viable product is. I still think it's #2367319-119: Implement automatic background updates for highly critical security issues, which is definitely achievable in both Drupal 7 and Drupal 8. (Along with #606592: Allow updating core with the update manager to extend it to core updates, which is harder but should be achievable also.)

Composer support, support for rollbacks, etc., are all useful and important features too, but I don't think they're part of the minimum viable product simply because the Update Manager doesn't support those features currently, yet many sites still use it to perform (one-click/semi-automattic) updates.

wturrell’s picture

Re: milestone 2 schema update risk mitigation, would it be safer to also take an entire sql dump rather than just relying on transaction rollbacks, in case a problem only comes to light later?

(I like that the security updates without schema changes have been prioritised in 1.0.)

tatarbj’s picture

Issue summary: View changes

Links were broken in the summary, so i just fixed them. Also looking forward desperately to join this initiative when issues will be defined and sprint on them!

serundeputy’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes
Issue tags: +MWDS2018

Added this to the issue summary based on conversations at #MWDS2018

There are three parallel efforts to be approached:

  1. Improving the manual update process by making Update Manager able to update Drupal sites (this is primarily to cover the vast installed base of older D7 sites)
  2. Updating the structure of Drupal itself (in future versions of 8 and moving forward) to better support symfony-like structure and composer workflows.
  3. Creating an unattended auto-update process (that would be available for *new* Drupal versions) that uses a net-loader style code-as-cache mechanism to swap between an A/B codebase for failsafe and reliable unattended updates.

Though for more detail @pwolanin, @mbaynton, @webchick, and/or @David Strauss will need to chime in

wvd_vegt’s picture

For me the fastest way to update drupal8 is to

  • unzip the new version
  • in the unzipped version, rename core to _core
  • in the unzipped version, rename vendor to _vender
  • upload _core and _vendor
  • put the site in maintenance mode
  • clear cache(s)
  • rename core on website to -core
  • rename vendor on website to -vendor
  • rename _core on website to core
  • rename _vendor on website to vendor
  • clear cache(s) (probably not necessary)
  • take site out of maintenance mode
  • done

Additionally have a look at changes in .htaccess and robot.txt.

Except for the additional task, this should be easy to implement in php as well.

handkerchief’s picture

With the recent security updates you have noticed again how fast you have to install Core Updates, before hacking scripts try to crack the website. I think at least these critical security updates should be able to run automatically. Maybe you should be able to select this option per instance, whether you want it or not. I know you've already planned it that way. With the recently released security updates it has become clear again how urgently we need such a function.

xjm’s picture

Issue summary: View changes

Adding a link to the meeting minutes and agendas, and removing the "related initiatives" to the end since this section seems to confuse people.

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Removing the old roadmap items (which were really just notes from the original core convo).

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

(Porting over info from the MWDS purple roadmap that @mlhess, @pwolanin, @David Strauss, @Mixologic, @mbaynton, @webchick, @hestenet, and I collaborated on. Sorry if I've forgotten anyone.)

xjm’s picture

Issue summary: View changes

Adding the section of the roadmap about completing and potentially forward-porting #2995202: Add a facility to update core in-browser that doesn't have core dependencies.

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

(And finally adding the later phases.)

pingwin4eg’s picture

Happy New Year, @xjm, and thank you for updates! You're the real Santa! 🎅

c_estDick’s picture

Not sure what now the status is, but I still be bothered by drupal 8 missing functionality:
Manual updates required

Automatic updates of Drupal core are not supported at this time.

The easiest fastest method to copy/migrate/core update/upgrade is using softaculous.

But softaculous has a delay in releasing their automatic installer so you will have a few days an extra security risc with core security updates.

Very pity that the functionality that softaculous delivers is not standard functionality in drupal.

It would be the best to implement devops and continuous integration and continuous delivery principles.

Very nice to have would be a fully automatic setup and maintenance and migration for a build street with environments for:
- Development
- Test
- Acceptance
- Production

(this would be a good start, since it might be later enhanced for branching, testing, ... , automatic testing/installation, ...)

This would be in my opinion a must have to facilitate developers, testers, ..... Their voluntary time is limited and any automation will help them spend their time more usefull.

For small sites would automatically create an acceptance environment by copying the production environment inclusive content and vice versa be super. It would deliver efficiently functionality to experiment with new plugins, updates, upgrades, test everything works fine at their server configuration before to release it to their production environment.

Now it is all to much effort for the less huge websites to do this with tools like git and jenkins and import export content with many scripts which are just customised for one specific production environment.

Why put effort this functionality in small (or medium) websites? Well they attract developers and other volunteers and their websites can grow huge and the less effort anybody needs to maintain a drupal site will boost drupal popularity.

heddn’s picture

Issue summary: View changes

There was a couple small informal gatherings at midcamp. Here's a summary of some of those conversations.

But first, we are making some progress over in #3039120: Create initial feature to display relevant PSA data in Drupal to build out some of the initial aspects of a PSA system. This is currently in contrib but there's also a core issue to take and move the features into the update.module.

Discussion points:

  • Backdrop has some disabled features around updating in core. It ended up being less complicated then expected to enable the functionality. See https://github.com/backdrop/backdrop-issues/issues/3105
  • the UX for manual updating is a little rough. authorize.php is what opens things up for FTP and updating files. That system wouldn't work for autoupdates. Backdrop is going to require that users who autoupdate must run in a security system where the web user owns the files. Yes, it isn't as good security in-depth, but doing autoupdates will need it.
  • Use 2 update feeds. Similar to how debian/ubuntu has a bleeding edge and a stable feed. Then give folks options of which of the feeds they wish to subscribe. Gives people the option in the case of the recent PHAR issue to subscribe to stable and not update as quickly. For low risk updates, the security team could quickly release to both stable and bleeding edge in short order. But for other updates, they could hold off a few hours or even a day or two while we have some "real world" testing take place.
  • Another idea, similar to two feeds was to provide options for folks to subscribe to auto updates for certain criticality of security updates. i.e. Only apply updates if the security release is a 15+, 20+, etc.
andypost’s picture

@heddn is there option to ship core as phar file? I recall some core issue about it

andypost’s picture

@heddn yes, exactly! not sure about performance effects but I sure updating one file is much each then a set

AaronMcHale’s picture

#58

@heddn yes, exactly! not sure about performance effects but I sure updating one file is much each then a set

Wouldn't Composer accomplish the same thing?

andypost’s picture

@AaronMcHale there's no composer on live sites and it should not be there to build, delivery most of time made with git or tarballs to be atomic

AaronMcHale’s picture

#60:

@AaronMcHale there's no composer on live sites and it should not be there to build, delivery most of time made with git or tarballs to be atomic

Ok sure not every live site will have Composer, some will, but yeah I can see where you're coming from where you have multiple environments for dev, test, stating and live. So if you're using Git for everything in Live, then would having core in a PHAR File really make much difference? What I'm really getting at is, would the investment of time and effort to make Core optionally become a PHAR File be worth it when (as far as I can see) there would be no change to the process or end result?

To be clear, I'm just trying to understand where you're coming from, not trying to be difficult

heddn’s picture

For the "how" of security updates, we'll be discussing that over in #3043235: [Policy, No patch] How to provide automated, in place security updates of Drupal core. There's already been some thoughts pulled together on that and over the next little while I'll start copying some of that into that issue. Let's take the phar or composer update, etc discussion over to there.

mbaynton’s picture

Issue summary: View changes
heddn’s picture

Issue summary: View changes

I'm noticing that we haven't published/updated anywhere the scope of the sponsored work in this space. It has a limited scope that might under-impress some folks. But in other ways, it will accomplish a lot.

From this issue's summary, the 3 main areas that are in scope are A.4, A.5 and B.2

heddn’s picture

A few things that one might be able to deduce from #64 is that there is no mention of composer. If composer is ready and able to work, we'll use it. But if it isn't, we can still do use update.module's features to auto update core. Turning on core's update in place mechanism is more a removal of protection code in update.module. It isn't particularly complex as discovered by backdrop. So, if we really, really want to use composer workflows with the funded auto update work in B.2, then we should spend a lot of time getting those external dependencies resolved.

heddn’s picture

Issue summary: View changes
cola’s picture

We create a small dev module, which works with composer and drush 9. You can combine this i.e. with a shell script, drush 9 and composer to autoupdate Drupal with security updates. https://www.drupal.org/project/autoupdate

mbaynton’s picture

Wrapping up DrupalCon Seattle, we will be updating this issue's summary with the consensus stakeholders have reached soon. The current issue summary needs to be dialed in so it's clearer what the initiative is doing, and this is going to be a big edit. If this results in someone's ideas currently expressed in the issue summary being lost, I apologize, and please point out the omission.

dstol’s picture

Issue summary: View changes
dstol’s picture

I trimmed down and clarified current scope and direction.

mbaynton’s picture

Issue summary: View changes
dstol’s picture

Issue summary: View changes
AaronMcHale’s picture

I posted this in the #autoupdates Slack channel yesterday, can't decide if it's in scope here but seems relevant and at least leaving it here might spark further discussion:

Here's a fun thought, when running composer update it will warn you if any packages are deprecated/unsupported and inform you if that package suggests an alternative. That made me think, we could replicate that behaviour in Drupal for projects hosted on Drupal.org (assuming that doesn't exist already), where on one of the update or status report pages you would get a warning somewhere if one of your installed modules or themes was marked as deprecated or unsupported on Drupal.org, we could also look at providing a field for projects on Drupal.org to suggest an alternative and display that in Drupal as well. We could also do that check when someone tries to install a module or theme which is marked as deprecated/unsupported, in the same way that you get a warning when installing Core Experimental modules.

SKAUGHT’s picture

As an idea, sounds cool yes: a composer plugin could be made to replicate the module update page.

But on the level where one would somehow have composer check on its own to see if core/contrib/other_venders updates/status exist would be needed for a site that is built from drupal-scaffold.

we could also look at providing a field for projects on Drupal.org to suggest an alternative

Sounds like a good undertaking too. But from my experience with contrib lifespan issues -- deprecated/unsupported is already the flag for 'user be warned' if needing to use. Having a way for that to be marked on Projects and the information actually being kept up-to-date (if alt project exists) would be amazing.

--> if Project Maintains had access to that field on other projects would be key in that. Opposite to that: if a Project had a field for a 'simliar to' reference.

Another side of that are project issue where 'maintainer wanted' never is applied for, or 'applying to maintain' is never allow to maintain. which can be some reason why projects fall to deprecated or unsupported flags by core. Then others come along build new modules (duplication) cuz old project namespaces (the project repo) is locked.

NaheemSays’s picture

I have watched the drupalcon presentation on this and it was very informative, thank you.

In relation to the Develop A/B version support for providing updates this will be an awesome feature if/when it can be delivered. I do think a though its scope can be reduced a little.

With contrib I assume that database schema changes are more likely than core, so rolling back wont ever be a simple step. This could be lft out of it.

1. Only provide this support for core.
2. leave the (contrib) modules and themes folders alone to work the "old way".
3. Instead of using symlinks, a file can be written in the sites folder containing the currently active codebase which the index.php should use.

In relation to Robots.txt, afaik there was an effort to turn that into a service. If that is completed, it should no longer be a problem. .htaccess generally propagates down a folder structure, so it should be less of a concern, but maybe it can be simplified to delegate to another .htaccess in the live code site.

This would could provide the following structure:

index.php
.htaccess
modules
sites ( and allow specificity for multisite by allowing it to be copied to sites/sitename)
--> livecodebase.inc
themes
core
--a > with current/stale tarball structure minus the modules/sites/themes folders
--b > with current/stale tarball structure minus the modules/sites/themes folders

Which ever path is taken to achieve it, its an exciting prospect to have this feature.

EDIT

Just as an afterthought, instead of A/B, why not use the version numbers? I am guessing something like this will require testing before it can be widely deployed, so maybe create a separate atomic core release for testing?

dstol’s picture

Issue summary: View changes
colan’s picture

Re: #73 & #74: We've been struggling with this issue in the Upgrade Status module via #2799369: Integrate with Contrib Tracker module. As of #303588: Show suggested replacement for obsolete modules, we've been hard-coding this data, but a better solution to help with these problems would definitely be a Replaced/Deprecated by field added to drupal.org projects.

Is there an issue open for this someone, or should we create it?

markmangoba’s picture

Component: Idea » Active Initiative
Assigned: hestenet » markmangoba
colan’s picture

ressa’s picture

Lucas Hedding posted this on Twitter:

Today we release parallel D7 and D8 alpha1 versions of of https://www.drupal.org/project/automatic_updates. Features include public service announcements, both in-site and emailed to site owners of upcoming highly critical updates + readiness checks for applying updates to the site.

https://twitter.com/lucashedding/status/1163527251176185863

heddn’s picture

Issue summary: View changes
grasmash’s picture

I'd like to circle back around to the issue of Composer usage for automated updates. There was a prior comment stating:

> If composer is ready and able to work, we'll use it.

It looks as though there's a good chance that Composer will be formally and well integrated with Drupal core by the 8.8.0 release. Notably, Drupal Scaffold has been moved into core and improved, and we're well on our way to completing #2982680: Add composer-ready project templates to Drupal core.

Given that, it seems as though Automated Updates should use Composer. Are there any blocking issues?

geerlingguy’s picture

@grasmash - One difficulty we will encounter (I presume) is the availability of RAM... a lot of Drupal sites, maybe even the majority of those which would use an auto-update mechanism (vs. manage updates via VCS / build process), might only have 256 or 512 MB of RAM available (total) on the system.

This already causes issues that I get pinged on quite often if someone runs composer install. Running composer update takes considerably more memory, and would likely run into lots of weird issues (e.g. someone has automatic updates enabled, but no updates ever get applied because the process fatal errors out, and they start filing issues on Drupal.org about it).

Do we have an issue somewhere about finding ways to limit Composer's memory usage or maybe even do something like "don't enable if system memory is less than X"?

AaronMcHale’s picture

Re #83:

Do we have an issue somewhere about finding ways to limit Composer's memory usage or maybe even do something like "don't enable if system memory is less than X"?

I like that kind of idea, maybe we could have a mechanism which attempts to use Composer but if it fails due to memory or other common issues the process falls back to another method.

andypost’s picture

Btw using https://packagist.org/packages/zaporylie/composer-drupal-optimizations and removing dev dependencies allows to use composer on 512M limit for years for me.
Surely it could be improved to make predictable if not scalable

geek-merlin’s picture

@geerlingguy:
> One difficulty we will encounter (I presume) is the availability of RAM

Good point. Others already solved it. The hard part is generating the lock from the constraints, and this can be done remotely by a service and for a vast number of cases it can be cached (Map hash(composer.json) => composer.lock). I have no link at hand, but Contao does this for quite some time and they serve a lot of low-profile installations.

geek-merlin’s picture

Mixologic’s picture

Contao's cloud resolver is functional because Contao is relatively small, usage wise.

Something the size of drupal would not be able to handle dependency resolution as a service due to the fact that when a critical security update is released, we would have to calculate *all* of the existing sites, all as fast as possible right after the release is announced. Basically we'd be building in a DDOS to drupal.org's infrastructure.

We can handle the current setup because of distributed caching of files. But if we had to actually do computation for every site, that would be a large, large load indeed.

NaheemSays’s picture

Can a resolver be written/done in javascript?

For the GUI update I suspect most users will have enough memory on whatever platform they are using the GUI from.

Mixologic’s picture

That's an interesting proposal, but would require what is essentially a Javascript port of composer. If there were people who wanted to maintain that in parallel with upstream composer, that would be great, but the number of people with the skillsets, needs, time, and desire to take something like that on may be rather rare.

AaronMcHale’s picture

Can a resolver be written/done in javascript?

For the GUI update I suspect most users will have enough memory on whatever platform they are using the GUI from.

That's an interesting proposal, but would require what is essentially a Javascript port of composer. If there were people who wanted to maintain that in parallel with upstream composer, that would be great, but the number of people with the skillsets, needs, time, and desire to take something like that on may be rather rare.

That also wouldn't really deal with unattended updates, assuming that's something which is in scope here?

Mixologic’s picture

That also wouldn't really deal with unattended updates

Definitely true as well.

David Radcliffe’s picture

Issue summary: View changes
8thom’s picture

Noting here that Yanick from Contao has released a more generic composer cloud resolver -

https://composer-resolver.cloud/

more info: https://medium.com/@yanick.witschi/composer-cloud-resolver-e64254f5728e

clemens.tolboom’s picture

Issue summary: View changes
clemens.tolboom’s picture

Issue summary: View changes

Add link to contrib project

hestenet’s picture

Assigned: markmangoba » hestenet
webchick’s picture

Priority: Normal » Major

Surprising ABSOLUTELY NO ONE, in an informal survey about 9.1 features on Twitter the other day—https://twitter.com/webchick/status/1235689652838776832—this was an extremely popular option. (There will be a formal survey coming in the next couple of weeks; this was an attempt to gather up some options for it. :))

xjm’s picture

Priority: Major » Critical

Promoting to critical, because if anything is a critical feature, automatic updates is.

xjm’s picture

xjm’s picture

Issue summary: View changes

Since the IS is basically 100% out of date currently, I've updated it with a rough outline for a current roadmap that we can populate with more detail.

xjm’s picture

Issue summary: View changes

Adding session recording links for sessions that are already public.

xjm’s picture

Title: Automatic Updates initiative » Automatic Updates Initiative overview and roadmap
effulgentsia’s picture

tedbow’s picture

I know in the Drupalbon BoF we had marked the readiness checks and the PSA as issues we could get into core before the rest of automatic updates gets in but I am not sure if the Readiness Checks makes sense without Automatic Updates.
I have looked through all the checks https://git.drupalcode.org/project/automatic_updates/-/tree/8.x-2.x/src/... and I am not sure any of them make sense except PendingDbUpdates and there is already a check in system_requirements() that checks this.

We don’t know how people are going to update their sites if they aren’t going to be relying on future Automatic Updates.
Here are the existing ones in the contrib module

  1. CronFrequency - with the updating via cron this doesn’t make sense
  2. DiskSpace this seem like it could be a problem that really isn’t just related to auto-updates. but also a lot people if not using composer probably delete previous core and replace with new one. auto-updates needs a temp area for the update but that is not necessarily true otherwise
  3. FileOwnership this makes sure the user running PHP is the owner of the files which doesn’t need to be true if you update manually.
  4. Filesystem abstract class but has base tests. Tries to finds webroot and vendor directories and that they are on the same logical disk if the site is currently running these were obviously working for Drupal. People might have different working setups though. It’s fine that they have to be in an expected place if updates are “auto” but otherwise it doesn’t seem like a good check
  5. MissingProjectInfo missing version or dev release
  6. ModifiedFiles this seem like it is going to warn a ton of sites of something they already know. We already have Composer plugins that help us manage patches so this seems like weird warning.
  7. OpcodeCache very specific to Auto-Updates running via CLI
  8. PendingDbUpdates already a check in system_requirements()
  9. PhpSapi very specific to Auto-Updates running via CLI(I think someone might know better)
  10. ReadOnlyFilesystem this is standard in many hosting systems
  11. Vendor if the site loaded obviously vendor directory is working though maybe in different location

I would love to get functionality in core in pieces to avoid huge patches but I don’t think we should add checks that are going to cause confusion if we don’t actually Automatic Updates functionality in core.

At Drupalcon we talked about adding Readiness Checks and PSA to the existing Update module and adding the actual Composer-based updating to a new experimental module with the functionality eventually being merged into the Update module.

I think the Readiness checks should be added as the first part of an experimental module I don't see any benefit in adding them to the update.module first since the plan is to have an experimental module anyways and thing we are actually checking for is the readiness for functionality of the experimental module.

dww’s picture

+1 to putting PSA reporting into existing Update Manager.
+1 to putting readiness checks into new experimental "Autoupdates" module (or whatever it'll be called).

At some point soonish, I'd love to have a conversation about the future structure of Update manager. In the era of composer, I'm wishing we had kept "Update status" as a separate module from the "Update manager" parts that let you install/update new code. Composer based sites might still want Update status enabled (e.g. to email them when security releases are available), although they almost certainly don't want the "update manager" parts. We have a settings.php killswitch to turn off the "manager" parts (allow_authorize_operations) but in hindsight it's a) poorly named and b) poorly advertised. Instead of adding yet more functionality to this beast, I wonder if it makes sense to keep things in separate modules to make it easier / clearer separation of responsibilities and easier to only enable the part(s) you actually want on any given site.

Should we spin off another child issue for this?

Thanks,
-Derek

solideogloria’s picture

I haven't read many of the comments yet, but is there any scope of what the automatic updates would be restricted to -- for example is it only security updates? Can you guarantee that manual core updates will always be an option to users?

I'm asking because a few years ago Microsoft began automatic updates for Windows 7 -> Windows 10 without asking people, which caused frustration and anger, and automatic updates also have the potential to be abused. It essentially makes it so that Microsoft is really the one in control, because they can push whatever code they want, whenever they want.

xjm’s picture

Issue summary: View changes
tedbow’s picture

Issue summary: View changes
  1. added link to drupalcon global presentation
  2. Added php-tuf information
xjm’s picture

Issue summary: View changes

Adding the DrupalCon EU video now that it is public.

xjm’s picture

Issue summary: View changes
tedbow’s picture

I am proposing a change in the order we implement features here #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow

The TLDR from that issue

should add an option to use a Composer aware backend using the Composer Staged Update plugin to the current Update module update functionality, first supporting modules and themes and then core updates. The UI could remain largely the same.

We should do this before we support unattended Automatic Updates because this does not open new security concerns compared to the current Update module and it allows us to implement a portion of the functionality eventually needed for unattended Automatic Updates.

More details and reasons on the issue.

effulgentsia’s picture

Issue summary: View changes

Renamed the a/b bootloader section to account for #3199171: Automatic updates: For MVP, instead of an a/b bootloader, implement a semi-atomic composer update with a brief maintenance-mode window being the approach we are currently favoring.

effulgentsia’s picture

Issue summary: View changes

Updated the "Update installation" section. Removed the notes about Composer 2 being a requirement, because Drupal 9's existing "require-dev" for composer/composer is already for Composer 2.

effulgentsia’s picture

tedbow’s picture

Issue summary: View changes

Added new recordings and updated slack meeting time

tedbow’s picture

Issue summary: View changes

Update PSA issue for core

tedbow’s picture

Issue summary: View changes
phenaproxima’s picture

Issue summary: View changes

Adding myself to the initiative team because I've been working on this for the past year. :)

phenaproxima’s picture

Issue summary: View changes
phenaproxima’s picture

Issue summary: View changes
tedbow’s picture

Issue summary: View changes
solideogloria’s picture

There appears to be a typo in the latest issue description.

This 8.x-2.x version of the module does not sites that sites that are not managed by Composer.

(Emphasis added)

tedbow’s picture

Issue summary: View changes

@solideogloria thanks, fixed I think

solideogloria’s picture

Yes, that looks right.

tyler36’s picture

Recently, I opened an issue to automate database updates via composer action scripts. It was closed as a duplicate because of this initive.

It seems the focus here is to automate the process of checking for and applying updates to core and modules.

However, my request was to automate a manual step that is required after a developer manually updates. This would be required as part of this initive but is much smaller in scope, and could be introduced into core now; a win for the community and an initive that is 5+ years old.

Greg Boggs’s picture

I believe this is the best issue queue to write about how to automate database updates:

https://www.drupal.org/project/issues/automatic_updates?categories=All

They may already have an issue for this.

xjm’s picture

Issue summary: View changes