Problem/Motivation

Many software programs (including CMS software such as WordPress) support automatic updates, in which the site applies an update on its own with no intervention from the site administrator.

We need to continue exploring this for Drupal. Here are some pros and cons identified in this and related feature requests:

Pros

  • Users want it. Some have found that it is worth leaving the platform for easier/cheaper to maintain alternatives.
  • It would be particularly useful for situations like SA-CORE-2014-005/PSA-2014-003 (very low-risk patch for a very critical security issue where speed is of the essence).
  • It would go a long way to improving the overall security of Drupal installations across the internet: the reality is many sites just go un-updated under the current manual processes. As a platform powering better than a million websites, it is the "good netizen" thing to do -- every site that doesn't get compromised translates to many outdated browsers not getting exploited, and personal computers not getting hacked.
  • Given the larger codebase and use of external libraries in Drupal 8, manually applying fixes is likely to become more frequent and on a less predictable schedule, making automatic updates even more valuable.
  • Even if a less conservative strategy is selected regarding the scope of code changes pushed automatically (ie, force sites to take feature releases when not at security release - 1), broken sites are much easier and cheaper to fix with confidence than compromised sites. Accepting some risk of breakage is therefore the preferred model of a segment of the userbase (additional attribution). Organizations running sites that secure billing and other confidential information may similarly find this risk model preferable.
  • It would not be forced on anybody. It would only empower site owners to choose whether it is the right solution for them. It could be built as a highly isolated module that can be disabled completely, which would make an exploit of the automatic update system itself very implausible on sites that don't think it is right for them.
  • Without some form of automated security update solution in place, I fear I may never be allowed to take a holiday again...(attr.)

Cons

  • Implementing automatic updates means substantial new code of nontrivial complexity. There is no way to guarantee that the new code will not itself have security vulnerabilities. From the original issue summary, "The risks of introducing automatic updates to Drupal are that we don't want to open up new security holes in the process."
    • Counter-argument: But can anyone argue that the net effect of the advent of automatic updates in other software has been decreased security? Automatic updates have been done, and done successfully, in many ways by many vendors with positive results.
  • A case is repeatedly made that it would require the site to be able to write to its own executable areas, and that this is by definition less secure.
    • Counter-argument 1: A number of implementation strategies have been suggested that would not require the site to directly write to its own executable areas. In one, favorable hosting environments would install purpose-built software running in a separate process from the site itself to verify and perform the update. For generic hosting environments that offer authenticated (s)ftp/ssh as the means to change code, site owners could choose to provide the update infrastructure with valid credentials ahead of time; these could be retained in an encrypted database whose private key is offline 99% of the time (attribution). Credentials would then be supplied back to the site (an encrypted channel may be achievable for this purpose through native php code) and kept in memory during an update event.
    • For sites that have been configured such that they can write to their own executable areas, "automatic security upgrades on systems with an already weakened security is a good idea. One might argue that every bit helps." (attribution)
  • It is a feature primarily targeting users with a low technical level, and they are not going to turn it on anyway (attribution).
    • Counter-argument: If you're smart enough to build the rest of the site, then you can make the decision that is right for you whether or not you wish to turn it on. If your site was built for you, the firm who did so is smart enough to turn it on when continuing support is not part of the agreement -- an option they don't currently even have. These assertions won't hold true all the time for all people, but that's not a reason for Drupal not to offer this feature.
  • "Assuming that the low technical level users do enable the feature, the very first time that using it "breaks" their site, either by forcing it into maintenance mode or by performing an update which causes issues with some functionality in the site, they will either start to resent the Drupal project or turn the auto-update functionality off." (attribution)
    • Counter-argument 1: The testing and review in place is sufficient that even if a less-conservative strategy is selected regarding the scope of code changes pushed automatically (ie, force sites to take feature releases when not at security release - 1), it will at worst break some sites and cause resentment for some people, and even that's not likely to be a regular occurrence. In the current state of affairs, all sites have a right to resent Drupal equally for its regular need to be manually updated, sometimes at inhumane hours of the day.
    • Counter-argument 2: Fixes to security vulnerabilities rarely require large-scale refactorings or other changes that pose a high risk of site breakage. The infamous example need not be named yet again. Drupal has an opportunity to decide to do better than WordPress and use the automatic update facility to backport and distribute critical fixes. Combined with the testing and review mechanisms in place, site breakage could become a very minor concern.
  • There are concerns that the infrastructure that is used for distributing updates to sites would become "the biggest, fattest target ever for hacking. Just hack your way to there and suddenly you can hack a million sites in one go!" (Attribution)
    • Counter-argument: This risk can be designed away by a good implementation. An implementation that merely serves up signed packages, probably Phar archives, that were signed using a private key on an airgap system nullifies this concern.
  • "My main concern would be with Version control systems and how would the updates be applies and tracked there." (Attribution)
    • Counter-argument: If you use version control, you're not the target audience, at least for an initial version of this feature. However, if the design keeps the distribution and deployment components decoupled, advanced users would be free to contribute additional modules to handle deployment to all these scenarios.
  • it's been mentioned twice in this thread that popular contrib modules with an equally severe issue could also benefit from a system like the one being proposed. The question becomes, where do we draw the line? (Attribution)
    • Counter-argument 1: This is a policy issue, and shouldn't derail the development of the automatic update technology. If a system was in place, there are some clear and appropriate uses for it that everyone can agree to.
    • Counter-argument 2: Site owners already choose which modules they trust on their sites, and if the system were used to push contrib updates as well, site owners could take that into consideration in the modules they select (or, potentially, allow to be auto-updated). When the software is free, the user accepts the risk of possible defects. Making updates available automatically wouldn't need to fundamentally change the relationship between the security team and module maintainers.
  • But the patch might not apply...
    • Counter-argument: What to do in the event of an attempted update that fails should be made configurable by the site owner, with the option to kill the site completely. This solution is discussed repeatedly, one attribution.

Proposed resolution

Implement a system to allow automatic background updates in Drupal.

It could be related to the current Update Manager functionality, but would likely need to be different because:

  • At least at first, we wouldn't want to use it on every security update, just a select few, and
  • We might want it to support patches (that apply against older releases as well, as was the case with SA-CORE-2014-005) rather than or in addition to actual version updates.

Security considerations

The risks of introducing automatic updates to Drupal are that we don't want to open up new security holes in the process.

Remaining tasks

Investigate how other systems implement this, including WordPress (see http://codex.wordpress.org/Configuring_Automatic_Background_Updates) see #82.

Decide if it's a good idea for Drupal and flesh out the plan.

Implement it.

User interface changes

There would likely have to be some kind of user interface to inform an administrator when automatic updates have run, or attempted to run but failed. There would have to be some method to turn this off for sites that absolutely don't want the functionality.

API changes

This would not be likely to involve any API changes, although certainly API additions. If approved and implemented, I see no reason it couldn't go in both Drupal 8 and Drupal 7.

Comments

David_Rothstein’s picture

Issue summary: View changes
galooph’s picture

Regarding the drush command, we've been working on something similar for the nagios module in https://www.drupal.org/node/1813340.

The drush command we're implementing there returns a list of modules with updates available and also allows the site admin to select modules for which updates should be ignored. In house, we're planning to hook this up to our Jenkins system to automatically update and launch a new stage build whenever updates are released.

mpdonadio’s picture

Another option for this would be to distribute Drupal as a special git repo (something w/o the whole commit history, just enough to go between tags). Cron could then `git fetch` nightly, and the updater could then either checkout or merge the new tag and then handle database updates. The would have the added bonus that you would also have a mechanism in place to detect whether core has been altered, since this would more than likely only be used in situations where the webserver has write access to core files, which add that additional security exposure.

dubcanada’s picture

I see no reason why you would not want this, I would much rather my website be broken then be vulnerable to the latest issue. I can't think of anything better then the way Wordpress handles their core updates.

So long as Drupal core releases have a proper release program and the auto update module has proper checking (as you said SSL, etc) I throw my vote in for this.

klausi’s picture

The problem with wordpress auto updates is that the webserver user must have write access to the code base, which is an inherently insecure configuration and is strongly discouraged. See https://www.drupal.org/node/244924 . For example with #drupalsa05 many attacks tried to plant evil PHP files somewhere besides the Drupal core files, which is not possible if you have locked down your permissions.

John_B’s picture

8thom’s picture

The concept is great but in reality I think will cause more issues than it will address.

Would it not be easier to automatically force a site into maintenance mode and email the site admin if a "highly critical" patch was available?

This could be an option of the update module (on by default) but then can be disabled for sites with a full time site administrator.
It can also be used for "highly critical" contrib module patches and targeted to only the sites which have these modules.

David_Rothstein’s picture

Would it not be easier to automatically force a site into maintenance mode and email the site admin if a "highly critical" patch was available?

It would be easier (although not totally trivial since currently sites get no information from drupal.org about which security patches are "highly critical"). However:

  1. Putting the site in maintenance mode would do nothing to protect against an issue like https://www.drupal.org/SA-CORE-2014-005.
  2. Many site administrators might really not appreciate having Drupal take down their site for them, even temporarily :)
8thom’s picture

Sorry, rather than just maintenance mode it would probably need to be something that completely kills the bootstrap.
I realise site admins would not appreciate having their site taken down but I'm sure they'd prefer it to the alternative in this case.

https://www.drupal.org/PSA-2014-003
https://www.drupal.org/node/2365547

I would imagine this only being used in only the most extreme cases and it should still be optional.

John_B’s picture

Having worked with a lot of non-technical small users with no budget for professional support, I do not think this would be acceptable on every security update. In the support forums we see many who are stuck for hours or days on a simple update (I have just answered a question where someone worked for 3 days without time to shower on updating 7.31 > 7.32, so far without success: this is not so unusual) . IMO Drupal has largely lost this market, and should maybe consciously move out of that space: nevertheless it is the low-end user group (i.e. those with almost no technical and financial resources) who this feature is mainly for. Fortunately a vulnerability this bad does not come up very often.

tsphethean’s picture

What if instead of a full blown upgrader, we took the approach of emergency patching and a prompt to do a proper upgrade?

Workflow could be something like:

1. Hook cron runs, detects there is an emergency patch
2. Determines if patch cleanly applies to file system, and if any dependencies (min version to apply to etc) are met
3. Apply patch if all ok, run updates if needed
4. Email site admin with success or failure of patching and prompt to do full version upgrade ASAP
5. Prompt on admin screens on subsequent logins that site is in patched state and needs upgrading

This still has the flaw of having to have the file system writable, which I'm not entirely comfortable with, and would assume the process would only be run for highly critical issues as described. It has the advantage of more easily being version agnostic (I.e. Patching 7.0 as easy in the recent issues as 7.31, whereas a full core upgrade would be huge), and catering for people who have patched/hacked core (as unadvised as it is many people still do for various good reasons).

catch’s picture

Status: Active » Postponed

The update manager is broken in 8.x, but if it worked, then #606592: Allow updating core with the update manager would allow for core updates with manual intervention.

I assume this issue is for automatic updates via cron, which seems like it'd have to be built on top of #606592: Allow updating core with the update manager, so postponing on that.

David_Rothstein’s picture

Status: Postponed » Active

As described in the issue summary (and then expanded on by @tsphethean in #11) this might makes sense to be primarily - or even entirely - about applying patches rather than doing an actual update.

Also, it's not necessarily just for core updates - it would make equal sense if a contrib module had a highly critical security issue in which the reward-to-risk ratio of applying the security patch is large.

Ec1ipsis’s picture

I believe this is likely to cause more problems than it will solve.

The low technical level of users for whom this feature might be beneficial suggests to me that many of them won't understand the feature well enough to understand the pros and cons of turning this feature on, and these users will likely ignore the feature, making the entire exercise nearly pointless. We could make it opt-out rather than opt in, but I don't think that's currently on the table, and for good reasons.

Additionally, assuming that the low technical level users do enable the feature, the very first time that using it "breaks" their site, either by forcing it into maintenance mode or by performing an update which causes issues with some functionality in the site, they will either start to resent the Drupal project or turn the auto-update functionality off.

I also feel that on a philosophical level, this isn't a good road to go down. Creating a system which is intended to take control away from the Drupal users seems to me to be an anathema to the OSS movement, and a step towards a more top->down type of project where some administrator tells you what and when you will be updating. I understand that at this time, we're discussing an opt-in system, but I feel that the farther we can stay from a top down system, the better off the project will be.

As an alternative, I feel that there's definitely space to improve administrative notifications and a space to educate users on the benefits of performing regular updates, and how to do that smoothly. That is a much less intrusive and less likely to cause angst amongst users than invisible hands making changes to their site.

catch’s picture

I don't think #11 requires the filesystem to be writable via the webserver, it just requires cron to not be run by the webserver (hence incompatible with the fallback cron in core).

John_B’s picture

Re #14, Opt-out is surely fine. The whole concept of protecting people who are likely to be late with updates could be called 'top down.' In State of Drupal London Dries compared iOS for the idea of something which 'just works' intuitively, as something Drupal should aspire to. I do not see that ideal as as objectionable if the user is given the keys.

The opt-out update system used by Wordpress for minor version core (and, optionally, plugin) updates seems pretty bullet-proof, and was made with robustness in mind. If it were breaking sites, the huge base of low-tech users would be making this clear.

Damien Tournoud’s picture

An approach to this problem would be to deliver Drupal and modules as signed PHAR archives. Have index.php be a simple, locked-down bootstrap script in which you select which signing certificates you allow, and then you can store the actual archives in a web-writable filesystem without any trouble.

chx’s picture

So there would be a drupal.org signing key so the packaging script could sign every release? And you guys want to deploy the results automatically? Well, that makes util.drupal.org (or wherever that lives) the biggest, fattest target ever for hacking. Just hack your way to there and suddenly you can hack a million sites in one go! Careful...

greggles’s picture

I think it's important as we explore this idea to consider problems, potential solutions to those problems, and the most likely users of different kinds of systems who are likely to use this automated update.

If we think about the users in terms of personas (even if we just break it down by "Samuel on shared hosting", "Roberta with Root (VPS/Dedicated)", and "Tamara's Talented Team") we can come to some decent conclusions. There are basically no shared hosting companies that support a configuration where the webserver runs as a different user than the owner of the files. This is frustrating and sad in a lot of ways, but that's the standard in shared hosting. We should push towards a world where that's not the case, but in the meantime...Samuel is most likely to ignore documents about secure file ownership, ignore @drupalsecurity twitter posts, most likely to ignore g.d.o/core posts, most likely to take a long time applying updates. He is the target user who needs auto updates and, conveniently, his hosting platform means that they will work. It's not a matter of arguing about whether his file permissions are right or not...he doesn't have a choice. You can argue he should spend more on hosting, but that won't make him do it. It's also the case that running cron as a different user to do blah blah is probably a feature not supported by his hosting platform and its probably too complicated to get widespread adoption.

Roberta, on the other hand, can do something that involves shell scripting or can probably pay someone to set it up. She can change her file ownership settings. She might weigh the costs/benefits of the auto-update feature and choose one of a few different solutions including auto-update inside Drupal or a shell script or doing it manually.

Tamara's team is going to have automated testing and a one-click deployment job built in some continuous integration tool. Tamara can and should disable the automated updates, but she would probably appreciate it if her site gave her 7 days of warning that a truly critical vulnerability were going to arrive at 17 UTC the following Wednesday.

Yes, this proposal introduces some new risks. Yes, we would need to add some security process and technology protection for triggering the "do an auto-update" xml feed. Let's be cautious as we explore and craft a solution, but let's be open minded. Let's all consider our personal biases/perspectives compared to the background of other folks and what makes sense in each case.

timodwhit’s picture

@Ec1ipsis, I agree that this is totally targeted at the low technical level person, and that person will most likely not understand what the configuration means and whether or not they should turn it on.

I however think that makes it more of UX/UI question rather than a should we do it or not. I think that those are the users that are often shipped complex sites and aren't sure how to properly maintain them, update them, change them, etc. Or they start out with Drupal because they want a complex site, but may be overwhelmed by the amount of options. It is a large eco system to learn, updating your site, vcs, etc can be overwhelming.

As far as control, we could always have it enabled by default on a site through a standard install profile/install profiles of the choosing etc and then for those who want more control, it can be disabled and they can roll their own profile.

My main concern would be with Version control systems and how would the updates be applies and tracked there.

John Pitcairn’s picture

she would probably appreciate it if her site gave her 7 days of warning that a truly critical vulnerability were going to arrive at 17 UTC the following Wednesday.

A big +1 to this. I managed to upgrade within a very short time of the SA-CORE-2014-005 notice, but consider myself very lucky to have been on deck early enough that morning (NZ time).

rickmanelius’s picture

A few other items to add to this discussion:

Drawing the line.
It's easy to point at SA-CORE-2014-005 as an extreme example when automated updating would work. It was a highly critical security vulnerability (arguably one of if not the most critical in Drupal's history). It was a single line change. It was Drupal core.

However, it's been mentioned twice in this thread that popular contrib modules with an equally severe issue could also benefit from a system like the one being proposed. The question becomes, where do we draw the line? How critical is critical for core, contrib, themes, and 3rd party libraries? Does a user set that threshold? (i.e. core only, core and contrib)

The issues are many. While the security team can be selective on how to release a core update containing only security updates, it's harder in contrib. Often a security release occurs in conjunction with any number of additional commits, some of which may be a work in progress before a maintainer is ready to cut an official release. It also puts a lot more onus on security team members that may not have the time to review the other commits a contrib module may roll in with the release.

My point is, the question of where to draw the line (and why there?) will be something that requires a gradient and a sane default. But what is that default? And will enabling by default potentially result in a broken site due to an automated deployment of a half-baked contrib module? I don't think there will be an obvious answer that doesn't come with it a list of caveats and edge cases.

Types.
I really like the suggestion by @greggles because it's (IMHO) unlikely that a one-size fits all solution will work. He provided a breakdown by hosting/infrastructure type. We may also wish to consider a 2nd dimension (website type). A site that updates infrequently (weekly) and has daily backups may not care either way because it can roll back and bug introduced from an update as quickly as it can a breach due to a security vulnerability. But what of sites with HIPAA or PCI Compliance? What about publishing and/or media sites where 30 minutes of downtime results in significant revenue drop?

Closing
My personal preference is to stay away from automatic updates because they fall outside of version control and open up new vulnerabilities. However, I do operate a few personal Drupal blogs and know that "set it and forget it" is appealing. I'm very curious to see how this thread unfolds.

chx’s picture

There are basically no shared hosting companies that support a configuration where the webserver runs as a different user than the owner of the files.

https://blog.nearlyfreespeech.net/2007/01/28/writing-files-in-php/

On our system, the user who owns your files is usually identified to you as “me” and the web server runs as user “web.”

Of course, NFS is unusual. And, awesome, btw.

For more normal hosting providers there's suPHP. http://forums.hostgator.com/showpost.php?p=283930&postcount=2

HostGator currently runs suPHP on all of its shared servers,

(Corroborated at http://support.hostgator.com/articles/phpsuexec-or-suphp)

chx’s picture

Note this doesn't necessarily rule out that automatic security upgrades on systems with an already weakened security is a good idea. One might argue that every bit helps.

greggles’s picture

I believe I have reviewed HG/suphp previously and I believe the default is that it runs the webserver with the same user that owns the files. This common default configuration is one of the major sticking points with the hosting advertising program on d.o.

8thom’s picture

My major concern is what hapens if/when the auto update fails.

Would it automatically roll back?
Could the site potentially be broken by the update?
Then do you leave the site active and vulnerable?

Like @greggles mentioned a large group of site owners will ignore security advisories but they'll be less likely to ignore the fact that their site is down.
In saying that, RE: SA-005 - is there a current method a diligent site owner could use to disable their site until they're able to contact their developer to upgrade/patch?

It's easier to explain a site outage to users than to have to contact them about a security breach, request them to reset their passwords and explain the potential privacy leak.

As for auto updates, we'd probably have to provide 2 options depending on weather the site was using a VCS or not.
Could possibly look into reviving the dog project which allows git based updates for core and contrib.

brylie’s picture

Automated anything should always allow administrative override.

helmo’s picture

Running such an update on cron could work in some cases. But lets not forget that the 12 hours interval which Wordpress uses would still mean that logs of sites would be updated later then the 7 hours mentioned in PSA-2014-003 .

nickholden’s picture

As someone who spent most of the time since October 15th (a) on holiday, and (b) cleaning up the subsequent #drupageddon mess (including some quite awkward conversations, as our site does collect sensitive personal information), I'd be keen for an automated update solution, although for the security reasons outlined above, I'd tend towards a drush-based one rather than a drupal-internal self-maintaining-code one. Something along the lines of the one-click CI solutions that the big set-ups (Tamara) will have in play but delivered in a mechanism that The Rest Of Us (TM) can feel comfortable with. I suppose I could just create a cron entry to run 'drush -y up' and mail me the results every hour, but something a little less potentially-destructive and a little more selective and tailorable would be great. I don't want trivial updates to modules and themes (especially themes) to be installed while I'm away, but I do want security patches installed at the earliest possible opportunity. Even having it running regularly overnight while I'm asleep might make a difference.

Yes, it raises the bar a bit in terms of technical requirements to set up such a tool. Not everyone is comfortable installing drush. But I don't think it's automatically the case that everyone savvy enough to install drush is necessarily going to have a full-service Continuous Integration tool to run bespoke one-click server upgrades, or a team on-call 24/7 to run the updates manually. There's a market out there for an automated security updates service that isn't quite as scattergun dangerous as the WordPress one. And maybe, with good documentation, it might help encourage some of the even-less-savvy users migrate towards a drush approach to managing their site.

There are big issues in terms of the security requirements implied by an automated update system, especially at the server end - but @David_Rothstein had that covered, I think, in his original post. I'd +1 the idea of security updates having to be pushed by more than one of the security team (each using two factor authentication) in order to actually be triggered in sites' automatic update systems too.

Without some form of automated security update solution in place, I fear I may never be allowed to take a holiday again...

timodwhit’s picture

Spitballing here: With semantic versioning, could we have a file that specifies what to update and what versions we are comfortable updating. For example: By default if 8.0.1 gets updated to 8.1.0, out of the box, that is an automatic update. For those who want more granular control, we treat it as a package manager with a .lock file and say only update patch releases: 8.0.1 => 8.0.2 using the similar flags found in other package managers like Berkshelf.

This could also then have a nice UI in front of it for module which because of detrimental updates/ lazy code/etc, should only be updated with patches unless tested.

This would put more responsibility on the community to release code in the semver way but seems like a pretty managable solution.

Thoughts?

hansrossel’s picture

I suppose next time we will have only 3 hours or less to update before the hackers have their automatic scripts ready, so there is probably no other choice than automatic updates.

I think that there will have to be two solutions anyway: a Drush solution for the professional sites, and some automatic update manager for the others.

For Drush I would like to point to #823146: Security Updates only with drush where there is a discussion on giving the drush up command more features, such as security updates only or maybe only core security updates, so we can run this on cron.

helmo’s picture

@hansrossel if you suppose 3 hours ... how often would a site need to poll for updates ... every hour? I doubt that the infra would be pleased with that :(

chx’s picture

@greggles thanks for pointing out I have posted without enough research, that was hasty of me.

hansrossel’s picture

@helmo security releases are on wednesday, so that's the only day you need to poll. Could be good if the security team would schedule an exact time when they are released, then we can run a drush update command 5 minutes after.

daffie’s picture

@hansrossel: I hope that everything will go as you suggest. How many sites will check at the same if there are updates. But with the right infrastructure it is manageable. And I hope that the security team will test the update as good as humanly is possible. If something will go wrong with the update, the problems will be massive.

greggles’s picture

Every patch that comes out of the security team queue gets some level of manual QA, but by the nature of the work the number of people doing QA is smaller than the number who do it in the public queue.

However, We run the automated tests on every patch to find regressions. So, if you want the security team's QA to be good, then you should write and contribute automated tests that run on drupal.org for the feature-set that you care about.

daffie’s picture

@greggles: I know that you are right. But at the moment I am struggling just to learn and understand Drupal 8. And not to try to say something stupid on an issue queue.

masipila’s picture

@greggles: I like how you analyzed different Drupal user groups in in #19. Especially I liked this:

[...] but she would probably appreciate it if her site gave her 7 days of warning that a truly critical vulnerability were going to arrive at 17 UTC the following Wednesday.

I didn't follow @drupalsecurity on Twitter so I didn't receive the pre-warning about SA-005 in advance (by pre-warning I mean the tweet that site owners should prepare to upgrade between noon-2pm). I was lucky to be awake when the actual SA was emailed to the mailing list. While I'm all in for the drush-based approach discussed in this thread, I also think that this kind of a pre-warning that a SA will be published on dd.mm.yyyy at hh:mm would help the situation a lot. I would also highly appreciate if these kind of pre-announcements would be sent to the mailing list and not just to Twitter in the future.

Cheers,
Markus

ofusriley’s picture

What about an option for this:

Instead of automatic updates, automatic maintenance mode. When a critical security patch is available and this option is enabled, the site will go into a defensive mode and contact the administrator via e-mail. This way, the site will be safer from hacking attempts before the patch is installed, the admin will be able to back up the site before an update, and the settings file can remain read-only in the meantime.

The downside would be that the site would be offline in the interim, but the option could be disabled if this is undesirable.

tsphethean’s picture

Unfortunately an issue like SA-CORE-2014-005 makes the site vulnerable whether in maintenance mode or not, so secure maintenance mode would need to become something which changed webserver configuration to point all requests to a static HTML file or renames/removes index.php

effulgentsia’s picture

If we can get automatic update/patching working, that would be great. But that also seems hard: for example, will the patch apply/work for every release that's in the wild?

So +1 to also considering the suggestions in #9/#40: a kill switch during very early bootstrap. Perhaps even doing that as a phase 1 of this issue, and then when automatic updates is implemented, have it supersede the killswitch.

Many site administrators might really not appreciate having Drupal take down their site for them, even temporarily

#19 mentions giving administrators some advanced warning of a highly critical SA coming. So, we could build something into settings.php that can kill the killswitch (e.g., $disable_kill_switch['SA-CORE-2014-005'] = TRUE). In other words, the administrator can either do that ahead of time, if she is confident she'll be ready to apply the update when it comes out. Or she can allow the killswitch to fire and then apply the update to get the site up again. But if an administrator takes no action at all, then I think having the site be down is far preferable to having it hacked.

Creating a system which is intended to take control away from the Drupal users seems to me to be an anathema to the OSS movement...I feel that the farther we can stay from a top down system, the better off the project will be

A consequence of OSS is that the code that fixes a security vulnerability is public information, which basically means that how to exploit it also becomes public information. I think that shutting off a site between the time that such information is public and when the site has incorporated the fix is a more responsible thing to do than the status quo of disclosing the vulnerability and then just letting sites be sitting targets.

timodwhit’s picture

Will the patch apply/work for every release that's in the wild?

This would seem unrealistic because who knows what versions people are running. I would say this makes a strong case for running stable versions, but if the system is "smart enough" we could build in a check for a stable vs dev release, and if the patch doens't apply because of a core hack, use the kill switch method.

I agree with both sides that the the kill switch seems too invasive for OSS, but also seems necessary because we are running real world sites where people don't notice all the security releases/patches etc. If sites we're killed early enough, then it would get their attention and patches could be applied. This might make some developers happy because now they are "guaranteed" to get paid for on going maintenance. :-)

So, we could build something into settings.php that can kill the killswitch (e.g., $disable_kill_switch['SA-CORE-2014-005'] = TRUE)

How long does this flag need to be in your settings.php? Would the killswitch look to see if the patch is applied and then the killswitch isn't needed? I would be hesitant of this because it can muck up a settings.php quickly depending on implementation.

But then the issue is how does drupal know the patch is applied? With different versions, different ways of patching, etc.

chx’s picture

If we go with the killswitch, where do you draw the line? If a sechole applies to everyone who install standard but not to those who installed minimal (say, help module) would we trigger the killswitch? What about node.module?

chx’s picture

> I would be hesitant of this because it can muck up a settings.php quickly depending on implementation.

Quickly???? This was the first such sechole since SA-2005-003! One line every nine years, let's hope.

Bevan’s picture

I think any sort of kill-switch solution is very annoying. Further, it might be useless; Drupageddon attacks were able to exploit Drupal websites in maintenance mode, for example.

Something as severe as Drupageddon (25/25) is unlikely to recur (but might). I still think it is probably a once-in-a-lifetime or once-per-decade sort of event. Severe security holes that we should plan for are more likely to be in the 10-20 range.

See also https://www.drupal.org/security-team/risk-levels

8thom’s picture

I agree that it would be very annoying, but that's the point.

Unfortunately, there will still be a large number of vulnerable sites out there and site owners still oblivious to the fact that a highly critical patch has been released.
These sites will still be collecting private information from unsuspecting users and some of these will be ecommerce sites collecting credit card info etc.
This might be a "once in a lifetime" or "once per decade" event but how else how we get though to these site owners? I hate to think but it's likely that some of these compromised sites will still be active 12 months or more from now..

With approx 1 million D7 sites, ironically not including sites which have disabled the update module to "fix" the annoying security messages, I think we have a responsibility to limit the impact of very critical security issues.

The "kill switch" could be an optional setting of the update module and would be extra awesome if it had another level to only disable logins for everyone (except user 1) in cases where the highly critical vulnerability is only exploitable by an authenticated user.

webchick’s picture

I had the chance to sit down with Andrew Nacin from WordPress at php[world] last week. He says one of the ways in which their auto-updater mitigates damage is if it detects a .git, .bzr, .svn, etc. root anywhere in the code base, it will just automatically skip auto-updates, figuring you must know what you are doing.

I don't think that's a fool-proof method—it's a fairly good indicator that the website developer knew what they were doing, but not necessarily the website maintainer, who may have taken over after the developer is long gone—but could be one approach.

greggles’s picture

On kill switches: There are certainly kinds of kill switches that would "work", e.g. removing index.php or altering the core .htaccess to make it "Deny from All" or similar.

re #47: I think we should use a $conf['automatic_self_updates'] in settings.php to determine whether or not to do updates. There could be a hook_requirements that looks for .git, .bzr, .svn etc. and makes a suggestion to change the $conf value if it's set to make automatic updates and it finds version control is present.

timodwhit’s picture

++ For the $conf['automatic_self_updates'] variable

However, I don't know if that approach of "if there is a version control" is that good. It seems like with a VCS, you would be able to see your changes apply them upstream and then merge other code with it, or scrap the change and manually apply where needed in a branch etc. It would be a tracking method for the changes.

The var would be the place for the developer to say I am caring for this site for now and I will apply the updates or I'm handing this over to a maintainer and thus it needs to be automatically update. Thus, negating the need for the check of the .git, .bzr, .svn etc.

pwolanin’s picture

Unless we get in the business of managing signed release tarballs, which means distributing the public key and protecting that somehow, and figure out how to validate the signature, any self-updating is totally a no go.

In any case, having the mechanism is a natural attack vector.

I think instead we need to auto-configure update module on install to email the site address for every security issue. It's foolish to me that we don't do that. We could have an "alert" mode maybe where we trigger an email to all admins? Maybe built into core and not something you can disable?

greggles’s picture

In this issue there's been some great discussion around a pretty great idea of doing pre-announcements inside of Drupal sites. I filed a new issue to keep track of that conversation since it seems like a somewhat separate topic #2378513: Create system for distributing pre-announcements of extreme security issues in core/contrib.

rickmanelius’s picture

This may be a naive question, but would this have to be in core? Given the discussion thus far, there may be multiple valid approaches towards addressing this issue, each with their own advantages and disadvantages. This type of experimentation should (assuming it's possible) be easier in contrib yes/no? And once 1-2 methods gain traction with some real world data and experiences, then it might make sense to talk about inclusion in core.

Stagger Lee’s picture

I vote for automatic security updates.

- Never made me problems in my WordPress websites.
- Will additionally teach people not to touch core files (as it teached me to change all aspects of WordPress websites without touching core/plugins code).
- If, only if...Automatic security update crashes website it is 90% easier to repair it than to clean all backdoors after hacker. And find what they inplanted in database, clean database.
- Will give a lesson to hackers to even stop trying to hack Drupal, pointless and hard time demanding.
- End some more, dont remember right now.

We should never forget that we here discussing very small code line changes, not real "updates", "upgrades" as we picture it often.

And last, if it is very high resistance against it, implement it and set it deactivated by default. And give us one line od code to activate it manually in main config file. A little bit oposite than in Wordpress, but at least we have a choice.

dblais’s picture

(sorry for my english)

#1
I think Drupal should have a exploit scanner installer on the website. As a proxy every request (before bootstap or anything else) pass inside this script witch check a database (file) on Drupal site IP is automaticly blacklisted or request is ignored. No need to webserver to have write access to all files...

#2
It can be also something like mod_security database for Drupal where Drupal.org create mod_security rules base on known exploit. Like https://www3.trustwave.com/modsecurity-rules-support.php or https://waf.comodo.com/. No need to webserver to have write access to all files...

#3
It's can also be a proactive scanner/patcher where Highly Critical 25/25 (like SA-CORE-2014-005) will automaticly patched a website who choose this option. Again base on a remote database with patern like mod_security.

Conclusion
I would like to have a good hacker to hack my website to apply the patch friendly... Or Drupal script/module can at least modify my htaccess to disable my website, send me a email asking to patch/update now because my website is down. Call that the Turtle effect ;). Me too i prefer a website down and clean...

Every website/client are different, but we must find a way to secure small website...

MarkU’s picture

TurnkeyLinux also offers an opt-in auto-update feature for security updates. There may be things you can learn from their approach

bofrost’s picture

The discussion about automatic security updates is older than 6 years and nothing happens.
There are many good points for pro and contra of this topic.
I believe there will be no one-fits-all solution for the problem, but we developer should take care about non-technical users or non-professionals.

So, I decide to develop a simple module (https://www.drupal.org/project/cms_updater) and web service to automate Drupal 7 updates.
( We will see if this is necessary for Drupal 8 )
This is definitely not aimed for enterprise Drupal sites but for small projects it is very useful.

However, people can decide if they use this service or not – having a Drupal secure or not.

raj45’s picture

Thanks for sharing your module @bofrost. I am considering automatic updates via cron, as outlined in Spend less time updating Drupal websites, look under "Use a shell script and crontab for core security updates".

Basically, it's just a cron-triggered shell script which fires off the command "drush -y up drupal --security-only". Simple, yet gets the job done. Backups are necessary, of course, if the web site for some reason gets hosed in the process. :-)

vegardx’s picture

I would think twice about running that in cron, as you have no way of knowing if drush exited with a non-zero value. If you manage a lot of Drupal sites you can take a look at Jenkins and make that run the update process.

Ideally you have a build process where you update the staging env first then, if it succeeds, deploy it to production. Possibly also add some sanity checks just to be sure that things are sort of working at least. You could run these tasks periodically, but I don't see why you'd want that. Personally I like to assess all the security updates before applying them.

As for preventing root kits there are lots that can be done in the way Drupal executes php, but I'm sure most of that is addressed in D8. Most root kits just dumps a bunch of files somewhere in your installation, accessing them directly. Making them have to patch the core in order to do things like that can be easily detected and removed.

raj45’s picture

Thanks for sharing your thoughts on the subject @vegardx.

Personally I like to assess all the security updates before applying them.

I agree. To only apply really important updates it would be great if there was an additional parameter like drush -y up drupal --security-only --security-risk-above=15, to filter out minor security updates, and only apply them if the level is above (like in this example) 15.

pwolanin’s picture

Category: Task » Feature request
Priority: Major » Normal

I personally think this is not in scope for drupal core, but there are ideas in this thread that should be documented as suggested practices in a better location. For eample s the drush update on cron from the article linked in #57 should be safe for most small sites and would mean you get the security updates automatically.

manuelBS’s picture

Is there already a "machine readable" way to get information about the "criticality" of a security update? Currently it's only possible to parse the body of a release log node on Drupal.org. This could help Drush to only apply the highly criticals. For a more controlled update process you are welcome join the Drop Guard beta for free at http://drop-guard.net/beta-form.html

catch’s picture

Version: 8.0.x-dev » 8.1.x-dev

Moving this to 8.1.x.

pwolanin’s picture

The RSS feed doesn't appear to be all that useful in terms of getting the level: https://www.drupal.org/security/rss.xml

We probably need a SA content type and specific fields to actually make that machine readable, or perhaps we could find a way to add that to the release node fields.

Andy Inman’s picture

@manuelBS

Is there already a "machine readable" way to get information about the "criticality" of a security update?

I've often thought that would be useful. But, I don't think it would be enough. Many, many security vulnerabilities are limited to authenticated users, often with specific permissions. And on many sites, all authenticated users are trusted users (internal staff, etc). So, in order for a script to determine if a security update is urgent, it would need to identify which users could exploit a vulnerability AND whether any of those users were untrusted. That would require not just the "criticality" of a vulnerability but also which specific permissions and conditions are required to exploit it.

One idea: For all security updates, create a PHP script which tests whether the vulnerability actually exists. Then, loop through all untrusted users (and anonymous) in masquerade fashion running the test. The test itself would need to be automatically downloaded. It would have to be automatically re-run every time site configuration changed or a user was created or updated, and so on. Then there's the question: who would write the tests?

The concept of "trusted" and "untrusted" users could be expanded to "totally trusted", "partially trusted" and so on. Clearly site admins would have to assign a "trust status" to each user, maybe using roles or a field in the user record.

Anyway, just my ideas.

masipila’s picture

One idea: For all security updates, create a PHP script which tests whether the vulnerability actually exists.

Did I understand correctly that you are suggesting that we would start to publish proof-of-concept-exploits at the same time as the security updates are published?

Markus

Andy Inman’s picture

@raj45

Basically, it's just a cron-triggered shell script which fires off the command "drush -y up drupal --security-only".

I've recently implemented exactly that, prior to finding this thread. In fact, I found this thread because I was searching for how to get drush to test if security updates are available without actually installing them. Then, my plan is to run a full backup if security updates are available, then install the updates. I am thinking drush -n up --security-only" may work - it seems to set errorlevel 0 when there are no updates, and 75 when there are updates. Maybe that 75 means "cancelled" (due to the -n parameter) but either way, it seems that non-zero errorlevel means that updates are available.

So:

#!/bin/bash
drush -n up --security-only
if [ $? = 0 ]; then
  echo "No updates"
else
  drush -y ard
  drush -y up --security-only
fi

... any thoughts?

Andy Inman’s picture

@masipila

Did I understand correctly that you are suggesting that we would start to publish proof-of-concept-exploits at the same time as the security updates are published?

If the attacker were able to run the test PHP script, then the site already has a big problem! The test script would be run by a module, not publicly accessible. An attacker could download the script, but the information there is basically the same as what's in the security announcement ("mitigated by", etc, etc) just in PHP form.

masipila’s picture

Okay, now I got what you meant. I first thought that you were suggesting that we would have a test that actually tests the vulnerability itself (i.e. if the test would be publicly available it would actually a be POC for exploit). But I guess you meant that the test would test if the site has a combination of permissions and other exposing factors that are listed in the security advisory.

Andy Inman’s picture

@masipila

Yep. And for specific users/roles, because as I said, on many sites, all authenticated users are trusted, so even for a "critical" vulnerability, if it requires logging in then it becomes relatively low priority to patch/upgrade.

Stagger Lee’s picture

@netgenius, yo dont need any script or test. Core version number is test. It is, or it is not.

I am still confuse how can it work perfectly for WordPress but for Drupal all have to be so complicated. But at the end number of installations says much.

WisTex’s picture

An ideal way to set it up would be:

  • Automatic updates can be turned on or off by the admin.
  • Admin can pick whether they want automatic installation of security updates only or core+security updates.
  • Admins should be able to manually update the core and add security patches from within Drupal (similar to WordPress)

This way automatic updates are optional, but can easily be turned on.

Having updates from the admin, with the option for automatic updates, would be ideal for:

  1. Non-Programmers technical enough to admin Drupal.
  2. Programmers who have dozens or hundreds of installs to manage and want an easy way to update their websites.
pwolanin’s picture

I think the summary needs to be updated to reflect the discussion and possible things to be done.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Michael Z Freeman’s picture

The issue seems to be one of the nature of Drupal which is often used for business sites that hold sensitive customer details; hence the strict security advice concerning file and directory permissions. However casual and other sites don't usually hold the same sensitive information. Maybe an autoupdate function should simply be provided as an optional module, then core Drupal development philosophy is left alone. People who use the optional module can override recommended security settings after agreeing to a disclaimer.

catch’s picture

Had another idea recently.

- it makes sense to have separate issues for security updates vs. automatic updates of everything, since the implications for the latter are far, far wider.
- a big sticking point here is having Drupal write to either core or module files directly, since most sensible configurations should not allow that. This means we either require command line cron, or we require web-writable PHP.

However, since this discussion started, we have PHPStorage.

It might be feasible to do something like the following. This is a very rough thought, not fully thought out so please pick holes:

1. Have an auto-updater run on cron
2. In the case of a highly critical security release, have the auto-updater download a complete version of a file and write it via phpstorage.
3. Have a line in settings.php, which includes another file, also in phpstorage
4. This file would swap the new file we've downloaded in the autoloader, leaving the old file where it is, that way the only writable directory is phpstorage which is writable anyway
5. The auto-updater could then monitor for the original file being updated correctly, and remove the reference to the replacement when that's done.
6. It'd be possible to provide notifications about all of this too of course.
7. We could also go a step further to prevent against hijacks, and while we automatically download the file, send a notification when this is done, then require a confirmation form to swap it in the autoloader showing a diff between the files alongside a checksum etc.. Or there could be a sliding scale we make configurable between full automation (but less protection against hijacks) vs. a manual step (but naturally slower updates if you don't respond to the notification quickly).

xjm’s picture

xjm’s picture

Priority: Normal » Major

I'd definitely consider this at least a major feature request.

pwolanin’s picture

In general, we expect sites to be kept p to date, so I'm not sure there is a realistic scenario where we can reliably provide a fix when core is generally of an unknown version.

We also alternate security and bugfix release, so you can't really afford to stay behind.

There are also scenarios that are very hard to automate including changes to settings.php, services.yml, .htaccess, and possibly composer.json

I think a possibly viable MVP for this is a script in core/scripts that can detect core updates and install a new core version. Probably it would have to leave files outside of core/ and vensdor/ alone?

This script should come with docs on setting it up securely, and could e.g. cause a warning to appear in the status page if not run daily (with a setting to toggle off?)

JurriaanRoelofs’s picture

I think this would be a great feature to improve the security of Drupal and protect websites of clients who do not understand that risk of a site that's behind on updating.

In general, we expect sites to be kept p to date

I think instead we need to auto-configure update module on install to email the site address for every security issue.

@pwolanin I think you overstimate how much most Drupal sitebuilders care about their site's security and their clients' sites security.

Here are some Drupal 7 statistics on installed updates:

62% of Drupal 7 sites are not updated to Drupal 7.43 yet, which is 3 months old and brings Critical security updates.
That means at least 620.000 Drupal 7 sites are probably hackable right now. This does not include websites that turned off the update module to stop those "annoying emails".

18% of Drupal 7 sites are on version 7.31 or lower, meaning that they have not been updated for nearly 2 years and are probably vulnerable to the Drupalgeddon attack

@see https://www.drupal.org/project/usage/drupal

colan’s picture

...and here's another recent one: Crooks Used SQL Injections to Hack Drupal Sites and Install Fake Ransomware.

I like what @catch is saying in #75.

mbaynton’s picture

I'm glad to see this being discussed now in the Drupal community! I have started advocating and considering it in the context of the Backdrop CMS project: http://eworldproblems.mbaynton.com/2016/07/automatic-security-updates-an.... We'll be discussing it shortly on their Google hangout, I think.

One quick reaction I have to the proposal as it currently stands: if you think like an attacker, when the next SA-CORE-2014-005 is disclosed, you're going to do these two things, in order:

  1. Program your botnet to brute force DDOS-attack this central server of updates on drupal.org infrastructure -- it's got a huge target painted on it. Plus, you can prep this part ahead of time, you don't even need to write the exploit code. It'd probably start getting attacked within minutes of a highly critical security disclosure.
  2. Then, take the time to program your botnet to exploit the vulnerability.

The model for update distribution that I've been thinking of turns it around, so that the update distribution infrastructure pushes to sites instead of sites hopefully running cron soon enough and then hopefully being able to contact a well-known update hostname that hasn't been DDOS'ed. Specifically:

  1. Sites opting into automatic updates actively enroll by contacting a central web service on the CMS's infrastructure. Each site then registers a well-known endpoint, say /admin/automatic-update. This path accepts digitally signed (gnupg looks like a good choice) messages.
  2. Updates are signed far, far away from even the distribution servers, potentially using a private key on an airgap machine. Once signed, the update "packages" are dumb static resources.
  3. The distribution servers are loaded with the update package and a portion of the overall database of registered sites. Each distribution server simply POSTs the static update packages to its share of the database. These servers need not accept connections on any ports and would not be at well-known addresses.
  4. As an additional safeguard, upon verifying a signed message received at the endpoint, sites could perform forward-confirmed reverse DNS verification that the server the message was received from is owned by the CMS's official infrastructure. This shouldn't be strictly necessary, but would be a substantial speedbump should an attacker somehow obtain the signing key: you'd still need to compromise the official distribution infrastructure as well to get your signed-but-malicious updates disseminated very widely.

This leaves out some details re: how to keep the database of registered sites from being flooded with false entries -- I'd limit to 1 or 2 subdomains per domain, charging money if more are needed, periodically verify endpoints remain valid, and register using a form with a captcha.

mbaynton’s picture

The issue calls for a review of how WordPress is doing it, which seems sensible, but I didn't see one yet. Some initial findings from a review of WordPress's auto-update code:

  1. Basic distribution model is for sites to connect to api.wordpress.org (see code) and downloads.wordpress.org during a cron run to check for, download, and deploy core updates.
  2. https is used when it is available to php, but the updated code will be downloaded and deployed anyway if not.
  3. The call to api.wordpress.org is used to determine what updates are available given the current WP version, PHP version, MySQL version etc. Depending on the transport used with api.wordpress.org, the URLs to updated code may be http or https. Example requests:
  4. WordPress publishes hashes of their update packages, and the built-in updater is capable of verifying downloaded packages against them, but it looks like verification doesn't occur as part of automatic updates. That makes some sense in that if you have https, the checksums don't gain you much, and if you don't have https, you can't download trusted hashes.

I personally still favor a model where signed updates are pushed as I described above, but the main takeaway for me is that the less rigorous security measures we see in WP don't seem to be actually causing problems in the wild. Sites that have https available are able to benefit from it, but updates will still be pulled down and installed totally unverified if not. Perhaps the correct question to ask is not "is the automatic update mechanism totally bulletproof & locked down like Fort Knox" but rather "is the risk of unpatched vulnerabilities in the CMS greater than the risk introduced by the automatic update mechanism?"

I'd suggest Drupal take a similar approach: support technologies like code signing and https for installations where they are available, but allow automatic updates to happen anyway even if no asymmetric-key based verification of the update is available. An inability to verify automatic updates would be a good item to call out on the site status report, though.

Edit 9/9/17: You can track WP's pursuit of a better solution in their similar issue on wordpress.org.

8thom’s picture

Right now I'd love some sites to just go down for a few hours rather than deal with the security consequences...

Even if this was opt in...

https://www.drupal.org/node/2764899
https://www.drupal.org/node/2367319#comment-9302861

greggles’s picture

@8thom you can add to your webserver confirmation something to block requests for all IPs except your own, e.g. for Apache this can go in the .htaccess or other configuration file:

order deny,allow
deny from all
allow from 127.0.0.1

For finding your public IP, I like this simple service http://checkip.amazonaws.com/

mbaynton’s picture

@8thom, as a thought experiment let's say taking sites down was the goal, rather than just fixing them. Since this is a rather extreme action, you would want the thing that automatically takes your site down to be very sure its getting authentic instructions, thus opening basically the same can of worms faced by securely distributing the fix. Also the way you would truly take the site down would be ie a .htaccess modification, so you need the means to write to the source tree too. It may seem like an easier thing to achieve, but it isn't.

Your suggestion might be a reasonable choice sites could opt-in to in the event that an automatically pushed update failed to apply, however... "If a highly critical security update fails to apply:
[] roll the update back
[] take the site down"

Kudos to the security team for the heads-up yesterday, btw. Much appreciated.

8thom’s picture

@greggles we can easily switch it off - I'm talking about no interaction with the server, just fix it up tomorrow (1am here in Australia)

ressa’s picture

I wonder if the Shield module could be of help to people on shared servers, without SSH access? https://www.drupal.org/project/shield

8thom’s picture

@mbaynton I think it could be very conservative and only disable the site if file is found or key is returned from update server...

Could this be in contrib?

Worst case, someone gets in a adds a file to disable your site :)

greggles’s picture

I wonder if the Shield module could be of help to people on shared servers

Drupal based solutions at blocking access require Drupal to be running - for some (many?) vulnerabilities that's a sufficient attack surface to exploit them.

ressa’s picture

Thanks @greggles, better use .htacces then :-)

8thom’s picture

lets be honest some of these sites aren't "enterprise" and might not have full time devs but a security breach could still kill their biz..

Stagger Lee’s picture

Whole this discussion (or most of it) is proof why Drupal community is doomed. Not even with Drupal 8 you have slightly chance to come to number of Joomla installations. Wont mention WordPress, it is Science Fiction. Cant believe what I am reading.

mbaynton’s picture

I'm launching a project to prototype the Drupal end of an automatic update system. As this feature request has shown, there are countless opinions and possible approaches for how to go about such a thing. I've tried to collect all the observations made here into essentially an updated summary, and for the last week or so have been attempting to synthesize those concerns into a technical design that attempts to best address them.

The document capturing all this is in a github repo. I apologize if going off d.o for such work is discouraged, but we're approaching comment 100 here and I think a git PR process might be helpful to keep this conversation constructive and focused. That said I definitely encourage any PRs or comments to further evolve the document in this repository, including radical changes, if you know a way that will better address the design concerns we've collected.

If a viable technical implementation emerges from this work, I hope we can have a discussion about how it could be used officially.

cilefen’s picture

mbaynton’s picture

@cilefen, not the way I'm proposing to do it. Updates would be pushed to sites, not pulled via cron, so the site's view of available updates and/or ability to validate that data wouldn't matter. (My reasoning for turning it around like this is explained in #81 and my github document.)

cilefen’s picture

@mbaynton I only meant that in the case of the traditional update module.

alexpott’s picture

sarciszewski’s picture

I work for Paragon Initiative Enterprises, which specializes in crypto and PHP security, and actually found this discussion because Alex Pott linked to one of our blog posts. :)

Cryptography is exceedingly difficult to get right. Even experts make mistakes, and peer review is very important.

To that end, I've been commenting on the Github repository created by Mark Baynton to standardize a solution to this problem.

I'd like to propose the MIT-licensed EasyRSA library to solve some of the challenges that Drupal faces, especially with shared hosting providers that prevent self-updaters from writing to the filesystem.

My comment explains some of the particular use-cases of EasyRSA to solve these problems. The most straightforward is $signature = EasyRSA::sign($fileContents, $privateKey) and a congruent EasyRSA::verify($fileContents, $signature, $publicKey) which will return TRUE if everything is fine.

I'd like to state the case for using my company's library, and I'm not just suggesting it because I wrote it. If Drupal could get away with requiring libsodium, you should unambiguously prefer that over EasyRSA any day of the week. (We did write a user-friendly libsodium wrapper, Halite, too, but I'm fairly sure that's not an easy pill for anyone to swallow today.)

First, RSA has a lot of trapdoors for implementors. Some historical examples:

  • SaltStack managed to deploy e = d = 1 RSA, which provides zero security. (Despite this critically bad mistake once upon a time, Salt Stack shouldn't be viewed as incompetent.) RSA makes this kind of blunder possible.
  • Zend\Crypt, which used the OpenSSL extension, was vulnerable to a padding oracle attack, which allows you to send modified ciphertexts repeatedly and, based on the presence or absence of a padding error, decrypt the original message. Our team disclosed this in November 2015, so it's not exactly ancient history compared to the original vulnerability, which has been public knowledge since a 1998 paper by Daniel Bleichenbacher.
  • Several libraries naively encrypt large messages with RSA directly. This isn't actually possible, so they do the obvious thing: Break their messages into chunks, then encrypt each chunk separately. See this slide for example. This allows attackers to rearrange each chunk at their leisure which, combined with the above padding oracle attack, can be used for all kinds of cryptographic fun (from an attacker's perspective).

RSA has a lot of levers and knobs to adjust (far more than just "are we using a 2048-bit key or a 4096-bit key?"), and unless you have years of experience in software security, the deck is stacked against you getting it right the first time. (I sure didn't!)

Enter EasyRSA.

First, EasyRSA uses e = 65537 (because that's what phpseclib uses), which means we've dodged the first bullet.

Second, for encryption, EasyRSA is hard-coded to use a padding mode called OAEP (Optimal Asymmetric Encryption Padding), with MGF1-SHA256 (Mask Generation Function #1, based on SHA256). The padding oracle attacks that plagued PKCS1v1.5 padding RSA (which is, once again, the ext/openssl default) are once again side-stepped. For signatures, we use a mode called Probabilistic Signature Scheme, which isn't vulnerable to a different Bleichenbacher attack on PKCS1v1.5 padding that affects digital signatures rather than encryption.

Finally, we don't use RSA encryption directly, because that's perilous and there's a better way forward. This is what we do, instead:

  1. Generate a random 256-bit key.
  2. Encrypt your message with defuse/php-encryption (which uses AES-CTR-256 + HMAC-SHA-256 in an Encrypt-then-MAC construction).
  3. Encrypt the random key in step 1 with the RSA public key.

In short, if you're forced to use RSA (as opposed to EdDSA or RFC 6979 ECDSA), you'll be hard-pressed to exceed the security of EasyRSA for both public-key encryption and public-key digital signatures. We strived for a simple interface, to boot.

EasyRSA can be used for both signing automatic updates and for allowing Drupal site operators to encrypt their SFTP credentials for a benign bot to apply the update for them, should PHP scripts writing to the filesystem be forbidden by their web host.

I know Drupal has security experts on their team (or at least that they talk with). Please loop them in on this conversation and ask them to critique what I've described above. If they're up to it, everyone should feel free to poke any holes they can in EasyRSA.

If this suggestion deemed to be acceptable, the cryptography part of the problem is essentially solved, and you can focus on the hard parts: Key management and the auto updating process itself.

mbaynton’s picture

An employee from Paragon Initiative Enterprises, the company that published the above blog post, has reached out to me today. He's provided some initial suggestions wrt. crypto algorithms and referenced a number of useful implementations, but also has extended an offer to provide some level of code review, possibly thorough, depending on time available if we have near-finalized code that is a candidate for Drupal core. I think such offers and advice from professional cryptographic consultants/code auditors is very valuable and so want to make sure others are also aware of it, in case someone else ends up being the one with candidate code.

I'll be continuing to review his suggestions and updating my design proposal appropriately in coming days.

^^ edit: oh, he beat me to a post by 8 minutes :)

mbaynton’s picture

I know Drupal has security experts on their team (or at least that they talk with). Please loop them in on this conversation and ask them to critique what I've described above. If they're up to it, everyone should feel free to poke any holes they can in EasyRSA.

Yes, it would be great to get some of those voices into this conversation. This is still really just a feature request, though. The question of whether Drupal should even offer automatic updates lacks consensus, and there are those who seem against it. Before a lot more software developing and crypto scrutinizing is applied to the problem, the community should adopt a formal position on the issue: a go/no-go decision on automatic updates. As that consensus hasn't developed by comment 100 of a 2 year old feature request, I propose a core conversation be held in Dublin for the purpose.

I strongly suspect the entire thing could technically be done in contrib (and I've got the beginnings of two modules to collectively achieve it as of this weekend), but contrib is not the way to go here. The cryptography should be reviewed and scrutinized by experts that a contrib project isn't likely to receive, and the updates themselves should have the security team's stamp (and digital signature) of approval.

Bojhan’s picture

@mbaynton I think its save to assume, if its a community - you will not get a formal go/no from the whole :) Product managers should be able to provide this, but given the significant technical challenges - exploring it a bit more before asking formal commitment might be worthwhile.

webchick’s picture

Well. This is the kind of change that's multifaceted and would need sign-off from product managers, framework managers, as well as the Security Working Group. FWIW though as a product manager, I'm definitely, hugely +150 for this feature conceptually! But there are a lot of technical details to be sorted out about it working in practice without causing more damage than it solves. Maybe one place to start would be to go through the thread in detail and enumerate the pro/con arguments in the issue summary so that we know what the roadmap looks like for addressing peoples' concerns?

sarciszewski’s picture

If anyone has any cryptography questions related to this issue, feel free to ping us on Twitter. That's usually the fastest way to get our attention. @ParagonIE

mbaynton’s picture

Assigned: Unassigned » mbaynton

I'm going to assign this to me long enough to properly update the issue summary at @webchick's request. That's a doozy of a task.

Stagger Lee’s picture

You will never go long looking for "totally bulletproof" solution.

Regardless of core updates problem is modules update. I never know what I am updating and changes are burried somewhere on module subsite, issues or forum. In WordPress you get notification about update, click one link and modal popup tells you what is changed.

WordPress is more disciplined community than Drupal, despite nobody from Drupal community will admit this.

Stagger Lee’s picture

Would one Diff checker implemented inside modules backend page be enough ? For core code to start with.
Preview changes in new version, if OK proceed with Apply changes. With option "Apply All". Optional, not to be activated by default. Can be activated with one line in conf file.

mbaynton’s picture

But there are a lot of technical details to be sorted out about it working in practice without causing more damage than it solves. Maybe one place to start would be to go through the thread in detail and enumerate the pro/con arguments in the issue summary so that we know what the roadmap looks like for addressing peoples' concerns?

On it -- the suggestion was pretty clearly directed at me ;) It is essentially a complete rewrite, but it hasn't been touched for two years, so it kind of had to be done. Now, I have my opinions about this feature request and this shows through in my inability to resist counter-arguments to the cons; if someone has a differing opinion you are welcome to do likewise to the pro's, if you can.

I think what is now gathered here covers about everything there could possibly ever be to say about the benefits and drawbacks of doing automatic updates. But I'm concerned that if things don't change, all that might happen is an ever-growing grab-bag of observations will nevertheless continue to collect in this feature request, showcasing to the world Drupal's lack of progress.

given the significant technical challenges - exploring it a bit more before asking formal commitment might be worthwhile.

Yes, more exploration of those challenges is necessary to implement it, but at this point it should not be necessary for the managers and groups @webchick mentions to look this over and decide whether or not they'll sign off on it. Forgive me but, as automatic updates have been done again and again in numerous software programs including very successful ones that closely match Drupal's system requirements, citing technical challenges as a reason we might not be able to proceed is not productive.

Don't misunderstand me, that doesn't mean I'm expecting advance commitment to a particular developer's implementation of it before it is a known quantity. But the problem is, as you can see, this feature request hasn't gone anywhere, and it sounds like the concept has been on the radar long before it was created even. Nobody is stepping up to do it.

What is missing is the commitment and direction I'm asking for from Drupal's leadership structure. Well, what is missing is volunteers to do it, but speaking for myself and if history is an indicator I think you're never going to get anyone willing to make the sacrifices necessary to do a really comprehensive job with such a multifaceted and challenging problem if they don't even have a reason to believe their efforts will bear fruit. I commend @bofrost wholeheartedly for cms_updater (see also #56), but it shows what happens if you attempt this without first gaining broad-based support. I think a few other recent posts have also, albeit more curtly, suggested the absence of leadership and direction as the missing ingredient that's stifling progress of this much-needed feature. Support ahead of time from everyone that could potentially throw large wrenches in the works down the line would go a long way to making development possible.

Perhaps it would be easier for the product managers and others necessary to agree to this endeavour conditionally, aka "I could support automatic updates, but only if...". Collecting that information would also be very helpful in informing the implementation strategy. Or alternatively, we could find out that automatic updates aren't compatible with Drupal, no matter the implementation. Finding that out now is the entire point. But hopefully my counter-arguments in the issue summary will help alleviate major fears...

This should be done by a working group, of a manageable-enough size that all the multitudes of decisions that will need to be made can be made efficiently, but not so small that the task is insurmountable or the Drupal leadership structure doubts its work product. Get me the backing I ask for and I'll be the first to sign up -- with it, I can see making the sacrifices necessary to really help with this, because I think it's important. And I'll be in a much better position to recruit others. Also, I think the consultation that's currently on offer from @sarciszewski adds momentum to this issue right now.

Now I don't honestly even know where to begin or who to talk to in order to get the ball rolling myself. (By the way, a listing of these product managers...really not easy to find. Is that by design?) But @webchick, I'm sure you do, and I'm glad to hear you are for this feature. So what do you say? Can you help me navigate the leadership structure and drum up the support needed to get this off the ground? Certainly others that may be lurking in the 80+ followers who are able to help would be much appreciated too.

catch’s picture

Now I don't honestly even know where to begin or who to talk to in order to get the ball rolling myself. (By the way, a listing of these product managers...really not easy to find. Is that by design?)

Product managers listed here: https://api.drupal.org/api/drupal/core!MAINTAINERS.txt/8.2.x (there are two and webchick is one of them).

daffie’s picture

@mbaynton: I know that you are frustrated with the speed that this issue have been going. The first problem is that if we do this and it goes wrong the consequences will be very big. The second problem is that any technical solution will not only be in Drupal core but also require a lot of work on the drupal.org side as well. Not only initially to set it up but also the solution also will be have to be maintained. The third problem is that the Drupal project has more or less become a very large organisation. And large organisations and big changes, which this one is, will take time. You shall have to be patient. :(
If we look at we look at what @webchick wants:

This is the kind of change that's multifaceted and would need sign-off from product managers, framework managers, as well as the Security Working Group.

So what we need is a sign-off from those people for the whole conceptional idea. And only the conceptional idea!
The step after that will be the worked out technical solution. And for that we also be needing a sign-off from all parties involved.
After the technical sign-off from all parties is it time to form a official initiative.

@webchick: The issue summary has been updated. Can you get product managers, framework managers and the Security Working Group to take a look at this issue? For a possible conceptional idea sign-off.

daffie’s picture

We have the conceptually sign-off from @webchick:

I'm definitely, hugely +150 for this feature conceptually!
catch’s picture

Issue tags: +Security
daffie’s picture

@catch: Can you give a conceptually idea sign-off for this issue? Then we would only need a conceptually idea sign-off from the Security Working Group. After that we can work on the technical plan.

daffie’s picture

catch’s picture

@daffie there's not yet a proposed architecture to sign-off on here, so not sure what framework manager sign-off would mean at this point.

I do think we should seriously explore this again - i.e. I agree in principle with the risk from unpatched sites in general being a more practical risk than the auto-update mechanism being compromised.

A major caveat for me is that (as with the existing authorize.php) there needs to be a way to 100% killswitch this for sites that don't want it.

daffie’s picture

@catch: Can you tell what you need for a architectural sign-off? How big does it need to be. What technical information/detail does it need to have. I think some good guidance is needed. A lot of people want to get this to move in the right direction and we need help on what and how to do it. This is just not about making a patch!

sarciszewski’s picture

there's not yet a proposed architecture to sign-off on here, so not sure what framework manager sign-off would mean at this point

See https://github.com/mbaynton/cms-autoupdate-design perhaps.

I built an open source CMS for my employer where "securely auto-updates" was a first class design decision to minimize contention. It still took us the better part of a year to get the first stable release out the door.

From what I follow, the architecture being worked towards is multi-faceted:

  • General case:
    • Sites will periodically check an API endpoint for updates, over HTTPS.
    • If an update is available, it will download the update file and a cryptographic signature. (RSA most likely.)
    • If the signature verification is successful, the update will be installed.
  • Edge cases:
    • If a site cannot self-update directly (files aren't writable because of web host security policies), there will be an optional place for site operators to provide SFTP credentials, which will be encrypted with a public key (using a hybrid encryption like EasyRSA provides). Then, during update rollout, a pool of short-lived benign bots will iterate through the sites that have opted in, decrypt the credentials for each one, and apply the update over SFTP.
    • Users have to be able to disable auto-updates, at which point they're on their own for security updates.

It may be worth planning to place this feature in a major release (i.e. Drupal 9 or Drupal 10), if the parties involved are concerned about breakage, but this should be a significant priority.

Remember these words?

You should proceed under the assumption that every Drupal 7 website was compromised unless updated or patched before Oct 15th, 11pm UTC, that is 7 hours after the announcement.

A secure self-updater can prevent that from ever repeating.

An insecure self-updater can also prevent that, but creates a significant possibility for the update infrastructure to be used as a mechanism for delivering trojans to all or some of your users.

This is just for Drupal core. If you're interested in going all out, by allowing extension/plugin/module (select appropriate verbage) developers to sign their own packages, you'll be essentially attempting to reinvent the work I did with Airship, but without the benefits of libsodium.

daffie’s picture

@sarciszewski: Knowing how things go in the Drupal community is that if we are going to do this we most likely will start with an expirimental module in which the updates are automatic installed. That can be fully automatic or on command through the user interface.
With the whole idea to see how it works reality and how users respond to the functionality. It is still new and different then we are used to. Also we can get all the bugs out of the system. Good for learning what can go wrong.

Mixologic’s picture

you'll be essentially attempting to reinvent the work I did with Airship, but without the benefits of libsodium

I dont think we would want to do any re-inventing. Its why we got off the island in the first place. If there exists an implementation that has already had the blessing/design chops of security experts we should look at that first as our starting point.

I would also point out that the Drupal Association has some ideas around the updates system as well - what we would like to see is a comprehensive re-architecture to make it more efficient and give us a better data and insight into the sites using Drupal - optional automatic updates seem like they would be a good part of that as well. We'll probably open another plan issue with those ideas soon - and I'll post the issue in a comment here so we can get all your eyeballs on it. Im 100% behind the idea that we can improve the drupal product and project, and there is a lot of value in focusing on this feature.

It would be awesome if we could get a number of interested parties together for a BOF/Mind Meld in Dublin.

Also is there a tag for "Needs DA review?" -We'd love to get involved early when it comes to planning the nature of services we're going to provide.

David_Rothstein’s picture

I like the work that has been going into this recently.

As the person who filed this issue originally, I think it may be worth asking what the absolute minimal-viable-product would look like for this issue. (Doesn't necessarily mean that's what should come out of this in the end, but still worth asking.) I currently think it's this:

  1. A checkbox on drupal.org that the security team can check when editing a security release, e.g. "Trigger automatic updates for this release".
  2. The checkbox simply results in a new binary flag in the data provided by updates.drupal.org.
  3. On Drupal sites, there is an optional "automatic updates" feature as part of the Update Manager module.
  4. If that feature is turned on, then when Update Manager sees the above flag in the data for a particular project, and if the project is otherwise up-to-date i.e. only one release behind the security update (???), and if the docroot is writable, then it applies the automatic update immediately. (This would essentially be similar to what update_manager_update_form_submit() and update_manager_update_ready_form_submit() currently do, the latter in the "If the owner of the last directory we extracted is the same as the owner of our configuration directory" case, just with non-interactive batches that don't redirect to authorize.php.)
  5. The admininstrator is notified (e.g. via email if the existing Update Manager email address is configured) that an automatic update ran and what the results were.

Again, that's the absolute minimum, but it has the advantage that it probably wouldn't even be that complicated to implement. And it requires only minimal changes to drupal.org (which may be important given what the funding situation would likely look like for this feature... i.e. grim?)

Things that the above minimal viable product would leave out:

  • No automatic updates in the case where Update Manager requires FTP/SSH credentials (those are a lot more complicated and getting to be an edge case these days anyway).
  • No automatic updates for core (obviously - since that depends on #606592: Allow updating core with the update manager anyway).
  • No method for applying patches to people's sites - this is for releases only. (Although the patch workflow could be supported as a followup with minimal changes to drupal.org, just an optional "patch URL" field that the security team could fill out once "Trigger automatic updates for this release" is selected. It would need some robust code on the Drupal end to make it work there though.)

The other thing the above is missing is any security considerations. That's deliberate:

  • I'm partially guilty of leading this issue down the "it must be completely secure" path (via what I wrote in the original issue summary) but I'm not actually sure it's a requirement.
  • The thing is, we already have a click-to-immediately-update-your-site feature in the Drupal admin UI. This is in both Drupal 7 and 8 core already.
  • And even if you don't use that (but rather download updates from the Update Status screen manually, check them into Git, test them locally, etc) it's not that different of a risk in practice unless you're in the habit of reviewing updates line-by-line to look for hidden trojans in the code before you deploy them to production. Certainly most people don't have the knowledge or time to do that.
  • The reality is therefore that updates.drupal.org is already a potential big target, both for spoofing it and for actually taking it over (that's essentially what https://groups.drupal.org/node/506128 is all about).

So are we sure that introducing a "zero-click" update process makes it that much riskier?

(Obviously improving security via methods discussed in this issue is a good thing no matter what; my question is to what extent it's really a blocker for this feature, and if so, how much of a blocker. The very nice analysis of WordPress in #82 backs this up too; I haven't looked at that code myself but from the summary it sounds like the only security step they do that Drupal doesn't already is the try-https-but-then-fall-back-on-http method that's already in discussion at #1538118: Update status does not verify the identity or authenticity of the release history URL.)

mbaynton’s picture

There's not yet a proposed architecture to sign-off on here, so not sure what framework manager sign-off would mean at this point...A major caveat for me is that (as with the existing authorize.php) there needs to be a way to 100% killswitch this for sites that don't want it.

@catch, thank you. This could be done iteratively. With some quick and simple guidance like this from other product and framework managers and the security working group, proposing an architecture would be less of a crapshoot.

@daffie, thanks for picking this up and running with it.

if we are going to do this we most likely will start with an expirimental module

Yes, I'm thinking that would be the ideal way to ease into this as well.

You shall have to be patient. :(

I'm not trying to make it happen faster. I'm trying to make it happen.

@Mixlogic,

If there exists an implementation that has already had the blessing/design chops of security experts we should look at that first as our starting point.

Definitely. But all the implementation variations that hold much interest to me involve a higher level of theoretical security than WordPress' design strives to achieve, and the only implementations I'm aware of to date that do that are more or less verified by security expert singular. @sarciszewski has himself suggested that the most prudent course of action would be to get more peer review on those. As I've said before it'd be great to start making that happen, but it'll be easier to line up if this endeavor looks more like a going concern and less like a black hole. Incidentally, libsodium specifically isn't going to be the answer: it's not php.

catch’s picture

There's a PECL package for libsodium: https://pecl.php.net/package/libsodium

mpdonadio’s picture

I think the changes to the IS are good, but I think it really downplays the risks involved with solutions that require webserver write-access to code files. Doing this really expands the attack surface for site, and expands it beyond Drupal itself. There have been historical problems with the PHP GD library in the past being an attack vector to alter site files, and this happened recently with the ImageMagick problem. Within Drupal, it worsens the impact of remote code execution problems that may go undetected.

And as an aside for @mbaynton, all of the different roles and responsibilities for Drupal are described in https://www.drupal.org/governance

mbaynton’s picture

There's a PECL package for libsodium: https://pecl.php.net/package/libsodium

And the set of users who could get it to work could also cron 'drush up -y' or such. One of the must-haves in my mind is a solution that runs everywhere Drupal runs.

@David_Rothstein +1 to basically all of #119, but I'll post a few specific followups later on..

@mpdonadio please read the issue summary wrt. the whole webserver write-access argument.

mpdonadio’s picture

#12, @mpdonadio please read the issue summary wrt. the whole webserver write-access argument.

I did, and was the reason I posted that: I think the language about this is too soft. I think that any proposal should explicitly have a requirement that is write access by the webserver is not needed for all of the reasons outlined by others (and means I respectfully disagree with the MVP in #119, even after having worked (I think) on some of the Update Manager related issues that eventually led to this issue).

catch’s picture

And the set of users who could get it to work could also cron 'drush up -y' or such. One of the must-haves in my mind is a solution that runs everywhere Drupal runs.

That's not necessarily true. Several hosts now have Drupal-specific configurations, and might install this if they generally wanted users to be able to auto-update. Whether they do or not is a different question but something we could ask.

Also I said we could use it 'if available' - we have several subsystems in core which use the most optimal library if they can, then fall back to something else if it's not available.

David_Rothstein’s picture

I think that any proposal should explicitly have a requirement that is write access by the webserver is not needed for all of the reasons outlined by others (and means I respectfully disagree with the MVP in #119

#119 doesn't imply the webserver needs write access. Since this would be a flag in the data returned by updates.drupal.org, anything that uses that data could work with it, including Drush. So in that case Drush would need write access, not the webserver. (Several people pointed out above that running drush up --security-only on cron is an effective automatic update system already, but it will do it for all security updates. This issue would allow something like drush up --automatic-updates-only to be introduced which would only do it for serious security issues where an automatic update was triggered.)

In the original issue summary I think I had a paragraph about making sure this can work with Drush. We could add it back if that's helpful as a reminder. I do agree that this use case (where you want automatic updates without the webserver having write access) is an important one to support.

catch’s picture

We can also do this via non-drush cli cron, which while lots of shared hosting site's don't use it, cpanel etc. has the capability to run.

sarciszewski’s picture

Also I said we could use it 'if available'

You really want to choose "one true cryptography protocol" for signing/verifying updates and stick to it. Negotiating this is how downgrade attacks and cross-protocol attacks happen.

If you add libsodium via PECL, it needs to be a unilateral dependency.

If that's actually on the table, I +1 it.

catch’s picture

@sarciszewski requiring the libsodium PECL extension would undermine making the auto-update mechanism widely available.

So we either have to require something widely available, or support more than one crypto library.

Mixologic’s picture

Is there a converse to that statement? requiring libsodium in order to have automatic updates would potentially cause hosting providers to *add* libsodium to their offerings in order to compete. "Yeah, you can run drupal on X host, but then you cant have automatic updates, which are awesome. Try out Y host instead, they have it."

mbaynton’s picture

@catch

we have several subsystems in core which use the most optimal library if they can, then fall back to something else if it's not available.

Okay, well let's establish that libsodium is the most optimal library then. Was #125 intended to suggest it was?

Also, maybe libsodium is the best. If we provide a fallback that is also pretty good, perhaps based on phpseclib/EasyRSA, what percentage of people and/or hosting companies will want to bother with installing libsodium and the pecl extension? I'm worried the main thing we'd add by supporting a relatively obscure and incompatible library is scope and complexity. OpenSSL falling back to phpseclib/EasyRSA might make sense, because php is generally packaged with a dependency on OpenSSL by the popular linux distros, and phpseclib is OpenSSL-compatible so you shouldn't need to produce separately signed updates or negotiate any protocols. Phpseclib would also have the nice property of being updateable itself.

@David_Rothstein, reading over #119 more carefully now. I'm for a minimum viable product, because it would get things done. But only if we know ahead of time how to build on it to produce a final version that does all the things the MVP ignores. If you end up rewriting, there's a pretty good chance drupalgeddon 2.0 doesn't happen during the interval the MVP is in play, making it a waste of time. So I think the way to go for an MVP is to determine the design for the final product, and then strip things off of it. I'm not sure whether that's how you derived your suggested MVP or not?

No automatic updates for core (obviously - since that depends on #606592: Allow upgrading core with the upate manager anyway).

So #606592: Allow updating core with the update manager is an interesting issue. I just reviewed it expecting to find a long line of patches, bikeshedding and anarchy blocking a merge, but it seems it just hasn't really been picked up. That actually makes it look a lot more doable to me...but that said, if we make feature sacrifices as extensively as you proposed in #119 to achieve an MVP, you can probably also do without the full-featured ideal of an update manager that can upgrade core in the general case. All the challenges mentioned in that feature request's #4 aren't likely to be things that come up if you just need to overwrite a file or two.

I very much like your arguments for why we can maybe chill out a little bit about the security, but I would just note that where automatic updates exacerbate the security risk is in number of sites likely to be impacted should a nefarious update find its way to those servers. We're trying to get updates out by some definition of fast. Digital signatures on the things that can get intstalled automatically is a pretty straightforward way to fix that, especially if they're done as signed phars that the overwhelming majority of installed php interpreters will verify for you for free. To me it's worthwhile.

In the original issue summary I think I had a paragraph about making sure this can work with Drush.

I removed it because I was thinking it makes an architectural assumption about how updates will be distributed that hadn't really been hashed out yet. Drush will only ever be able to pull updates; in my github design I favor an exclusively push model where the update servers connect out to the sites. There's benefits and drawbacks to that approach though and I might be able to be talked down from it. But I guess, even if there were pushed updates, flagging the equivalent release as you describe and having drush learn how to install only those ones would be so easy it could be done too. Feel free to add that back to the issue summary if you want.

webchick’s picture

Already gave my sign-off on the concept; can't sign off on anything else until there are some proposed screenshots/mockups to review.

mbaynton’s picture

That issue review and summary rewrite didn't seem to do much.

One could reasonably accuse me of doing lots of talking and not a lot of doing on this issue. This is sort of true, my excuse being that my past attempts to contribute to Drupal, which have been much less ambitious, have largely not even received a serious review. Feel free to look at my issues. Wasting a few days here and there is one thing; this would be much more than that.

Nevertheless, I still think it is important to make this happen.

#606592: Allow updating core with the update manager was instrumental in setting my sights on a different way to approach this. I haven't even seen this way suggested before, and there have been plenty of ideas thrown out there, but I strongly believe it is the best way forward.

Rather than building the functionality that updates Drupal on top of Drupal, as a component with many dependencies on the thing it is supposed to update, I've been working on an small web application whose sole purpose is installing/updating the code of other applications. This approach immediately solves many of the technical challenges to updating core via a web UI noted in 606592, will provide a more reliable user experience (even if an update WSOD's every page of the main application, the installer app is fully self-contained and will still be there to let you fix it), and conveniently assuages my fears that lack of adoption by Drupal specifically would result in a colossal wasted effort. Also, in the long run, solving the hard problems of automatic updates once in a single and generally applicable project, rather than repeatedly in every CMS/forum/wiki ever, seems like it could yield benefits for all. So I am actually focusing on "doing" now.

The MVP for this project will just tackle user-attended core upgrades via web browser, and in lieu of the mockups @webchick now asks for I'll just demo that (and share a lot more details...github organization etc) once it's there -- we're still on week 1 :). A few things with working proof-of-concept level code already that I can say confidently, though, are

  • While it is a full, modern php web application inside (written on top of Silex), it will only require two files: a Phar archive containing the application, and a .php file that calls include on the .phar and provides some application integration, like telling the .phar the main application root directory and current version. The .phar produces no output and invokes very minimal internal code (just exports some classes for typed communication between the .php and the .phar) unless the .php tells it to start, so the entire thing has a killswitch. Thus, one way to easily package and distribute it is as a Drupal module.
  • It will use its own patch-based format for update packages. A collaborator is working on tooling to autogenerate update package patches from tags/references of git repositories. The patch application engine is yetanotherape/diff-match-patch.

So that's my update for now. More surely to come. And of course, if what I've described already interests you sufficiently that you want to get involved at this early stage, you are most welcome to help out -- just shoot me a message.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mpdonadio’s picture

Would it be possible to post your work here as a patch, or is the process going to be too involved? The main reason I am asking is to see if there is way we could reach a broader testing audience who can spin up an instance on https://simplytest.me/ w/ this patch and give it a go.

mbaynton’s picture

@mpdonadio, that's a good idea.

Sadly, all there is at present is code that provides services and code that tests the code, but no functional UI. A consequence of going with zero dependencies on Drupal is having to re-implement a number of things Drupal would give me, such as filesystem access interfaces and even a way to persist settings. My schedule allows an hour or two a day on average, so it's going to be a bit...but I'm working at it consistently. (Well, I did get off track and go and make https://prophusion.org, but I'm using it for the CI now so it sort of counts...)

I wonder if simplytest.me offers any means to modify code once the environment is created? I recall not being able to install additional modules via the Drupal admin UI.

mpdonadio’s picture

I pinged @patrickd to see if he thinks https://simplytest.me/ can support this.

patrickd’s picture

oh, looks like mpdonadio didn't see my reply :>

I didn't read through the whole issue so I'm not quite sure I understood everything..

You can modify files and install additional modules (via the D8 UI afaik) but only through Drupal / PHP itself.
There's currently no way to access the sandbox via FTP / or SSH. It is not possible to spawn another process through PHP on simplytest.me.
I'm working on something that might enable this stuff, but I'm not going to make any promises.

cilefen’s picture

See #606592-94: Allow updating core with the update manager about Wordpress's potentially catastrophic auto-update vulnerability.

sarciszewski’s picture

Slight update: I've just released the first alpha version of a pure PHP implementation of most of libsodium (sans crypto_pwhash, which isn't easily polyfillable).

https://github.com/paragonie/sodium_compat

It has yet to be audited by a cryptography expert (or experts) but we can seriously consider having Ed25519 signatures baked into Drupal's hypothetical automatic update system as a first-class feature.

If Drupal is interested in helping get sodium_compat audited, please express your interest at this ticket: https://github.com/paragonie/sodium_compat/issues/8

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dww’s picture

Haven't yet had a chance to read this thread, but I wanted to share these links for interested parties. The article was a useful read. Haven't looked at the github repo, but it might be promising.

https://paragonie.com/blog/2016/10/guide-automatic-security-updates-for-...
https://github.com/theupdateframework

Cheers,
-Derek

DigitalFrontiersMedia’s picture

I've started a project for autoupdating D7 branch sites (https://www.drupal.org/project/dfm_autoupdater). I use it with Drush but it could be upgraded for D8 and run Composer scripts, too. It needs a security review, though. And it's not really an out-of-the-box solution since it requires some technical setup but it's transportable and flexible. It basically opens a login shell connection to run whatever set of commands you need to perform the updates and maintenance procedures that suit your workflow.

I'd be interested to hear thoughts on it and if something like that could be modified to suit this thread's need more.

Michael Z Freeman’s picture

This looks great. So the module uses Drupal's inbuilt already defined paths and variables to simplify using Drush ? I was using Drush at one point but, as you put it, the "weeds" got in the way. I needed to write pre and post drush filters to first remove then set permissions on directories that allow the update to happen. Some of the recommended permissions can be relaxed but I think Drupal demands some of them. But I never got to doing the filters, so what could be a relatively simple task got delayed again leaving my site unpatched. So I'll try your module.

DigitalFrontiersMedia’s picture

@Michael Z Freeman, I think it could be automated further, such as adding "pre-made" script options for different workflows that are automatically copied to the desired location based on some additional config setting choices. That would leave the only technical aspect being copying the host user's SSH public key to ~/.ssh/authorized_keys (aside from the two requirements of the host allowing SSH access and Crontab being invoked using CURL or WGET, etc).

It might even be possible for the module to scan ~/.ssh in some cases to produce the appropriate command to copy the public key. I don't know. It may be possible in some hosting environments to have the module copy the key directly for the user. I just tried

drush php-eval "echo shell_exec('cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 2>&1');"

and it copied my key. I'm assuming that it did so as PHP user, but I could be mistaken. Some experimentation may be worth it to simplify the process even more.

Anonymous’s picture

Didn't know this issue existed. However, I developed a script for both D7 and composer-managed D8 projects to do automatic updates and submit the update as a pull request in a Github repository.

This is not the exact scope that this issue seeks to provide, but could be useful as an example use case. The main use case being that something gets committed to VCS when an update happens.

https://github.com/gfs-web-marketing/Automatic-Drupal-Updater

mbaynton’s picture

I've posted over in #606592: Allow updating core with the update manager last month as well, but I think I should mention here too -- I've put in substantial work on a project to enable core updates. Although the first milestone will be in-browser attended core updates, my end goal has always been this issue, and I've been developing in a way to make automation a natural extension.

I'll be sprinting on this all day tomorrow in Baltimore and surely at various other times throughout the week. I would love to connect, go over what I've got, and collaborate with others interested in working on this issue while I'm there.

groovedork’s picture

I'glad to see some things are happening.

Not having auto-update as an option makes me use Wordpress in situations where I'd rather/better use Drupal.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

manuelBS’s picture

Auto updates have been prioritized at Drupalcon Vienna 2017 in the Driesnote. There are different solutions and two different opinions about auto updates in Drupal. I've summed up all the input from BOFs, Sessions and personal talks at http://www.drop-guard.net/blog/auto-updates-in-drupal for those that couldn't join.

ressa’s picture

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

ressa’s picture

xjm’s picture

I'm going to bump this to critical in my capacity as a release manager. We need to be careful in how we try to resolve this issue, but there is a big need for it if we can find a way to mitigate security risks it might introduce.

mbaynton’s picture

Should we have some sort of meeting at Nashville for individuals interested in pushing this forward? Identifying some is honestly the most important thing / the bottleneck ;). Perhaps there could also be brainstorming or even sprinting.

For my part, this remains a topic I am interested in contributing on, but life's just pulled me in too many other directions lately. Probably anyone else showing signs of development activity that I could collaborate with would be a nice motivator.

Lalith Manage’s picture

Issue summary: View changes
jcnventura’s picture

Priority: Major » Critical

Updating to Critical as per @xjm comment in #154.

Mile23’s picture

I think implementing other tools which are adjacent to this tool in terms of functionality would give us a better chance of actual forward motion. (This issue was filed in 2014, btw.)

We should implement #2940733: Site Builder Tool/Project Browser initiative in order to support #2940731: Automatic Updates Initiative overview and roadmap because then we'll have worked out mechanisms for doing all the tasks required for this issue as well. And we will have done it by normalizing a build process for Drupal + extensions that we can rely on for automatic security updates of contrib, as well.

Because right now there is no single build process, and if this issue applies to contrib then it can't be accomplished unless we make one. See #2940733-10: Site Builder Tool/Project Browser initiative

mbaynton’s picture

Yesterday's drupal.org instability during the announced update release window might lend some credibility to an architecture like the one I proposed in #83, where the update code's integrity is assured via code signing rather than https. I'm not sure if the instability this time was in any part due to malicious traffic targeting the canonical distribution point for the update, but I still think it's conceivable it could happen, and being prepared for it would be nifty.

Signed code, which is achievable nowadays via libsodium and sodium_compat in PHP without hard dependencies on any special C libraries, could free us to push updates from locations that are not well-known and so not subject to DDoS.

But how you'd roll all that into Composer-enabled builds would be real interesting, and I do agree with @Mile23 that since we have the Site Builder, broader Automatic Updates, and Composer initiatives as well, we ought to use the same mechanisms that would be required for those more ambitious goals rather than creating a whole other disparate thing that performs a similar task.

John_B’s picture

The delay proposed in #158 makes sense only if Composer is going to be a requirement. Making Composer a requirement is arguably the summary of #2477789: Use composer to build sites but will hold up all issues which depend on it, and will cause controversy.

In addition, those issues, even if solved quickly, are unlikely to be backported to Drupal 7. The present issue should be easy to backport to Drupal 7, which still has the largest number of sites and probably the largest percentage of sites lacking active maintenance.

Therefore, whlist integration of this issue with Automatic Updates, Site Builder inititiative, and Composer, is the cleanest solution to the present issue, it is unlikely to turn out to be the prompt and pragmatic solution. A simple patching script which does not require Composer, and which applies all security updates flagged as critical in a way which does not break Composer installs (and possibly without changing the Drupal or module version numbers) would be a pragmatic solution.

k4v’s picture

I added a feature request to Drupal console:

https://github.com/hechoendrupal/drupal-console/issues/3853

For my use case, that would be simple and elegant -- of course this is not a fix that works for everybody.

k4v’s picture

Another thing I was thinking about: Drupal could have a simple built in "Web application firewall".

We could update just the rules automatically from d.o. In case of a criticial bug we could first push new filter rules to protect the sites immediately, then publish the updates later.

groovedork’s picture

Could we add "+ would have helped avoid drupalgeddon 2" to the list of positive things?

As a sitebuilder I really need this feature. Wordpress works fine for me, and I'm more than willing to have a feature that works like that for Wordpress. While there might be some security concerns, I must honestly say they mostly feel academic / I am willing to accept those risks.

Please let me tick this somewhere in the Drupal interface:

[ ] I am a n00b and would like to have automatic updates.

dqd’s picture

Let me begin with respectfully saying: I can fully understand the sorrows of new Drupal users or "half-time" Drupal users strifing against updating or maintaining complex Drupal projects. Really. I do.

But -- what I really don't understand is and what is eating me since ages is: why people always try to make Apples from Oranges and frequently vote for making Drupal to a 2nd Wordpress(-ish)? I really don't understand that. Use the right tools for the right job. Do not complain about complexity of a power tool. There is no way around it unless you make it less powerful to be easy to use. Nowadays trend to lean, neat and clean is great for One Page Websites, sure it is, but if you try to build a university intranet or groupware, you will miss exactly these cropped settings, which make things powerful but complicated. If you need an easy tool, then do accept that it will be not that powerful like the other. There is an "easy vs flexible paradigms" issue with tools since ages and centuries and I really wonder why nobody understands it? The reason for that Drupal still exist is, that it differs from Wordpress a lot when it comes to big web applications. Many would argue that this is not true and will count tons of third party plugins etc. making Wordpress to half a Drupal. But this is exactly the same issue, only the other way around. Why? And this is exacly where it comes in: In this moment Wordpress is on the same level of security and update trouble like Drupal is for some users.

One thing what made me always screaming in case of I had to use it was Windows automatic updates. Do you really want to make a mass of people contributing to such a mammut task against such a mammut code like Drupal? I can see the issue queue already. Good luck!

PS: I have really much respect and probs (1+) for long term Drupal core and top module contributors and maintainers to "hear and listen" to the croud and to reflect that by adding such discussions and tasks to D.O.. This is really unbelievable nice and I guess many users do not even understand what a monster task that would be. But, hey, who knows, Drupal has always surprised me in terms of easy vs flexibility. If any web application is able to climp that mountain and break that rule, then it is Drupal.

webchick’s picture

This is a cost of ownership issue vs. a "trying to make apples into oranges" issue. Just because a site is built around an "ambitious digital experience" for which Drupal is incredibly well-suited doesn't mean it has a dedicated development team to handle further development/maintenance after it's live. And even organizations with dedicated developer teams often have dozens or hundreds of other sites/apps/technology for which they're responsible. Or have dozens of "in-progress" dev sites sitting around that they've forgotten about.

This initiative is simply about ensuring that Drupal sites that aren't "babysat" stay secure (without getting into the way of those sites that are), and it's incredibly beneficial to both Drupal's reputation and adoption rate.

dqd’s picture

This initiative is simply about ensuring that Drupal sites that aren't "babysat" stay secure (without getting into the way of those sites that are), and it's incredibly beneficial to both Drupal's reputation and adoption rate.

Fully understandable, like always when you bring it to the point, @webchick. :) Thanks for extracting the underlying motivation for me again. It helps me to focus (this long comment threads ...) and to support and embrace this task. But I can't still help it still having a Windows logo in my head while thinking about it :-p (just joking ... )

On the end, as you sad, it's maybe a win win situation. Even if it maybe is not for everyone having it turned on, the update situation as a whole of Drupal will surely win on this initiative as well.

I am only curious if I possibly lost a little bit focus by reading all the comments and was thinking that we now already discuss full updates to next versions which made me wonder how you want to handle all the individual installs and dependencies (not only Drupal, also Symfony, etc.) which are so widely spreaded that I can't really see where this feature would not break a not-"babysat"-ed site without the admin knowing of it, because of one updated part which can't support another enviremontal part no more which can't be updated because .. etc ... or will spit errors. Or do we stilll talk about security patches only? That would even make more (and any) sense and that would obviously make me looking dam stupid in the light of the headline :D ... Because this would render all my worries obsolet. *facepalm*

webchick’s picture

Yay! Happy to help. (Insert Clippy image ;))

And, correct. At least in the short-term as an MVP, this is around automatic background upgrades for highly critical security issues only. (Any further functionality would only be added once this foundation is rock-solid and we've learned some real-world lessons.) And no matter what we do here, it'll also need some kind of "opt-out" mechanism, as Update Manager has itself (e.g. by turning the module off, by specifying a custom updates server, etc.) for advanced use cases.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

geek-merlin’s picture

NiklasBr’s picture

automatic background upgrades for highly critical security issues only […] it'll also need some kind of "opt-out" mechanism

Just throwing a thought out here regarding the severity. Every SA has a risk level up to 25 assigned, what if site owners could configure which risk level would be allowed to apply automatic background updates. E.g. I could configure so that any patch tagged as >15 would be installed automatically, while the default (to begin with) might be >20. Just an idea.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

catch’s picture

Version: 8.9.x-dev » 9.1.x-dev

There is progress on this in contrib at https://www.drupal.org/project/automatic_updates

Moving this issue to 9.1.x

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

tedbow’s picture

This is happening!

See #3253158: [PP-1] Add Alpha level Experimental Automatic Updates module for how the contrib module 8.x-2.x branch is being convert to this alpha level core module

tedbow’s picture

Status: Active » Closed (duplicate)