This post was created jointly by Michael Hess of the Security Working Group, and Tim Lehnen, Executive Director of the Drupal Association.

Last year, with the security release of SA-CORE-2018-002, the most significant security vulnerability since 2014, we heard the pain of site owners and development teams around the world staying up at all hours waiting for the complex security release process to complete and the patch to drop. We heard the pain of agencies and end-user organizations required to put teams on late shifts and overtime. We heard from some users who simply couldn't respond to patch their sites on the day of release, because of lack of resources or entrenched change management policies.

We've heard calls from the community for rotating the timezones for security advisories from release to release, or for having more on-call support from security team members across the globe, or simply for a longer horizon between the release of PSA and SA.

Yet at the same time, we're cognizant that these solutions would put increased burden on a security team composed of dedicated volunteers and contributors. There are a number of generous organizations who sponsor many of the members of the security team, but relying on their altruism alone is not a sustainable long-term solution—especially if we consider expanding the role of the security team to address the larger pain points above.

Last week, with the release of SA-CORE-2019-003, we heard these concerns for site owners and the sustainability of the security team echoed again.

The Security Team and the Drupal Association have been developing solutions for this issue for well over a year.

The goals are simple:

  • Provide a new service to the Drupal community, from small site owners to enterprise-scale end users, to protect their sites in the gap from security release to the time it takes them to patch.
  • Create a new model for sustainability for the Security Team, generating funding that 1) covers the operating costs of the program 2) can support security team operations and 3) can support additional Drupal Association programs.

Although the execution will take care and careful partnership, we are happy to announce that we've found a solution.

We're tentatively calling this: Drupal Steward. It is a service to be provided by the Drupal Association, the Security team, and carefully vetted hosting partners.

Drupal Steward will offer sites a form of mitigation through the implementation of web application firewall rules to prevent mass exploitation of some highly critical vulnerabilities (not all highly critical vulnerabilities can be protected in this fashion, but a good many can be - this method would have worked for SA-CORE-2018-002 for example).

It will come in three versions:

  • Community version - for small sites, low-budget organizations, and non-profits, we will offer a community tier, sold directly by the DA. This will be effectively at cost.
  • Self hosted version - for sites that are too large for the community tier but not hosted by our vendor partners.
  • Partner version - For sites that are hosted on vetted Drupal platform providers, who have demonstrated a commitment of contribution to the project in general and the security team in particular, protection will be available directly through these partners.

Next Steps

The Drupal Association and Security Team are excited to bring this opportunity to the Drupal Community.

We believe that the program outlined above will make this additional peace of mind accessible to the broadest base of our community possible, given the inherent costs, and are hopeful that success will only continue to strengthen Drupal's reputation both for one of the most robust security teams in open source, and for innovating to find new ways to fund the efforts of open source contributors.

We will announce more details of the program over the coming weeks and months as we get it up and running.

If you are a hosting company and are interested in providing this service to your customers, please reach out to us at drupalsteward@drupal.org.

Please also join us at DrupalCon for any questions about this program.

If you are a site owner and have questions you can join us in slack #drupalsteward.

For press inquiries, please contact us at: security-press@drupal.org

Comments

sandervd’s picture

Releasing on a fixed time instead of somewhere during a four hour window would also help. There are better ways of spending time.

Are there plans to coordinate this rules based approach with the CDN vendors (e.g. Cloud Front, Fastly...), as to have them deployed there automatically?

rivimey’s picture

I am with sandervd - release the patch information at a specific time of day, plus/minus a minute or two. If there are good reasons for the time to be variable - and I can't think what they are - then perhaps the release team could post progress notes on drupal slack to let people know. The frustration of not knowing what was happening during the 2018-02 event was considerable.

DrupalSteward sounds - with my community user hat on - like a money-making activity. Possibly something Acquia or Pantheon would get with well, but not something that feels "right" for the open source community.

tedstein’s picture

> a money-making activity. Possibly something Acquia or Pantheon would get with well, but not something that feels "right" for the open source community.

Exactly.

Community: We would love to know when patches come out.

Drupal Association: It's a complex problem and we need to let various select companies (selected by us, of course) offer a solution as a paid-for-service. And no, we won't tell you when patches will come out.

I hate to sound so negative, but it does not feel like you "hear our pain" as nobody asked for this.

hestenet’s picture

This is a problem we'd like to solve separately. There's a discussion of the current logic behind the release timing and the release window here: https://www.drupal.org/drupal-security-team/security-release-numbers-and...

In essence, it boils down to the fact that the security team can't necessarily guarantee that the release will hit an exact hour, there's some wiggle room involved depending on the nature of the release, how core and contrib releases might need to be coordinated with each other, and a number of other factors.

Don't get me wrong, I think it would be great if we can make the window more precise - but I think that's something we run in addition to this program.

And even if we were able to guarantee the release happened exactly at 3pm EST, for example, that would still put most of the international community in a tough spot, so I think there's a role both for the Steward program, and for improvements to the patch process itself.

--
Tim L
Drupal Association - CTO

generalredneck’s picture

When some person decided it was was a good idea to DDOS D.O during a release window which delayed the creation and deployment of the SA that accompanies the announcement. Knowingly or not knowingly, these types of things happen which means the security team needs some wiggle room. 

From a maintainer's standpoint, we ALSO need the wiggle room as we work with the security team and for some of the maintainers, that window falls in the middle of the night. Sometimes there's technical difficulties pushing a change to D.O that is beyond our control as well that doesn't fit well within an hour. 

tedstein’s picture

I worry that this is a path that ends in a place where Drupal can only be safely hosted by a few high-end service providers and large enterprises. Will a single developer be able to host Drupal safely on a Droplet for small business/artist/non-profit type clients? Or are non-enterprise-scale projects expected to host with the association or Pantheon/Acquia/etc.?

We already have a situation where certain companies learn about security exploits and patches before the general community. Do we really want to create a new tier of hosting providers with access to knowledge and tools that isn't shared with everyone else in a timely manner? There are thousands of Drupal contributers. Why aren't we treated equally?

Again, I hate to sound so negative, but the more I think about this the less comfortable I am with this.

hestenet’s picture

The community tier will not require hosting with one of the major platform providers. In essence it will be a WAF/CDN product, which will likely require community-tier sites to make a dns change to route site traffic through this protective layer.

--
Tim L
Drupal Association - CTO

tedstein’s picture

Thanks. If I were to sell Drupal hosting, would I be able to run this protective layer myself? Will the self hosted version be free or paid for? Will the code be open source?

hestenet’s picture

There will be some strict requirements on what hosting partners will be able to run the protective layer themselves, based on their level of contribution to the security team and the project as a whole. There is a good reason for this: being given the tools to implement this protection means having access to potentially reverse engineer vulnerabilities early, and so we need to keep the pool of people and organizations with that sort of information as small and trustworthy as possible.

Each tier will have some amount of cost associated with it, because of the inherent costs of operating these services. Open source code is free, but services have inherent costs, so we want to cover those while making this accessible to as many people as possible. But it will certainly be some form of sliding scale. The inherent costs largely scale with the amount of traffic sites receive.

In terms of the code itself being open source - that's sort of question is really... perpendicular(?) to the sort of service we're talking about. There won't be much 'code' to speak of- but rather firewall rules implemented by our partner service providers. Those rules can't be disclosed because they would mean disclosing the vulnerability. Hopefully I explained that well enough to make sense!

We are working on an FAQ which we will post soon.

--
Tim L
Drupal Association - CTO

tedstein’s picture

This will be my last comment on this thread, because all the questions that affect me and my business have been answered, and I look forward to the FAQ, and (although my griping may hide this but) I greatly appreciate your hard work and don't want to waste your time.

But there are some things I am still unclear on.

How will the firewall rules disclose any vulnerabilities that the patches will not? Will the firewall rules be released before the patches? Aren't you planning on sharing on sharing the firewall rules with the self hosted tier? If so, how do you vet them (other than them paying a bill)? Do you see why small development firms or hosting companies might be bothered by this?

hestenet’s picture

No worries! Those are good clarifying questions.

  • Yes, the idea is that the firewall rules are put in place before the patches are released.
  • This self-hosted tier would be operated the same way as the community tier, i.e: this tier accommodates sites that are larger than the community tier, but not hosted on a platform partner. However, they still do not directly receive the rules, they simply have the ability to route through essentially the same WAF service that the community tier uses. (I can see how that would be unclear from the name, so we'll work on that language)

Hope that helps.

--
Tim L
Drupal Association - CTO

sandervd’s picture

Why not publish the rules in a mod_security channel, at the same time as the patches?

I can only assume that WAF vendors already have a system in place to apply mod_security rules, or transform them into their internal format?

hestenet’s picture

That may be possible! And we've been talking with the SIWECOS initiative a bit about this possibility: https://www.cms-garden.org/en/projects/siwecos

To be clear, I think this would *supplement* the service provided - providing rules at the same time as the patch still doesn't help folks up at odd hours of the night, or with large teams they need to put on call/overtime. So there's room for both approaches.

--
Tim L
Drupal Association - CTO

sandervd’s picture

providing rules at the same time as the patch still doesn't help folks up at odd hours of the night

Only I do think this would help. Running a bash script on cron that fetches WAF rules from a trusted source (git ?), and reloads the rules if they are changed would solve the issue in an open way. No need for third parties to solve this.

tedstein’s picture

There is also a subset of users whose organization's policies prohibit deploying untested code but would allow firewall rules to be implemented for security reasons until the code is tested and deployed. I can't understand how it could hurt to provide the rules with the patches.

ShaneOnABike’s picture

I also don't see how this avenue is going to be that accessible to the types of clients I work with and for. But perhaps this is going to be a free open source solution we can implement rather than a paid service (I SURE hope so).

Nonetheless, I agree with others. Having a fixed hour and time makes sense in general. Also, why not build a special module that simply applies patches to existing modules and drupal system. It can't be that hard to do I would imagine. I also don't understand why in the security releases we don't have a link to a patch that can be applied. This is waaaaaaaaaay quicker then having to download and change all the source for a quick-fix patch (and later proper install).

Anyway, I'm in agreement with others and think we need to think much more out of the box than providing a service. I think there are loads of awesome solutions that could be had that still makes Drupal open-source, free, and awesome!

hestenet’s picture

We're targeting a *very* inexpensive (more or less at cost) level for that community tier, so we hope it will be accessible as widely as possible.

Re: automatic updates/patch application - there is an initiative for that underway right now as well - and we've managed to secure some funding from the European Commission for that purpose.

To be clear - we want to pursue all these improvements to the extent that we can :)

--
Tim L
Drupal Association - CTO

Wolf_22’s picture

From what little I'm able to understand from this so far, it sounds like this service would be required to implement an in-Drupal functionality seeking to allow site operators the ability to add said web application firewall rules... Is this correct? I ask because not everyone has access to a given configuration interface like a normal networking operations staff does. (Maybe I'm just lost here?)

As for the fee, I can respect it but I just wish a better option existed for this entire open source foundation Drupal has been built upon. I would think something similar to what the Autoban Drupal module provides could be implemented in such a way as to replicate the ubiquitous architecture that the AdBlock Plus definitions provide where you have a single Drupal module that acts as a means of preventing site access by specific sets of rule-based evaluations and when said blocks occur, a supposed DA service worth mulling is a centralized database storing the blocks and broken down by rule sets that site operators could select from. THIS, in my opinion, would be a much more valuable route to take and for those preferring not to pay for the "centralized rule set database definitions," they could still utilize the module on their own but accept the risks of being responsible for their own rule set blocking definitions that they could define themselves (again, similar to what Autoban does).

hestenet’s picture

As envisioned this would not be an in-Drupal feature or special module - it would be a globally distributed WAF provider partnering with the DA and the Security team to provide the service. Likely the primary configuration for site owners would just be adding a DNS record.

--
Tim L
Drupal Association - CTO

Wolf_22’s picture

How would that translate for C-PANEL site operations?

hestenet’s picture

In most cases I think the DNS changes would happen on the domain registrar side, rather than on the hosting/cpanel side.

--
Tim L
Drupal Association - CTO

Wolf_22’s picture

Ah, so it sounds like this service would be mostly for the companies that provide C-PANEL for users (and not the users of C-PANEL). Do I have that right?

hestenet’s picture

It will depend a little bit. An individual site owner (Cpanel user or otherwise) may be able to implement it for themselves if they have direct access to their DNS records. (Almost all hosts, including shared providers like GoDaddy and the like do provide this access). For some folks in special hosting situations they may need to contact their host/domain registrar for help in making the change.

I wish I could be more specific but we're still sorting out the details so we'll be able to provide more accurate instructions when closer to launch.

--
Tim L
Drupal Association - CTO

Wolf_22’s picture

That makes sense. I think I know what you're talking about in C-PANEL in the section(s) where you can do things like change MX records, etc.

Thanks for the info.

hestenet’s picture

Yes, exactly.

--
Tim L
Drupal Association - CTO

rivimey’s picture

The more I read about this the more I feel it is the wrong way forward, not just for small drupal sites (who *normally* have *zero* budget and so have to beg/borrow whatever they can), but for Drupal as a whole.

My reasoning is that (a) this whole apparatus depends on firewall rules, and so cannot mitigate anything that cannot be matched by that route. Matching rules vary considerably in how detailed they can be, how to apply them to a site, and what physical form they take. Any framework for serving rules would therefore have to involve massive amount of cross-application and platform testing, or the use of global level proxying similar to Cloudflare, which is very much not possible for many sites.

And (b) Drupal is an open source, open access system. Sure, there are commercial solutions aplenty, but they are at the edges. Many thousands of people have given many millions of hours of time, and countless other resources, to make an open source and open access Drupal happen. For one of the pillars on which Drupal is built - good security - to become a paid-for service is surely a slap in their face and a violation of the premise under which they contributed.

Needless to say, I completely disagree with both the technical approach, which I feel is very shortsighted, and also with the lack of community involvement and consideration, which will alienate many current Drupal users.

hestenet’s picture

To be absolutely clear, Drupal Steward will not change, eliminate, or put a paywall in front of any existing activities of the Security Team which have made Drupal's security reputation what it is.

This endeavor is an effort to add an additional, service-based layer to provide additional peace of mind for the site owners who choose to participate, and to support the sustainability and future programs of the security team and DA.

Other programs are already underway to enhance Drupal's security, such as the Automatic Update Initiative, the Drupal 7 Extended Support program, and the Drupal Bug Bounty, and all of that will be contributed back according to the Free Software ethos.

That said, we want to approach this from all angles, both enhancing the community-driven open source model for Drupal security and providing supplementary services(which, yes, do have an expense).

In other words, we can have our cake and eat it too.

There should be more information about this soon coming out of DrupalCon, so I hope this brief answer can serve for now while we address these and other concerns we've heard.

--
Tim L
Drupal Association - CTO

grantkruger’s picture

Anything that makes most security updates less urgent sounds great to me. There were so many people at DrupalCon complaining about the burden of the sheer volume of security updates we now have in D8. When it also came up in a security presentation I was please to hear security team members say they were already on this and that Drupal Steward was on the way. I'm pretty sure we'll be one of the first to sign up, even though I usually get security updates live within an hour or two, because you can never be too safe and some of those times caused me other issues that could have been avoided if I'd had a few more days to finish other things. I also wonder if Drupal Steward will start to become a service something like the Wordfence plugin on Wordpress that really saved my bacon back when I had to work on a Wordpress site.

ressa’s picture

In Drupal 7, if there aren't resources to keep a dedicated security team on stand-by to keep your Drupal web site safe, a solution is to cron-trigger drush up --security-only every hour. Daily backups are of course required, to allow rollback, should anything go wrong.

This has worked well for me for many years, in different Drupal 7 projects, but isn't possible with Drupal 8, since composer update can't be run automatically, without human interaction.

An alternative solution would be to check if there is a security update above a certain level (for example 20/25) hourly, and lock down access to the web site entirely, by updating the firewall rules, and blocking the relevant ports.

The next day, the security update can be applied, and the firewall re-opened.

Has anyone tried something like this, and feel like sharing their method?

Wolf_22’s picture

Along with that idea would be something inherent to core that allows for a temporary ban on all IPs until a security patch / upgrade has been applied. For example, a special security-centric setting that would allow site owners to ban all traffic to the site except any whitelisted IP addresses or ranges if a type of defined update or upgrade hasn't yet been applied. What I envision is essentially what you describe only at the app layer but agnostic of any firewall system requirements, so any site owner could do this even if a host lacks firewall configurability for its customers.

ressa’s picture

That's a great point: Many site owners won't have permissions to alter the firewall configuration, and doing it at app level is probably also easier, if doable.

Might updating the root .htaccess file allow what you describe, and would it cascade down the temporary ban of IPS defined in it recursively to include all files and folders? If for example you run a University web site, you might want to shut off external access after a security update release, yet allow access internally, from a set range of IPs.

Wolf_22’s picture

The whitelist could remedy that scenario within the "special" security-centric configuration. Emphasis on this idea not being like a standard range ban you could achieve using the IP Ranges module and this situation of banning everyone except a whitelist IP address or range would only occur in specific security scenarios, such as the release of security updates, a configuration site owners could configure themselves.

And yes, the HTACCESS route is possible, as long as people know how to PROPERLY edit and use HTACCESS (something too easily done wrong). From what I understand, it's also something that isn't always 100% effective but don't ask me why because I'm not sure what the reasons are.

I just think the more you can do at the app layer, the better off Drupal and anyone using it will be. Given the base is there to incorporate something like this into core, I wouldn't think it would be that difficult to do but I could be absolutely wrong.

ressa’s picture

Thanks for sharing the IP range module, I haven't about it before ... though the last update to 8.x-1.x-dev was 2. June 2014 ...

I will experiment a bit with .htaccess. A simple script which injects a rule in case of security update might do it. The hard part is probably determining 1. Is there a security update?, and 2. The level of the update ...

And I agree with you that something like this baked into Drupal core would be great, until automatic core update is ready, sort of like a Poormans security update.

thronedigital’s picture

It was tedious but I've got automation started for our Drupal 8 builds. A good starting point is to use Drush 9's pm:security

i.e. /usr/local/bin/drush9/vendor/bin/drush pm:security --format=list 2>/dev/null

Loop through and if there is a security update present use a function to exec composer update [security item] --with-dependencies-n followed by DBupdate from drush.

/usr/local/bin/drush9/vendor/bin/drush -y updatedb 2>/dev/null

ressa’s picture

Interesting, thanks for sharing your approach. I had previously read that pm:security was going to be removed from drush 9.6, but it seems to still be there in, drush 9.7. Would it be possible for you to share the code you use? I am sure many others would be interested in seeing it. What do you do when Composer requires human interaction, for example when updating patches?

thronedigital’s picture

I'm aware of that as well. I will have to go back to the drawing board when that day comes if ever.

Sure i'll look at setting up a public repo for example.

for the human interaction bit of it composer allows you to append --no-interaction to the end of the command or for short just -n

thronedigital’s picture

Which just says Yes to all questions. A bit dicey but I assume if you want to install the security patch the answer is going to be Yes anyways.

ressa’s picture

Ah yes, I have been experimenting with Composer commands like this:

$ composer config -g discard-changes 1

$ composer update --no-interaction

Still, I seem to remember that often "something" would happen, which would make Composer present a prompt and wait for human input (... I seem to remember there was a git issue with the coder module and uncommitted changes). So I would feel leery of automatically updating Drupal 8 with Composer, but I am interested in looking at your example, and experiment with it.

rivimey’s picture

While I knew of the cron-update thing, I hadn't thought of trying to limit by criticality level - that is a good point.

Using .htaccess seems to me a good idea, because it is read every time and so no restart of service processes is required. I would first look to the php code changing an embedded SetEnv in the .htaccess - you can base multiple other decisions on that, and it would be a relatively simple line to find and change (read/modify/write).

As for how the criticality level for an update is determined, I don't know... any offers?

ressa’s picture

When I think about it, just discerning whether there is a Drupal core security update (not ordinary) available for the installation, which is not rolled out yet would be ok for my use case, for a start.

In case there is, I thought about inserting the four lines for Password Protection with htaccess in the root .htaccessfile, displaying a "The web site is under maintenance" message above the login prompt.

When the web site is updated manually with composer update, a fresh .htaccess will be added, which is fine, since I don't want to maintain a custom .htaccess file.

Perhaps the Automatic Updates project has some inspirational code to determine if there is security update available for the current web site, and perhaps even the level of the security update?