In the Open is a weekly series from Drupal Association Interim CEO Tiffany Farriss, sharing what she is hearing across the Drupal community and where the Association is headed. In this episode: how AI is changing vulnerability discovery for everyone, what the Drupal Security Team and the Alpha-Omega funded SEIR program have achieved in five months, and how site owners, hosts, and organizations can help.

"In Drupal now that finding these bugs is really cheap, fixing them and fixing it safely is not."

The following is a transcript of Tiffany’s remarks, lightly edited for readability.

What’s changing for everyone

Today I want to talk about security. I think Drupal’s story is about a community that is adapting pretty well to some pretty trying times.

I want to talk about what’s changing for everybody, what we do, what we’ve seen so far, and then how you can help and get involved.

It’s a completely new situation in the last 12 months. AI-assisted analysis is making finding vulnerabilities cheap and automatable in commercial software and open source software. It’s not a Drupal problem. It’s not an open source problem. It’s an everyone problem.

AI is astonishingly good at finding vulnerabilities. And it’s not just individual vulnerabilities, it’s about chaining them together. In COVID, we all learned about the Swiss cheese model of safety, the idea being that the holes in the cheese don’t line up in the various layers. But AI is just so good at figuring out how to chain one vulnerability to another one, to another one, and it navigates it at a pace and in a way that we just haven’t ever seen before.

From an open source perspective, we know that open source is in almost everything. 96% of all applications use open source libraries, and every vendor’s product inherits the security of all of the dependencies, whether they’re in your software or they’re in open source.

One of the core problems that all of open source is looking to solve, and that I’m looking to solve for Drupal, is that companies expect enterprise-grade security from people who are fundamentally volunteers. They’re either volunteer maintainers or, in the case of Drupal, a largely volunteer security team.

To address this, Alpha-Omega, which is a project of the Linux Foundation, has been funding security experts in residence for a broad swath of the ecosystem. They’ve been doing it for PHP, Node, Ruby, Rust, Python, Perl, Erlang, FreeBSD, Eclipse, and now Drupal. I think Drupal is the first one of our layer of open source software to be included, and we’re super, super grateful.

And it’s not just about the funding that we’ve gotten from Alpha-Omega. It’s the fact that we’ve been invited into this cohort to share ideas and resources with each other. That makes it feel like every ecosystem is really working on this together.

How Drupal security works today

For those who aren’t as familiar with what Drupal already does, we have one of the most amazing security teams. If you see any member of the security team, you need to thank them, because as a largely volunteer effort they have protected millions of sites for over two decades.

One of the things people are often surprised about is that not only do we have very mature practices, but we use our own scoring method and we have our own CVE numbering authority, which a lot of folks aren’t. Our coverage includes core and then contributed projects that opt into security coverage.

Several years ago, our security team came up with the idea of Drupal Steward. Drupal Steward is web application firewall rules that protect against any critical or highly critical issue that is preventable by a WAF, or web application firewall. We distribute those rules to any of the trusted Drupal Steward partners. There was also a community tier that allowed people to put the Drupal Steward WAF in front of their WAF. That last product hasn’t been as successful. People haven’t really been able to put their WAF behind our WAF. So really, the folks who end up being protected are the ones who get it through their hosts.

In the first several years of Drupal Steward being available, it wasn’t activated very much. We didn’t have very many vulnerabilities that were preventable by a WAF or that were that critical, and the pace was much, much slower. This year it has been activated, I think, three times already. Given what we’re seeing across all of open source, I do expect there will be more activations to come.

If you’re already working with one of the partners that is a member of Drupal Steward, you’re protected. But we’re also working on a new way to distribute our WAF rules through CDX, and I hope to be able to share more about that in the future.

I want to talk a little bit about those partners who are members of Drupal Steward, because they’re ones who are known to us. The way the program is designed, these partners have members of the security team. That’s critical for us for a couple of different reasons.

One, it keeps our circle of trust really small. Advanced notice is only as safe as the smallest group that holds it. When we’re working through an organization that already has an existing security team member, the early details stay with an organization that already had access to them. They know they’re bound by the security team’s disclosure policy, and that policy makes it really clear that you can’t disclose this beyond what gets disclosed publicly to everyone. So there’s no inherent advantage.

But security work in Drupal takes Drupal expertise. Any of the fixes that we put into Drupal Steward require knowing how to block the exploit itself without blocking or breaking existing real sites. So the Drupal context is as important as the WAF rules.

What the last five months have shown us

Our team responds to reports. That’s the operating MO. One of the things the SEIR grant has allowed us to do was to actually begin doing some scans on different pieces of it. What we found, and this is in a blog post I released a few days ago, is that AI reporting has gotten so easy to do that we instead shifted the time from the folks who are in SEIR to really focusing on fixing, and then creating some very durable tools to help us manage this onslaught.

If you hadn’t caught up on the post: in the five months that we’ve been involved in the SEIR project, from June through the end of October, there were 256 advisories issued by our security team. That’s more than the previous 30 months combined. Prior to the start of this June wave, there were generally around eight per month.

We also found that the number of credited reporters since June has more than doubled since last year. There were 134 credited reporters since June alone, in this five-month period. In all of last year, there were 68. I think this gives us some sense of how fast these issues are being found and disclosed.

But I also think it’s important that we’re really clear that it doesn’t mean the code got worse. It just means that we’ve gotten better at finding these issues, and it’s now cheaper to do so.

In Drupal, now that finding these bugs is really cheap, fixing them, and fixing them safely, is not. That still takes deep Drupal knowledge. What we promise, both in our security team releases and in Drupal Steward, is that you fix the vulnerability but you also care that you can’t break anything. So the security fixes make the smallest change that closes the security gap. We want to make sure that there are no new APIs and no breaking changes when possible. If any of our updates break sites, it decreases the number of people who’d be willing to apply them. And an unapplied fix isn’t really a fix, is it? The cleaner, proper version can be worked on in a public issue queue once it’s disclosed and once we have a good patch.

It is important that everybody on the security team be first and foremost Drupal experts. So if you are someone who does Drupal security work, or you have worked with the security team on your own contrib modules, and you’re interested in levelling up and learning more, please reach out to the security team and see how you might be able to get involved.

Drupal’s threat model is its own thing. We have very specific trust boundaries and permissions that restrict access, and things that are meant just for our trusted administrators. What we find in reports that come from folks who don’t necessarily know Drupal, or even in fixes that come from folks who don’t know Drupal, is that they might miss those details. It impacts how you tell a real vulnerability from an intended behavior that Drupal already has.

The funded reviewer pool that we have comes from the existing security team, and this is why any of the hosting partners or CDNs who get active in Steward would have to have a security team member. Even as we scale the program, that’s still going to continue to be true. We need to focus on adding capacity without widening the circle, and make sure that we actually have folks who are there.

Who is carrying the load

Alpha-Omega has been providing the SEIR program for years with some of the projects, but this was Drupal’s first term in it. We’re in the first six-month engagement. What was different about the way that Drupal did it is really a reflection of that entire process. You have to fund not just one person to go in and find and fix bugs, you have to fund the whole security workflow. If we just focused on one researcher, everything would be stuck. You need to make sure that throughput is the primary focus. So we have several roles that are partially funded, instead of the usual single person funded full-time. I think our results five months in have shown that it made a big difference.

That SEIR team, which includes Drew Webber, Greg Knaddison, Michael Hess, and Neil Drumm, supported by both Tim Lehnen and myself, participated in about 87% of the advisories, including those two critical core fixes a few weeks ago. They’ve also started to focus on creating reusable tooling and on finding and fixing some upstream bugs that were in fact impacting us downstream. There have been maybe half a dozen hardening changes, and also fixes to issues in Drupal.org’s infrastructure itself. So I’m very grateful to them and their work, and to Alpha-Omega for funding it.

But it would be remiss of me not to note that it didn’t fund all of the work. This is a huge influx and a huge increase in the number of vulnerability disclosures that the team is working on. As we have seen throughout Drupal’s entire history, it’s largely funded and largely done by people who are volunteers. Almost every member of the security team, even those who weren’t funded by SEER, increased their contribution year over year, if I look at June through October of this year versus June through October of last year. It impacted maintainers as well. They needed to write, test, and ship fixes on timelines.

Some members of the team also have time that’s sponsored by their employers. We know that they likely go above and beyond that. I think that’s just always true, and I am deeply grateful for folks’ willingness to do that. Some were funded through the Drupal AI Initiative to do some work, and that allowed it to spill over into their security team subsidies. And then there’s also some Drupal 7 end of life work.

If you are one of the companies who encourage people to do this work, I’m thinking here of Tag1. The security team has told me that without Swan Colada’s participation, the last few months would have been impossible to manage. So I really appreciate their work, and also Tag1 for sponsoring some of it.

But it took more than 100 volunteers to work on all of the security issues that Drupal has had to endure over these last five months. So I thank you deeply. I thank any companies that are sponsoring you, either directly or indirectly, to do this work. I appreciate it.

But it’s really clear that we need to find ways to increase the amount of funding that goes not just to the security team, but also to the core release team. Every vulnerability that we address only matters once it’s been released to the public. We need to make sure that we aren’t just relying on people’s spare time indefinitely, and find a way to improve that.

What I’m looking at are ways that we can move beyond just relying on grants or individual organizations to fund huge chunks of people’s time, and thinking about how we might make it easier for other people to opt in, and how we can make Drupal Steward accessible to more people. At the end of the day, my priority, my mission as part of the DA, is to get as many Drupal sites as secure and as protected as possible. And I want to do that in a way that respects the time and the expertise of the people who build it: maintainers, core release managers, security team members, all of us. I think that’s how we protect the commons as we move forward.

How you can help

First and foremost, please make sure that you are covered by Drupal Steward. If you’re a site owner, you can ask your CDN or your host provider, “Are you a Drupal Steward partner?” If they’re not, please encourage them to become one. If you are a shared host, a Drupal platform, or a CDN, get in touch. We’re happy to have some conversations about how you can become a Drupal Steward partner. You pass that protection on to your customers, and then help us increase the capacity of our security team.

Also, if you haven’t yet, and I really hope you have by the time you watch this video, please patch PHP and keep your sites updated. Drupal Steward will buy you time for Drupal vulnerabilities, but it can’t replace anything upstream, and it certainly never replaces updating your sites to secure versions of the modules that we release.

And if your organization depends on Drupal, sponsor people who know Drupal, either in your own organization or in the community, to contribute to security specifically.

I want to end this with huge thanks, as always, to the volunteers who do our core release management, to those on the security team, and to all of the maintainers who’ve been involved in absorbing this AI wave. I unfortunately don’t see it abating anytime soon, but I think we can continue to collaborate and stay in conversation about how we can make it more manageable and more sustainable for all of us.

So with that, I want to thank Alpha-Omega again. I want to thank our colleagues over at the PHP Foundation for all of their work last week, Trail of Bits for helping us find our critical vulnerability, and OpenAI, as well as Anthropic, who have both been providing us with AI tools to manage this work.

Follow the conversation

New episodes of In the Open publish each week. Subscribe on the Drupal Association YouTube channel to follow along.

With thanks to Stephen Cross of Second Signal Media for producing this series.