Drupal Association collaborates on new, long-awaited tech initiative as featured on TagTeamTalk

The Drupal Association collaborated on Automatic Updates, one of the Drupal Core Strategic Initiatives that was funded by the European Commission. We are excited to partner with MTech, Tag1 Consulting, and the European Commission FOSSA program on this new initiative and share information with you about its features.

Automatic Updates has three components.

Public safety messaging

This feature pulls a feed of alerts from Drupal.org directly into Drupal's administrative interface. This helps ensure that critical Public service announcements (PSA) or Security Advisories (SA) from the Drupal security team will be seen directly by site owners. 

• This provides yet another communication mechanism before an update so site owners can verify they are ready for an upcoming update, before it lands.

• The feed of alerts comes directly from the feed of PSAs and SAs that the security team and release managers are already producing. 

• This will vastly increase the ability of the Drupal project to get the word out about critical and highly critical updates - ensuring the community can respond fast. 

Readiness checks, or “Pre-flight” checks

These automated and extensible readiness checks are built into the Automatic Updates system to verify that a site doesn't have any blockers that would prevent it from being updated.

• These checks are slated to run at least every 6 hours on a site via Drupal Cron and will inform site owners if they are ready to auto update their site.

• Examples of the readiness checks include:

  • Is the site is running on a read-only file system?

  • Have any files included in the update been modified from what they should be? 

  • Does the site still need to run database updates, etc.? 

There’s about 8 or 9 of these readiness checks and some are warnings (Cron isn’t running frequently enough to automatically update the site in a timely manner) and some are errors (the file system is read-only). Warnings won’t stop automatic updates, but errors will.

In place updates

Finally, the key pillar of the automatic updates feature is the update itself. Drupal.org generates a signed and secure package of files which can be overlaid atop the existing site files in order to apply the update. 

• This update package is downloaded as a signed zip file from Drupal.org. The automatic updates module on the site then compares the signature of the zip file using drupal/php-signify, which is based on BSD’s Signify and libsodium to verify the package.

• It then proceeds to backup the files about to be updated and updates the site.

• If all goes well, the site is upgraded. If something fails, the backup is restored.

• Many workflows are supported and you can customize how the updates are performed. Updates can flow through your CI/CD system, be staged for review and approval, and or automatically go live.

In the past few weeks, the Drupal Association has been invited to participate in TagTeamTalks, a new recorded talk series about various tech projects supporting the Drupal project. This bi-weekly format provides real-time shared collaboration and informative discussions. 

TagTeamTalk launched its webinar focused on Automatic Updates this week. The group dives deep into the nuts and bolts of Drupal's groundbreaking Automatic Updates feature, and the strategic initiative sponsored by the Drupal Association, MTech, Tag1 Consulting, and the European Commission. Guests include Preston So (prestonso), Contributing Editor at Tag1 and Moderator of the TagTeamTalks; Michael Meyers (michalemeyers), Managing Director of Tag1; Lucas Hedding (heddn), Senior Architect and Data and Application Migration Expert at Tag1; Fabian Franz (Fabianx), Senior Technical Architect and Performance Lead at Tag1; and Tim Lehnen (hestenet) CTO at the Drupal Association. Read the TagTeamTalks blog.

“Content marketing is one of the most effective ways to promote your brand and capabilities - it has been a really powerful approach for the organizations that I’ve worked for,” said Michael. “The goal is to give our team an opportunity to talk about the cool things they’re working on and excited about and to share it with people. It helps get the word out about the latest developments in the open source communities we contribute to, and it promotes Tag1’s expertise - it helps us recruit new hires, and drives new business.” 

Meyers is the Managing Director of Tag1, and has been involved with the Drupal community for over 15 years. He was Founder and CTO of the first venture backed drupal based startup, CTO of the first Top 100 website on Drupal, and VP of Developer Relations at Acquia before joining Tag1.  “The great thing about TagTeamTalks is that it doesn’t take a tremendous amount of effort or energy. Our engineers are subject matter experts. We decide on a topic for the week, spend 15 minutes brainstorming a rough outline as a guide, and then record the talk. We don’t want to be rehearsed. The conversation is what makes it dynamic and enjoyable for us to do, and for people to listen to. And, the team loves it because they want to talk about what they are working on, and this format doesn’t take a lot of time away from what they enjoy doing most - writing code.” 

Hedding is one of the top 20 most active contributors to Drupal 8, and is also the Drupal Core Migrate Sub-system Maintainer, a core contribution mentor, and a D.O. project application reviewer. “Auto Updates has long been one of the most requested Drupal features, it is a capability the platform really needs that will help everyone using Drupal. Now that the alpha is available, we need to early adopters to start using it, we need feedback so we can continue to improve it. We also need to get more people involved in development, and we need to raise more money from organizations to support the project - it might sound like a simple feature, but it is actually really complex and requires a lot of effort. TagTeamTalks are a great way to get the word out and to enlist support from the Drupal community.”

Lucas added, “The European Commission provided generous funding for this initiative. The focus has been exclusively or largely around the European Commission’s features and functionality. The funding is running out very soon. There is a need for other people to help continue to build Automatic Updates by adding the features they need with their developers or by providing funding.”  

“It is critical for us to spread the message and make that call to action; that this is a community-driven effort and that without continued community support, it is not going to be as successful or as robust in the timeframe that we would like,” said Meyers.

The first year of funding from the European Commission provided for readiness checking, delivery of update 'quasi-patches,’ and a robust package signing system. The focus of this first phase of the Automatic Updates initiative has been on support for security updates in particular. 

In the second phase, as yet unfunded, we hope to extend this foundational work in the following ways:

• Provide more robust composer support. The first phase of the automatic updates project should be compatible with composer-ready sites, but as the site’s composer.json file and vendor directory of a site change from the default, then more controls and though need to be implemented. 

• Create an A/B front-end controller for the site being updated to further increase our confidence in the success of the update, allow for additional post-update testing and provide an easy mechanism to roll-back the update. This is also when updates will be able to move into Drupal core from the contrib project.

• Expand to more types of updates (particularly further support for contrib updates), and also handle multiple updates in a row, for sites that are several versions behind. 

To accomplish all of this, we will continue to seek more funding and more partners. 

“I’m looking forward to seeing where this goes now that we have the first release out, ” said Hedding. “ There’s a larger community needed to get this initiative completed.”

The initial alpha version of the Automatic Updates module can be tested by the community right now. The plan is to: demonstrate Automatic Updates at DrupalCon Amsterdam this month, complete the scope of the funded work by the European Commission by the end of this year, and stabilize Automatic Updates by DrupalCon Minneapolis in May 2020. 

“The Automatic Updates initiative is designed to reduce the friction in keeping a Drupal site secure and up-to-date. The team behind the initiative is architecting a robust system, secure by design, and building components that can be shared with the broader PHP community,” said Tim Lehnen.

Many thanks to MTech, Tag1 Consulting, and the European Commission FOSSA program for funding this initiative. The Drupal Association is proud to be a part of this initiative.