Drupal 10 roadmap
#3319030: Drupal 10 Core Roadmap for Automatic Updates
Initiative Overview
The goal of the Automatic Updates Initiative is to provide safe, secure automatic updates for Drupal sites. This is an essential strategic initiative for Drupal 9 core for two reasons:
-
Manually updating sites is difficult, time-consuming, and expensive, and the difficulty of updating Drupal is repeatedly raised as a top concern for using Drupal.
-
Delays between the release of a security advisory and the deployment of the corresponding Drupal update result in a time window where sites are vulnerable to exploit. Many sites do not apply updates for 2-4 weeks after the release of a security advisory, and site owners who want to ensure the security of their site need to have site maintainers available during the 1200 EST security release window (which is in the middle of the night for much of the Eastern hemisphere).
Both these factors increase Drupal's cost of ownership and decrease user confidence in the platform.
How to get involved
Join the #autoupdates
channel of the Drupal community Slack.
Meetings are held on the every other Tuesday at 1:00 PM EST in the #autoupdates
channel. Minutes from past meetings and agendas for upcoming meetings can be found in the Automatic Updates project queue.
Work is currently going on in 4 projects:
- The 8.x-2.x branch of the Automatic Updates Contrib project. This work will be port to Drupal core directly. Issues to be worked on can be found in the issue queue for 8.x-2.x
- The Github PHP-TUF library: this will ensure updates are correctly signed
- A Composer plugin to integrate PHP-TUF signing with Composer. This depends on drupal.org's packaging pipeline and Composer facade supporting PHP-TUF signing, which is being built under the supervision of the DA.
- The Github Composer Stager library: this library will enable Drupal core to stage updates before they are applied
Session recordings
- Nashville
- Seattle
- DrupalCamp London
- DrupalCon Global
- DrupalCon Europe 2020
- DrupalCon North America 2021 Initiative Keynote
- DrupalCon North America 2021: Technical Overview
Automatic Updates architectural spec/overview
Background: Personas
- Deploy and Ignore: Once the site has the functionality needed, there's little maintenance or updating. Doesn't subscribe to PSAs.
- Site owner who cares once a year: Deploy and ignore w/ an exception once a year
- Diligent but with Simple Needs: Typically applies updates within a week of release, possibly longer for non-security updates. Follows up on PSAs by directly updating the live site.
- The Sophisticate: Needs to apply at least one build step (for CSS, Composer, etc.). Runs QA in a pre-production environment. May deploy to a multi-head cluster.
-
Validation, readiness checks and notifications
The Automatic Updates module verifies that it is safe to apply an update to a site prior to downloading an update, and informs the site owner if updates cannot be installed automatically.
The readiness check API is being developed in the 8.x-2.x branch of the contrib project. See #3230045: [Meta] Create validation listeners
The readiness checks include (but are not limited to) the following:
- Is the site on the latest release of core?
- Does the site codebase match the official release? (I.e., no patches applied, no hacks to core, etc.
- Does the site have outstanding database updates?
- Is there available disk space?
- Are file permissions configured appropriately?
- [Still needed / not yet added] Is the memory available sufficient for updates to be installed? Issue: #3159920: Add a PHP Memory Readiness checker
In addition to the readiness checks, we're also adding display of critical security Public Service Announcements on the Drupal site status report and administration pages: #3041885: Display relevant Security Advisories data for Drupal.
-
Signing and verification
Signing of Drupal releases will be handled according to The Update Framework (TUF) specification. drupal.org will provide signed versions for Drupal releases and vendor libraries that will be needed for updates.
Core will use the PHP-TUF library which implements a client library. This library is currently being developed may be used by other PHP CMSes.The PHP-TUF client library will require Composer integration to verify the packages before they are installed via Composer. This is done by way of a plugin for Composer 2.1 or later: https://github.com/php-tuf/composer-integration. The plugin depends on TUF signing information being available for all requested packages, which is being added to drupal.org's packaging pipeline and Composer facade under the supervision of the Drupal Association.
-
Staged update and rollback
The update functionality in Drupal core will need attempt to run an update via Composer and be able to determine if the update succeeded. If the update was not successful the site should be left unaffected and in a working state. If the update was successful, the production site should switch to the new codebase with 0 or minimal maintenance mode window needed.
An earlier proposal for this was #3093093: Replace single index.php with a frontend controller that supports A/B updating (automatic updates).
However, we encountered some issues with that, and our current proposal is #3199171: Automatic updates: For MVP, instead of an a/b bootloader, implement a semi-atomic composer update with a brief maintenance-mode window.
-
Update installation
-
Composer-managed sites
Composer-managed sites are supported in the 8.x-2.x version of Automatic Updates contributed module. This 8.x-2.x version of the module does not support sites that are not managed by Composer.
The 8.x-2.x version of the this module is being prepared for inclusion in Drupal 10 core.
Per #2 and #3 above, https://github.com/php-tuf/composer-integration will provide the TUF integration and https://github.com/php-tuf/composer-stager will provide the staging and rollback support.
-
tarball-managed sites
In-place automatic updates for tarball-based sites for Drupal 7 are provided in the 7.x-1.x version Automatic Updates contributed module (development funded by the European Commission).
The 8.x-1.x version of this module which supported tarball-based sites for Drupal 8 is no longer supported.
[More detail here]
-
Roadmap
[Complete!] Phase 1
Phase 1 included developing a Drupal 7 and 8 contributed module with readiness checks, security public service announcements, and in-place manual and automated updates for tarball-based sites.
[In progress] Phase 2 (Initial core requirements)
Phase 2 of the initiative is to add the initial requirements for automatic updates to Drupal 9 core. The initial core module will provide automatic updates for:
- Core only
- Composer-built sites only
- Patch/security releases only
- Updates for the installed minor version
Phase 2 requirements:
[Outline and issue list here, based on most recent meeting]
-
Completed 🎉: Port PSA functionality to core
system.module
#3041885: Display relevant Security Advisories data for Drupal
-
Port the 2.x branch of the contrib module to core
This will add two new modules to core: Automatic Updates (
auto_updates
) and Package Manager (package_manager
). Package Manager provides the underlying API to create and manage a staging area in which to modify the Drupal code base, and will be used by the Project Browser module as well. Automatic Updates will implement the readiness check API along with several basic checks, plus the functionality to update core (and only core) in the UI or during cron.Core will also receive new third-party runtime dependencies, specifically
php-tuf/composer-stager
.The modules are being added to core in #3253158: [PP-1] Add Alpha level Experimental Automatic Updates module.
-
Composer support
[Not yet in scope] Phase 3 (Post-stable features)
The following features are not part of the initial core scope, but may be added in future releases once the core module is stable:
- Automatic updates for contributed modules/themes
Initiative Team
- dstol (Initiative Coordinator)
- david-strauss
- heddn
- mbaynton
- pwolanin
- xjm
- tedbow
- effulgentsia
- phenaproxima
Comments
Comment #2
hestenetComment #3
hestenetComment #4
hestenetComment #5
mherchelI want to chime in that I think this is an 💯💯💯 excellent idea! This feature definitely won't be for large enterprise level sites, but for smaller sites that have writable file systems. It's important for us to still cater to small sites because that's where the community comes from. Every long term Drupaler I know (including myself) started using small sites. Initiatives like this will help immensely.
I had some discussions over twitter on this, and people have expressed expected concerns. I think these could be mitigated by having this automatic updates disabled by default. We also need an extremely clear policy on what updates will be pushed, and maybe even have that be configurable (all updates, security updates, armageddon-level updates, etc).
Yay!
Comment #6
hchonovOne of the problems we have to account for is that the updates doesn't run in a specific order, which might break the site if not tested previously. The problem arises mostly when having contrib modules installed. We've recently encountered such a problem where we were unable of updating between major releases and had to update step by step - from minor to minor and from the last minor update to the next major.
Comment #7
bramvandenbulcke CreditAttribution: bramvandenbulcke commentedAt the Driesnote in DrupalCon Vienna there was a slide showing around 60% of users still use manual update procedures (FTP'ing tarballs). This huge number shows there is a large, mostly silent, group we should be aware of.
I would say that this initiative is vital for Drupal. For normal users, for sure. But also to prevent negative press when there will be big security hole similar to Drupageddon.
Of course, there should be an option to disable it.
Comment #8
anthes CreditAttribution: anthes commentedThe “auto updates for Drupal” discussion is going on for years now (see https://www.drupal.org/node/2367319 ). I’m happy to see a new “wave” of discussion and communication, let’s bring it to the next level. Hackers are used to automate scripts to outsmart us. Why shouldn’t we learn from hackers so we can improve and finally use our automation skills to fend off these attacks?
@mherchel, I want to add my thoughts regarding your opinion, that update automation won't be for large enterprise level sites - I’m part of a team, which provides a Drupal update automation management solution, also for enterprise customers.
Not only a few agencies automate Drupal updates (at least partly) with custom scripts already, that are executed by Jenkins or other CI tools. They face a common challenge: maintaining the bunch of scripts and keep on track with development of DevOp changes, new libraries, tools and workflows.
The lack of control of the update process is what concerns most DevOp teams and developers, so here’s one point which needs to be optimized so enterprise user will be happy with automatic updates as well. So yes, using update automation as an enterprise level Drupal shop sounds like a bad idea, if the update will be applied automatically without processing a whole pipeline with quality assurance etc. But that should not be the reason to exclude enterprises from update process optimization and increased website security, but to especially address this issue and find solutions for it.
So yes, automatic updates shouldn't be a default feature, every Drupal site owner should be able to decide about the way his or her site should be managed.
I absolutely agree with @mherchel, that a clear policy on update handling is needed. It's unlikely that a one-size fits all solution will work - that’s why a solution must be able to handle the different update behaviors, individual update workflows and security requirements of a user and a website. So, @hchonov is totally right, that the update pipeline needs to be managed within a specific test and time schedule system as well.
I’m looking forward to see more people joining the update automation discussion and exchanging their thoughts, expectations and concerns.
Comment #9
Mile23An obvious use-case not already linked here is #2367319: Implement automatic background updates for highly critical security issues
It's a bit of a chicken-egg type development issue because enabling auto-updates for only super-scary security issues could happen without whatever infrastructure requirements are needed for a more general-purpose updater.
Comment #10
bramvandenbulcke CreditAttribution: bramvandenbulcke commentedWe should talk after the 28th of March: https://www.drupal.org/psa-2018-001.
Comment #11
Mile23I just did some very very rough spec for #2940733-8: Site Builder Tool/Project Browser initiative
It seems that whatever build tool is generated for that issue would be used to generate a round-trip updater, particularly if one of the requirements here is that the user might download an updated tarball from d.o, generated to their spec.
#2940733: Site Builder Tool/Project Browser initiative seems like it could happen in the short- to mid- term. It could even create a tool that is used by the testbot to build the system under test.
Comment #12
pwolanin CreditAttribution: pwolanin at SciShield commentedComment #13
pwolanin CreditAttribution: pwolanin at SciShield commentedComment #14
giorgio79 CreditAttribution: giorgio79 commented"Many organizations have change management policies that would not allow this."
Like whitehouse.org that jumped the gun when a cheaper alternative came along? (Hint WP)
"Many Drupal-optimized hosting providers would not support writing new code on the webserver."
Really? How do they support Wordpress then? Drupal is a shrinking market size, so they wont last long.
"If the main beneficiary is build-and-forget sites on low-end hosting, does that justify a huge effort?"
Wow... Which site does not benefit from easier maintenance, and lower total costs of time spent on site updates??
All these questions seem to have been raised by nonsensical security "experts". We have them at my company too. Frankly, according to them the most secure solution would be to not have a website. The web is insecure, so have a backup. Period.
Comment #15
phenaproximaI'm not a security expert, and I believe that automatic updates are worth doing, but this is attitude is incredibly dismissive and naive. Security doesn't simply come down to "have backups in case you get hacked". If code can be executed remotely, for instance, the entire server is at risk, and not just the web site(s) running on it. Hell, it's possible that the entire network could be at risk. Depending on what network that is, the results could be downright catastrophic. Although it had nothing to do with Drupal, the Equifax breach (for one) came down to the company not taking security as seriously as they should have.
It would be extraordinarily irresponsible, bordering on downright mean, to just shrug our shoulders at the complex security issues presented by automatic updates. I think it's worth addressing those issues, but we need to do that in concert with the security experts. As you pointed out, the unfortunate reality of the Internet is that it's a dangerous place, and security is inconvenient for developers, but tough beans; that is the nature of web development. It's not fair to dismiss the security team's concerns and expertise. It's easy to scoff at security until you get hacked, your clients are freaking the hell out at you (in many cases, they could even take legal action against you), and you are in over your head and don't know what to do. Security breaches are the 11th circle of hell, and the security team deserves credit for trying to keep us all out of it.
So, if we're going to have automatic updates, let's figure out how to do them as securely as we can. I'm sure we can do it.
Comment #16
giorgio79 CreditAttribution: giorgio79 commentedOk, lets rephrase the question. Should Drupal be concerned with military grade security? If yes, the military better cough up the budget and not bog down Drupal development with excessive sec requirements.
Comment #17
phenaproximaLet's let the security team decide what level of precaution and security work would be appropriate for this initiative, rather than throw vague conjecture around.
Comment #18
dsnopekI suspect there are just as many plain technical challenges to make this work as security challenges!
However, to put the security challenges into perspective - the Wordpress auto-updater had a serious security vulnerability that could have allowed an attacker to deploy arbitrary PHP code to all Wordpress sites that have the auto-updater enabled:
https://www.wordfence.com/blog/2016/11/hacking-27-web-via-wordpress-auto...
It never was exploited, because it was discovered by Wordpress developers who followed responsible disclosure: it was reported to the Wordpress security team and fixed before the issue was made public. So, they got lucky. :-) But Drupal definitely wants to avoid a similar situation.
Comment #19
colanThe way I see it, there are several ways to go about this:
Comment #20
imclean CreditAttribution: imclean commented@colan,
This sounds like a good approach. Pantheon also supports this via git using upstream repositories, both within Pantheon and custom. There's a little bit of work to get the process to work with composer but perhaps that's where the effort should go.
Comment #21
mbaynton@pwolanin, you made some pretty big issue summary edits in #12 a few days ago that I think may need more discussion. Mostly I'm not sure about
Could you explain some more why you think this is necessary? Is your concern essentially about what badness might ensue if you attempt to update code that's used to perform the update itself?
Comment #22
Rob C CreditAttribution: Rob C commented2 should prolly be a separate issue (or even github project, this can support more then just Drupal). Got a library i wrote 5 years ago that's called by a git hook on the remote to deploy new releases. Supports wp and drupal7/8 atm and uses composer / drush make to build a new codebase during a release. If we split these issues we can dive deep into this concept and let the good people here fix auto updates via the UI.
Comment #23
SKAUGHTIf there were 'how to upstream' instructions, with the same idea more common now of 'using composer to build' that would be good for those who at least keep project repo's; would be nice.
But really, Core does need to support it as functionality. We can't assume 'we know how most sites run' on any server.
Comment #24
colan#22:
It's funny you mention that, because the whole point of Aegir 5 is that it will be able to host anything, not just Drupal. Instead of Drush, the back-end will be Ansible to start, and possibly Kubernetes later. This is enough of an aside though; sorry about that.
#23:
Right. Given that there are no fixed hosting infrastructures for Drupal at the moment, maybe we should reserve this issue for things that Core can do. Other hosting solutions (e.g. Aegir) can try to come up with solutions in their own projects. It's just that it would be nice if hosted applications didn't need to worry about this, but we're not there yet.
Comment #25
groovedork CreditAttribution: groovedork commented"2. Don't do this in Drupal itself; let the hosting infrastructure handle it."
The point of this project is to get people on shared hosting some security. I am one of those people. I don't even know what Aegir is, and I don't believe I should have to.
There was an idea (excuse my noobishness if I explain it wrong) to let the drupal.org server do the composer part. It would then tell the installation what files to download. Outsource it. That sounded like an interesting compromise: still have composer run the show, but not needing to have it on your own server.
Comment #26
Mile23@groovedork: That's #2940733: Site Builder Tool/Project Browser initiative where the basic idea is to let D.O build your codebase for you based on some specs.
Comment #27
effulgentsia CreditAttribution: effulgentsia at Acquia commentedFor additional background on these proposed requirements, check out the proposal from @pwolanin and @David Strauss at DrupalCon Nashville.
I disagree that their proposal is not feasible for a Drupal 8 minor. I think it's worth exploring whether it can be done without breaking semver, and with reasonable backwards compatibility.
Comment #28
hestenetComment #29
hestenetComment #30
hestenetComment #31
hestenetComment #32
hestenetFinally got settled after DrupalCon and updated the issue summary with notes from both the Automatic Updates Architecture core conversation (https://events.drupal.org/nashville2018/sessions/drupal-core-auto-update...) and the 'Hard conversation' conversation about this topic.
There seems to be broad consensus that this and several other similar initiatives are dependent on the same precursor task:
"Improving Core's relationship with Composer"
To be more specific, the initial tasks (which may have issues, or may still need issues created) seem to be:
Comment #33
anavarre@hestenet could you please make the GDoc readable by everybody? Currently it requires asking for permissions to read it.
Comment #34
Mile23The Composer initiative is here: #2958021: Proposal: Composer Support in Core initiative
Comment #35
colanSo should we postpone this one then?
Comment #36
MixologicI dont think so. I think the direction this is heading is 'create an updater for a composer based application' -> so something can be POC'ed and work can begin on it, with the aim that eventually core will be 'a composer based application' and will be able to leverage the updater. The other initiative will work in conjuction with this one, but doesnt block progress on this.
Comment #37
David_Rothstein CreditAttribution: David_Rothstein commentedAlong those lines, I think it's also important to think about what the minimum viable product is. I still think it's #2367319-119: Implement automatic background updates for highly critical security issues, which is definitely achievable in both Drupal 7 and Drupal 8. (Along with #606592: Allow updating core with the update manager to extend it to core updates, which is harder but should be achievable also.)
Composer support, support for rollbacks, etc., are all useful and important features too, but I don't think they're part of the minimum viable product simply because the Update Manager doesn't support those features currently, yet many sites still use it to perform (one-click/semi-automattic) updates.
Comment #38
wturrell CreditAttribution: wturrell commentedRe: milestone 2 schema update risk mitigation, would it be safer to also take an entire sql dump rather than just relying on transaction rollbacks, in case a problem only comes to light later?
(I like that the security updates without schema changes have been prioritised in 1.0.)
Comment #39
tatarbjLinks were broken in the summary, so i just fixed them. Also looking forward desperately to join this initiative when issues will be defined and sprint on them!
Comment #40
serundeputy CreditAttribution: serundeputy at Tandem for Backdrop CMS commentedComment #41
hestenetComment #42
hestenetAdded this to the issue summary based on conversations at #MWDS2018
There are three parallel efforts to be approached:
Though for more detail @pwolanin, @mbaynton, @webchick, and/or @David Strauss will need to chime in
Comment #43
wvd_vegt CreditAttribution: wvd_vegt as a volunteer commentedFor me the fastest way to update drupal8 is to
Additionally have a look at changes in .htaccess and robot.txt.
Except for the additional task, this should be easy to implement in php as well.
Comment #44
handkerchiefWith the recent security updates you have noticed again how fast you have to install Core Updates, before hacking scripts try to crack the website. I think at least these critical security updates should be able to run automatically. Maybe you should be able to select this option per instance, whether you want it or not. I know you've already planned it that way. With the recently released security updates it has become clear again how urgently we need such a function.
Comment #45
xjmAdding a link to the meeting minutes and agendas, and removing the "related initiatives" to the end since this section seems to confuse people.
Comment #46
xjmComment #47
xjmRemoving the old roadmap items (which were really just notes from the original core convo).
Comment #48
xjmComment #49
xjm(Porting over info from the MWDS purple roadmap that @mlhess, @pwolanin, @David Strauss, @Mixologic, @mbaynton, @webchick, @hestenet, and I collaborated on. Sorry if I've forgotten anyone.)
Comment #50
xjmAdding the section of the roadmap about completing and potentially forward-porting #2995202: Add a facility to update core in-browser that doesn't have core dependencies.
Comment #51
xjmComment #52
xjm(And finally adding the later phases.)
Comment #53
pingwin4egHappy New Year, @xjm, and thank you for updates! You're the real Santa! 🎅
Comment #54
c_estDick CreditAttribution: c_estDick commentedNot sure what now the status is, but I still be bothered by drupal 8 missing functionality:
Manual updates required
Automatic updates of Drupal core are not supported at this time.
The easiest fastest method to copy/migrate/core update/upgrade is using softaculous.
But softaculous has a delay in releasing their automatic installer so you will have a few days an extra security risc with core security updates.
Very pity that the functionality that softaculous delivers is not standard functionality in drupal.
It would be the best to implement devops and continuous integration and continuous delivery principles.
Very nice to have would be a fully automatic setup and maintenance and migration for a build street with environments for:
- Development
- Test
- Acceptance
- Production
(this would be a good start, since it might be later enhanced for branching, testing, ... , automatic testing/installation, ...)
This would be in my opinion a must have to facilitate developers, testers, ..... Their voluntary time is limited and any automation will help them spend their time more usefull.
For small sites would automatically create an acceptance environment by copying the production environment inclusive content and vice versa be super. It would deliver efficiently functionality to experiment with new plugins, updates, upgrades, test everything works fine at their server configuration before to release it to their production environment.
Now it is all to much effort for the less huge websites to do this with tools like git and jenkins and import export content with many scripts which are just customised for one specific production environment.
Why put effort this functionality in small (or medium) websites? Well they attract developers and other volunteers and their websites can grow huge and the less effort anybody needs to maintain a drupal site will boost drupal popularity.
Comment #55
heddnThere was a couple small informal gatherings at midcamp. Here's a summary of some of those conversations.
But first, we are making some progress over in #3039120: Create initial feature to display relevant PSA data in Drupal to build out some of the initial aspects of a PSA system. This is currently in contrib but there's also a core issue to take and move the features into the update.module.
Discussion points:
Comment #56
andypost@heddn is there option to ship core as phar file? I recall some core issue about it
Comment #57
heddnAre we referring to #2910136: Experiment: package PHP libraries in a single Phar file?
Comment #58
andypost@heddn yes, exactly! not sure about performance effects but I sure updating one file is much each then a set
Comment #59
AaronMcHale#58
Wouldn't Composer accomplish the same thing?
Comment #60
andypost@AaronMcHale there's no composer on live sites and it should not be there to build, delivery most of time made with git or tarballs to be atomic
Comment #61
AaronMcHale#60:
Ok sure not every live site will have Composer, some will, but yeah I can see where you're coming from where you have multiple environments for dev, test, stating and live. So if you're using Git for everything in Live, then would having core in a PHAR File really make much difference? What I'm really getting at is, would the investment of time and effort to make Core optionally become a PHAR File be worth it when (as far as I can see) there would be no change to the process or end result?
To be clear, I'm just trying to understand where you're coming from, not trying to be difficult
Comment #62
heddnFor the "how" of security updates, we'll be discussing that over in #3043235: [Policy, No patch] How to provide automated, in place security updates of Drupal core. There's already been some thoughts pulled together on that and over the next little while I'll start copying some of that into that issue. Let's take the phar or composer update, etc discussion over to there.
Comment #63
mbayntonComment #64
heddnI'm noticing that we haven't published/updated anywhere the scope of the sponsored work in this space. It has a limited scope that might under-impress some folks. But in other ways, it will accomplish a lot.
From this issue's summary, the 3 main areas that are in scope are A.4, A.5 and B.2
Comment #65
heddnA few things that one might be able to deduce from #64 is that there is no mention of composer. If composer is ready and able to work, we'll use it. But if it isn't, we can still do use update.module's features to auto update core. Turning on core's update in place mechanism is more a removal of protection code in update.module. It isn't particularly complex as discovered by backdrop. So, if we really, really want to use composer workflows with the funded auto update work in B.2, then we should spend a lot of time getting those external dependencies resolved.
Comment #66
heddnComment #67
cola CreditAttribution: cola commentedWe create a small dev module, which works with composer and drush 9. You can combine this i.e. with a shell script, drush 9 and composer to autoupdate Drupal with security updates. https://www.drupal.org/project/autoupdate
Comment #68
mbayntonWrapping up DrupalCon Seattle, we will be updating this issue's summary with the consensus stakeholders have reached soon. The current issue summary needs to be dialed in so it's clearer what the initiative is doing, and this is going to be a big edit. If this results in someone's ideas currently expressed in the issue summary being lost, I apologize, and please point out the omission.
Comment #69
dstolComment #70
dstolI trimmed down and clarified current scope and direction.
Comment #71
mbayntonComment #72
dstolComment #73
AaronMcHaleI posted this in the #autoupdates Slack channel yesterday, can't decide if it's in scope here but seems relevant and at least leaving it here might spark further discussion:
Comment #74
SKAUGHTAs an idea, sounds cool yes: a composer plugin could be made to replicate the module update page.
But on the level where one would somehow have composer check on its own to see if core/contrib/other_venders updates/status exist would be needed for a site that is built from drupal-scaffold.
Sounds like a good undertaking too. But from my experience with contrib lifespan issues -- deprecated/unsupported is already the flag for 'user be warned' if needing to use. Having a way for that to be marked on Projects and the information actually being kept up-to-date (if alt project exists) would be amazing.
--> if Project Maintains had access to that field on other projects would be key in that. Opposite to that: if a Project had a field for a 'simliar to' reference.
Another side of that are project issue where 'maintainer wanted' never is applied for, or 'applying to maintain' is never allow to maintain. which can be some reason why projects fall to deprecated or unsupported flags by core. Then others come along build new modules (duplication) cuz old project namespaces (the project repo) is locked.
Comment #75
NaheemSays CreditAttribution: NaheemSays commentedI have watched the drupalcon presentation on this and it was very informative, thank you.
In relation to the Develop A/B version support for providing updates this will be an awesome feature if/when it can be delivered. I do think a though its scope can be reduced a little.
With contrib I assume that database schema changes are more likely than core, so rolling back wont ever be a simple step. This could be lft out of it.
1. Only provide this support for core.
2. leave the (contrib) modules and themes folders alone to work the "old way".
3. Instead of using symlinks, a file can be written in the sites folder containing the currently active codebase which the index.php should use.
In relation to Robots.txt, afaik there was an effort to turn that into a service. If that is completed, it should no longer be a problem. .htaccess generally propagates down a folder structure, so it should be less of a concern, but maybe it can be simplified to delegate to another .htaccess in the live code site.
This would could provide the following structure:
index.php
.htaccess
modules
sites ( and allow specificity for multisite by allowing it to be copied to sites/sitename)
--> livecodebase.inc
themes
core
--a > with current/stale tarball structure minus the modules/sites/themes folders
--b > with current/stale tarball structure minus the modules/sites/themes folders
Which ever path is taken to achieve it, its an exciting prospect to have this feature.
EDIT
Just as an afterthought, instead of A/B, why not use the version numbers? I am guessing something like this will require testing before it can be widely deployed, so maybe create a separate atomic core release for testing?
Comment #76
dstolComment #77
colanRe: #73 & #74: We've been struggling with this issue in the Upgrade Status module via #2799369: Integrate with Contrib Tracker module. As of #303588: Show suggested replacement for obsolete modules, we've been hard-coding this data, but a better solution to help with these problems would definitely be a Replaced/Deprecated by field added to drupal.org projects.
Is there an issue open for this someone, or should we create it?
Comment #78
markmangoba CreditAttribution: markmangoba commentedComment #79
colanFollow-up to #77: I just created #3049420: Add a "Replaced/Deprecated by" field to drupal.org projects.
Comment #80
ressa CreditAttribution: ressa at Ardea commentedLucas Hedding posted this on Twitter:
https://twitter.com/lucashedding/status/1163527251176185863
Comment #81
heddnComment #82
grasmash CreditAttribution: grasmash commentedI'd like to circle back around to the issue of Composer usage for automated updates. There was a prior comment stating:
> If composer is ready and able to work, we'll use it.
It looks as though there's a good chance that Composer will be formally and well integrated with Drupal core by the 8.8.0 release. Notably, Drupal Scaffold has been moved into core and improved, and we're well on our way to completing #2982680: Add composer-ready project templates to Drupal core.
Given that, it seems as though Automated Updates should use Composer. Are there any blocking issues?
Comment #83
geerlingguy CreditAttribution: geerlingguy at Midwestern Mac, LLC commented@grasmash - One difficulty we will encounter (I presume) is the availability of RAM... a lot of Drupal sites, maybe even the majority of those which would use an auto-update mechanism (vs. manage updates via VCS / build process), might only have 256 or 512 MB of RAM available (total) on the system.
This already causes issues that I get pinged on quite often if someone runs
composer install
. Runningcomposer update
takes considerably more memory, and would likely run into lots of weird issues (e.g. someone has automatic updates enabled, but no updates ever get applied because the process fatal errors out, and they start filing issues on Drupal.org about it).Do we have an issue somewhere about finding ways to limit Composer's memory usage or maybe even do something like "don't enable if system memory is less than X"?
Comment #84
AaronMcHaleRe #83:
I like that kind of idea, maybe we could have a mechanism which attempts to use Composer but if it fails due to memory or other common issues the process falls back to another method.
Comment #85
andypostBtw using https://packagist.org/packages/zaporylie/composer-drupal-optimizations and removing dev dependencies allows to use composer on 512M limit for years for me.
Surely it could be improved to make predictable if not scalable
Comment #86
geek-merlin@geerlingguy:
> One difficulty we will encounter (I presume) is the availability of RAM
Good point. Others already solved it. The hard part is generating the lock from the constraints, and this can be done remotely by a service and for a vast number of cases it can be cached (Map hash(composer.json) => composer.lock). I have no link at hand, but Contao does this for quite some time and they serve a lot of low-profile installations.
Comment #87
geek-merlinHere some pointers:
https://docs.contao.org/books/manager/de/installation-manager.html
https://medium.com/@yanick.witschi/composer-cloud-resolver-e64254f5728e
Comment #88
MixologicContao's cloud resolver is functional because Contao is relatively small, usage wise.
Something the size of drupal would not be able to handle dependency resolution as a service due to the fact that when a critical security update is released, we would have to calculate *all* of the existing sites, all as fast as possible right after the release is announced. Basically we'd be building in a DDOS to drupal.org's infrastructure.
We can handle the current setup because of distributed caching of files. But if we had to actually do computation for every site, that would be a large, large load indeed.
Comment #89
NaheemSays CreditAttribution: NaheemSays commentedCan a resolver be written/done in javascript?
For the GUI update I suspect most users will have enough memory on whatever platform they are using the GUI from.
Comment #90
MixologicThat's an interesting proposal, but would require what is essentially a Javascript port of composer. If there were people who wanted to maintain that in parallel with upstream composer, that would be great, but the number of people with the skillsets, needs, time, and desire to take something like that on may be rather rare.
Comment #91
AaronMcHaleThat also wouldn't really deal with unattended updates, assuming that's something which is in scope here?
Comment #92
MixologicDefinitely true as well.
Comment #93
David Radcliffe CreditAttribution: David Radcliffe as a volunteer commentedComment #94
8thom CreditAttribution: 8thom commentedNoting here that Yanick from Contao has released a more generic composer cloud resolver -
https://composer-resolver.cloud/
more info: https://medium.com/@yanick.witschi/composer-cloud-resolver-e64254f5728e
Comment #95
clemens.tolboomComment #96
clemens.tolboomAdd link to contrib project
Comment #97
hestenetComment #98
webchickSurprising ABSOLUTELY NO ONE, in an informal survey about 9.1 features on Twitter the other day—https://twitter.com/webchick/status/1235689652838776832—this was an extremely popular option. (There will be a formal survey coming in the next couple of weeks; this was an attempt to gather up some options for it. :))
Comment #99
xjmPromoting to critical, because if anything is a critical feature, automatic updates is.
Comment #100
xjmComment #101
xjmSince the IS is basically 100% out of date currently, I've updated it with a rough outline for a current roadmap that we can populate with more detail.
Comment #102
xjmAdding session recording links for sessions that are already public.
Comment #103
xjmComment #104
effulgentsia CreditAttribution: effulgentsia at Acquia commentedI added #3160933: [META] Things to figure out in order to use Composer 2 to perform automatic updates as a child issue.
Comment #105
tedbowI know in the Drupalbon BoF we had marked the readiness checks and the PSA as issues we could get into core before the rest of automatic updates gets in but I am not sure if the Readiness Checks makes sense without Automatic Updates.
I have looked through all the checks https://git.drupalcode.org/project/automatic_updates/-/tree/8.x-2.x/src/... and I am not sure any of them make sense except PendingDbUpdates and there is already a check in system_requirements() that checks this.
We don’t know how people are going to update their sites if they aren’t going to be relying on future Automatic Updates.
Here are the existing ones in the contrib module
I would love to get functionality in core in pieces to avoid huge patches but I don’t think we should add checks that are going to cause confusion if we don’t actually Automatic Updates functionality in core.
At Drupalcon we talked about adding Readiness Checks and PSA to the existing Update module and adding the actual Composer-based updating to a new experimental module with the functionality eventually being merged into the Update module.
I think the Readiness checks should be added as the first part of an experimental module I don't see any benefit in adding them to the update.module first since the plan is to have an experimental module anyways and thing we are actually checking for is the readiness for functionality of the experimental module.
Comment #106
dww+1 to putting PSA reporting into existing Update Manager.
+1 to putting readiness checks into new experimental "Autoupdates" module (or whatever it'll be called).
At some point soonish, I'd love to have a conversation about the future structure of Update manager. In the era of composer, I'm wishing we had kept "Update status" as a separate module from the "Update manager" parts that let you install/update new code. Composer based sites might still want Update status enabled (e.g. to email them when security releases are available), although they almost certainly don't want the "update manager" parts. We have a settings.php killswitch to turn off the "manager" parts (
allow_authorize_operations
) but in hindsight it's a) poorly named and b) poorly advertised. Instead of adding yet more functionality to this beast, I wonder if it makes sense to keep things in separate modules to make it easier / clearer separation of responsibilities and easier to only enable the part(s) you actually want on any given site.Should we spin off another child issue for this?
Thanks,
-Derek
Comment #107
solideogloria CreditAttribution: solideogloria commentedI haven't read many of the comments yet, but is there any scope of what the automatic updates would be restricted to -- for example is it only security updates? Can you guarantee that manual core updates will always be an option to users?
I'm asking because a few years ago Microsoft began automatic updates for Windows 7 -> Windows 10 without asking people, which caused frustration and anger, and automatic updates also have the potential to be abused. It essentially makes it so that Microsoft is really the one in control, because they can push whatever code they want, whenever they want.
Comment #108
xjmComment #109
tedbowComment #110
xjmAdding the DrupalCon EU video now that it is public.
Comment #111
xjmComment #112
tedbowI am proposing a change in the order we implement features here #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow
The TLDR from that issue
More details and reasons on the issue.
Comment #113
effulgentsia CreditAttribution: effulgentsia at Acquia commentedRenamed the a/b bootloader section to account for #3199171: Automatic updates: For MVP, instead of an a/b bootloader, implement a semi-atomic composer update with a brief maintenance-mode window being the approach we are currently favoring.
Comment #114
effulgentsia CreditAttribution: effulgentsia at Acquia commentedUpdated the "Update installation" section. Removed the notes about Composer 2 being a requirement, because Drupal 9's existing "require-dev" for composer/composer is already for Composer 2.
Comment #115
effulgentsia CreditAttribution: effulgentsia at Acquia commentedI opened #3228567: Decide on consistent naming for future core modules that implement Automatic Updates and Project Browser functionality to discuss naming and would love some feedback on it.
Comment #116
tedbowAdded new recordings and updated slack meeting time
Comment #117
tedbowUpdate PSA issue for core
Comment #118
tedbowComment #119
phenaproximaAdding myself to the initiative team because I've been working on this for the past year. :)
Comment #120
phenaproximaComment #121
phenaproximaComment #122
tedbowComment #123
solideogloria CreditAttribution: solideogloria commentedThere appears to be a typo in the latest issue description.
(Emphasis added)
Comment #124
tedbow@solideogloria thanks, fixed I think
Comment #125
solideogloria CreditAttribution: solideogloria commentedYes, that looks right.
Comment #126
tyler36 CreditAttribution: tyler36 at Inter Quest commentedRecently, I opened an issue to automate database updates via composer action scripts. It was closed as a duplicate because of this initive.
It seems the focus here is to automate the process of checking for and applying updates to core and modules.
However, my request was to automate a manual step that is required after a developer manually updates. This would be required as part of this initive but is much smaller in scope, and could be introduced into core now; a win for the community and an initive that is 5+ years old.
Comment #127
Greg BoggsI believe this is the best issue queue to write about how to automate database updates:
https://www.drupal.org/project/issues/automatic_updates?categories=All
They may already have an issue for this.
Comment #131
xjm