Problem/Motivation

Updating Drupal core currently requires manually uploading core files or a complex environment such as Drush. Drupal's competitors, Wordpress and Joomla, allow core updates within the web UI. We should provide this, so novice users can easily keep their site secure and up-to-date.

Proposed resolution

Allow core updates to be installed the same way contrib updates are installed.

Remaining tasks

#538660: Move update manager upgrade process into new authorize.php file (and make it actually work)
#22336: Move all core Drupal files under a /core folder to improve usability and upgrades
#606190: Fix handling of database schema updates in update manager workflow
#509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages

User interface changes

None

API changes

None

Data model changes

None

Original report by dww

Update manager is now in core (#538660: Move update manager upgrade process into new authorize.php file (and make it actually work)), meaning end-users can upgrade missing updates (and install new code) via the web UI. Yay!

Unfortunately, making it able to upgrade Drupal core was just too complicated for the initial patch, and threatened to derail the entire effort getting into D7. Sadly, I think this is going to have to be punted to D8, but I'm opening an issue about it for discussion and so we have a stake in the ground for this effort in the future.

CommentFileSizeAuthor
#99 curator-606592-98.tgz614.67 KBmbaynton
#33 coreupdate-790224-41.patch4.77 KBxaav
FAILED: [[SimpleTest]]: [MySQL] 29,995 pass(es), 1 fail(s), and 0 exception(es). View
#25 coreupdater-606592-25.patch3.48 KBxaav
PASSED: [[SimpleTest]]: [MySQL] 30,033 pass(es). View
#24 coreupdater-606592-24.patch1.17 KBxaav
PASSED: [[SimpleTest]]: [MySQL] 30,020 pass(es). View
#12 upgrade_core-606592-42.patch24.43 KBJacobSingh
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch upgrade_core-606592-42.patch. View
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

Dave Reid’s picture

Subscribing...

Being able to upgrade contrib modules is the biggest pain and we've take a huge leap forward already. Realistically I can live with pushing core upgrades to D8 if we need to since core doesn't have releases all that often, and drush will probably figure out how to do it in the meantime. :)

Bojhan’s picture

Priority: Normal » Critical

Sounds pretty critical, that by default plugin manager can't upgrade its biggest use case?

Paul Natsuo Kishimoto’s picture

What precisely makes it complicated? I'm scanning the other issues in vain for a clue :(

dww’s picture

Right, sorry. Jacob and I started a conversation about this in IRC but neither finished, nor wrote up the partial results here. Here are some of the main tricky issues:

A) authorize.php runs in a reduced bootstrap, so broken contrib can't harm it, but it's still bootstrapping and running parts of core: (Form API, Batch API, system.module, user.module, maintenance theme system, language system, plus the FileTransfer and Updater classes). Updating those things via authorize.php while it's running is slightly scary and hard to get right.

B) We can't necessarily assume there's another directory higher up in the tree than the webroot, so we can't exactly move the current core out of the way, extract the new one in place, make sure it's working, and then switch over. Basically, recovery from a botched upgrade and rolling back is hard -- this is probably going to have to be destructive (simply overwrite your existing version with new files). If we auto-generate a backup of your existing files (our Archiver classes would need to support create, not just extract), and we write that to your webroot somewhere, we'd have to write it to your private/files folder to avoid the security risk of someone being able to download the thing, extracting your settings.php file and getting your DB credentials (not to mention all your files).

C) Related to (B), but slightly different -- although you're not supposed to hack core, people sometimes do. It's not clear how to detect these cases and what to do about it when we hit a locally modified file.

D) Since we can't just move your sites folder out of the way and then destroy everything else, it's not obvious what to do when core removes files between versions. It's easy to go through the extracted new version of core and move all those files into place, but what about the files that aren't in the extracted copy that were part of an earlier version of core? A few options we discussed:

  1. Core starts shipping with a manifest.txt file of some kind, listing every file included in the archive. This might be useful for #509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages anyway (since the manifest.txt can contain file checksums, and could be digitally signed). However, we don't yet have any infrastructure on d.o for such a file -- we'd have to change the packaging script to generate it, figure out where/how to present that info for each core release outside of the tarball (e.g. if someone wanted a copy and they checked out from CVS), etc. Only once that file exists can we put in the logic to use it, which would involve comparing the new manifest.txt we downloaded to the one in the current installation, and any files listed in the old manifest that are missing in the new manifest would be removed.
  2. When upgrading core, instead of using a whole new tarball release file, download patches and just apply them. This would require implementing patch in PHP (apparently something in PEAR exists that does that, I haven't looked), and again, lots of changes to d.o infrastructure. We'd have to auto-generate a patch file for every official release, attach the patches to the release nodes, include data about the patches in the release history XML files. If your site is missing a few releases, it'd have to know to download the patch for each release you're missing and apply all of them, etc.
  3. Ignore the problem entirely. ;) Upgrading core from 7.x to 8.x is completely outside the scope of this tool. So, if we're only talking about going from 7.x to 7.x+n, I don't think core is ever going to remove files in a point release along a stable branch. Sure, obviously, tons of files are added and removed between M.x and N.x, but never (?) between X.m and X.n.

E) It makes solving #606190: Fix handling of database schema updates in update manager workflow even more urgent. Even though schema updates from point releases of core are rare along a stable series, they can happen, and it's possible (though unlikely) that a schema change which changes some of the lower-level bootstrapping code might break authorize.php -- we update the code, but the schema hasn't changed, so now the code trying to generate the landing page to report about your update is broken with a fatal SQL error in one of the queries. :( We'd have to study how update.php handles this problem a little more closely, and then basically replicate that in authorize.php.

...

There are probably a lot of other smaller details, but this should already give people enough to chew on...

dww’s picture

I forgot to list option #4 for the removed files case:

D.4) Bake in some special-case mechanism in update manager to deal with this. We definitely can't add a system_update_N() that knows about this, since DB and code updates must remain completely separate (else we break the non-update manager case of people deploying sites for real with drush, aegir, etc). But, we could have some special file or other mechanism (TBD) that tells update manager that if it's upgrading core, it needs to remove certain files.

dww’s picture

JacobSingh’s picture

I disagree about B). I think assuming that the webroot is not the only dir the client has is okay. Perhaps for people with only one directory - has anyone ever seen this? backups would fail.

C). I don't think we need to worry about core hackers, just let them know that they will lose any changes they made

D). As the jack-ass who brought this up and proposed / researched the patch solution, I like Derek's idea more: Don't worry about it. At least for now, it's unlikely to be a problem. If need be, we can update our core updater probably with a minor update if we realize it will cause a problem later. Not ideal, but possible.

E). Is actually a larger problem than specified because we want to run updates in a separate batch process, not as part of the file moving page load (generally). So we would have to bake it in separately into the process.

I actually think the only ways to do E) with absolute certainty is to redirect to update.php immediately after the upgrade, to make sure it is run immediately OR run the updates in the same page request as the core update (which is probably cool, but if an index changes on the node table or something, it might cause an execution timeout or something).

-J

JacobSingh’s picture

Assigned: Unassigned » JacobSingh

Run for your lives!

nbz’s picture

Something to add here as a special consideration - preserving "special" files. I suspect many people modify the .htaccess file and some also robots.txt, so when the site is upgraded, such files will need to be somehow protected - one method could be to simply have an upgrade version of drupal that does not include such files...

JacobSingh’s picture

Okay, getting very close to a prototype here.

One problem we've got to solve though:

In update.manager.php around :425

foreach ($projects as $project => $url) {
      $project_location = $directory . '/' . $project;

This is kinda hacky to begin with, but when you DL drupal core, it unpacks into drupal-7.x-dev or whatever so it doesn't match up, unlike modules and themes. I'm thinking we should probably refactor this so that we are just grabbing the first directory out of our tarball and calling that the project directory. We then have to make sure we can track this throughout the project so we don't have to guess it based on the name of the thing we are updating later.

dww’s picture

@JacobSingh: Any more progress here? We're doing a Update manager code sprint today and it'd be really nice to be able to get this issue fixed. Can you post your work-in-progress prototype at least? Thanks!

JacobSingh’s picture

FileSize
24.43 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch upgrade_core-606592-42.patch. View

Hey Derek,

Sorry I'm getting this so late. I've got a million kids running around the house, work is just not going to happen.

Here is a patch I sent to someone who was having issues with copy-contents vs copy directory. The upgrade core patch also addresses this, so I thought it might solve his problem. I haven't messed with this at all in over 2 weeks, so it might not even apply, but much of the useful bits will.

Last time I ran it, it worked, although warning: you may blitz your install.

Best,
Jacob

mcrittenden’s picture

Sub.

int’s picture

me too

bsherwood’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, upgrade_core-606592-42.patch, failed testing.

joshmiller’s picture

Strange idea that might make "upgrade core" easier...

There is always an upgrade patch that fixes security holes, but does not include "additional bug fixes" -- what if we supported that kind of an upgrade using some sort of code that is capable of applying a patch to core files?

From the drupal.org announcement to upgrade: "To fix the security problem, you can either (1) upgrade Drupal or (2) patch Drupal. We strongly recommend you do a full upgrade (which is also detailed in the security announcement) as the patches do not contain the additional bug fixes that went into the releases. Applying the patches will leave your site in an unversioned state and confuse the update status module, which will keep reminding you to upgrade to 6.16 or 5.22. Please read the announcement for details on the patch."

Just a thought?

Also, #22336: Move all core Drupal files under a /core folder to improve usability and upgrades seems like it will be D8 and would make this much easier (as dww originally pointed out).

Josh

Bojhan’s picture

Version: 7.x-dev » 8.x-dev

After talking to webchick moving this to Drupal 8, simply to much work left to be done

catch’s picture

Priority: Critical » Major

Downgrading all D8 criticals to major per http://drupal.org/node/45111

xaav’s picture

Assigned: JacobSingh » Unassigned

1. I don't think this needs to wait until http://drupal.org/node/22336 is fixed. It's just an excuse, and this could be fixed w/out that.
2. Subscribe
3. I think simply extracting Archive.tar.gz/drupal-7.0 -> DRUPAL_ROOT would be perfectly fine.
4. I think we should make sure all files are is_writable before extracting the archive.
5. I'm unassinging this, as I don't think Jacob Singh is going to do any work on this issue.

But that's just my opinion.

xaav’s picture

Category: feature » task
AdrianB’s picture

Subscribing.

xaav’s picture

Bump...

xaav’s picture

FileSize
1.17 KB
PASSED: [[SimpleTest]]: [MySQL] 30,020 pass(es). View

Uploading a partial patch for this.

xaav’s picture

Status: Needs work » Needs review
FileSize
3.48 KB
PASSED: [[SimpleTest]]: [MySQL] 30,033 pass(es). View

This one is more improved.

dww’s picture

Status: Needs review » Needs work

Excellent, thanks for getting this patch started!

A) There are some code style bugs, especially in the CoreUpdater class.

B) There's no sense keeping all the "manual" updates plumbing if there's no longer anything that needs manual updating. ;)

C) This shouldn't be here:

-  $archive_errors = update_manager_archive_verify($project, $local_cache, $extract_directory);
+  $archive_errors = array(); //update_manager_archive_verify($project, $local_cache, $extract_directory);

We still want to verify the archive if we're upgrading core.

Also, this patch doesn't address any of the points I raised in comment #4...

Anyway, this is a totally legit start, but "needs work" is definitely the appropriate status here.

Thanks!
-Derek

aspilicious’s picture

Subscribe! Drupal 8 needs this!

xaav’s picture

+ $archive_errors = array(); //update_manager_archive_verify($project, $local_cache, $extract_directory);

Yes, that was kind of a hack (update_manager_archive_verify() was saying that it didn't have a .info file in it).

I'll try to work on this more today.

pillarsdotnet’s picture

delta’s picture

Sub

xaav’s picture

A) authorize.php runs in a reduced bootstrap, so broken contrib can't harm it, but it's still bootstrapping and running parts of core: (Form API, Batch API, system.module, user.module, maintenance theme system, language system, plus the FileTransfer and Updater classes). Updating those things via authorize.php while it's running is slightly scary and hard to get right.

We could do this:

1. Copy all system modules to a tmp backup dir. Verify integrity of ea. copied module immediately after it is copied.
2. Exit and go to the next iteration of the batch.
3. Copy all includes to a tmp backup dir. Verify integrity of ea. copied include immediately after it is copied.
4. Exit and go to the next iteration of the batch.
5. Copy the new authorize.php, then copy modules and includes from archive location into place. Copy the ones that authorize.php uses first. Verify integrity of ea. copied file/module after it is copied. If we run out of time, authorize.php will be able to run on the next iteration of the batch.
6. Copy the rest of the files.

The idea is this: if something fails, authorize.php will still be able to run and revert changes/ finish the job/ etc. Since we're upgrading core, if anything goes wrong, we should abort immediately. We may also calculate beforehand (maybe in step 3) if the server is fast enough to copy the files required by authorize.php before the script times out.

xaav’s picture

In fact, we could use get_included_files();, then copy those files from the archive first.

xaav’s picture

Status: Needs work » Needs review
FileSize
4.77 KB
FAILED: [[SimpleTest]]: [MySQL] 29,995 pass(es), 1 fail(s), and 0 exception(es). View

Here is a little further.... I still can't seem to get drupal to pick up CoreUpdater.

aspilicious’s picture

Status: Needs review » Needs work
+++ b/modules/system/system.updater.incundefined
@@ -144,3 +144,55 @@ class ThemeUpdater extends Updater implements DrupalUpdaterInterface {
+    public function getInstallDirectory() {
+
+        return DRUPAL_ROOT;

you can leave out these blank lines, they are not needed.

There are a couple of them in the code

Powered by Dreditor.

toddmbloom’s picture

It would be lovely to have - I haven't used Wordpress' version to update their core but the plugin updater works well.

The only concern I would have would be that it should definitely save the current files to a /tmp directory before an update.

webchick’s picture

Category: task » feature

Would still love to have this. A huge win for security.

However, this remains a feature request, not a task.

jddunn’s picture

This feature is vital for devs who ever want to be able to hand off a site built with Drupal, train the client to maintain it, and be done with it. I build stuff with Drupal for community organizations and the like who don't have the budget to have a full-time IT person, and you can usually find and train a generally computer-literate person to do module upgrades or other basic in-interface tasks, but that leaves core un-updated and eventually vulnerable. And so I eventually have to go back and help basically everyone I've ever built a site for, which is just not sustainable in the long run. This really, really needs to get fixed in 8.

ontological’s picture

Agree entirely with jddunn in #38. One of the main reasons to use a CMS is to allow this hand off.

webchick’s picture

No one disputes its importance, but unfortunately people who need it arent stepping up to work on it. :( if this is an itch you possess, then best thing to do is dig in and see how you can help get the ball rolling with a patch, or some funding for someone else to take it on.

shad1981’s picture

Is anyone working on this feature? - Upgrading core with the update manager automatically without Drush or manual upgrades?
I'm sure there are many people out there (including me) that will be happy to fund this.
How much will it cost? and who wants to take this job?.
I'm sure that all of us together can make it happen.
I guess not in time for Drupal 8 release but it could be added later.

dww’s picture

After Drupal.org is running D7 (my current top priority), if no one else has solved this in the meantime, I'd probably be interested in working on this if people want to fund my time. I'd have to spend a few hours I don't have right now re-assessing what it would take to do this before I can give you a guess as to how much it would cost. "Not cheap" is my initial approximation. :)

Thanks/sorry,
-Derek

shad1981’s picture

Thanks for your reply.
Maybe others here want to help me fund this feature?

philbar’s picture

If we can get it into 8.x (even through a contrib module), I'd fund some of this.

jhedstrom’s picture

Version: 8.0.x-dev » 8.1.x-dev
Issue summary: View changes

Will need to be 8.1 at least I think.

MediaFormat’s picture

Issue tags: +d8ux

Allow me to revive this by boldly paraphrasing Dries, from drupalcon Barcelona:

To improve the User Experience (UX), Drupal needs to:

  • Focus on the non-coders: builders, authors, and editors.
  • Improve the UX, making it easier to use and more seamless.
  • Focus on the 80% of the world that does not use a content management system, and convince them that Drupal is right for them.

D8 needs this!

jcnventura’s picture

Yes, it does and we can probably create a feature branch to start work on this. There are a few core updater modules out there already.

But it certainly won't get done and approved in 7 days, so it will have to wait until something later than 8.0.

MediaFormat’s picture

There are a few core updater modules out there already.

Aside from Aegir and Drush which are rather like projects, I am not aware of any other modules.

If you know of any others please list them here.

jcnventura’s picture

CMS Updater (https://www.drupal.org/project/cms_updater)

And maybe: Drop Guard (https://www.drupal.org/project/dropguard)

Yes, these are all commercial gateways that only update your site if you paid for the license. That being said, I'm pretty sure that the update code should be somewhere in the d.o GPL module.

MavMan’s picture

Hi,

Can someone, please, enlighten me over my doubt: This feature would mean that Drupal would be able to also Upgrade itself via WebUI (e.g.: D8->D9) pretty much like Joomla (something like Notification to Admin About New Version>Press Button "Upgrade">Upgrade in Process>"Upgrade done"), right?

(If so, this would be Security's Paradise for me while also being able to have the latest features - of course this upgrade option would only be available to Admin)

MediaFormat’s picture

This feature would mean that Drupal would be able to also Upgrade itself via WebUI (e.g.: D8->D9)

This would likely be for minor versions D8.1->D8.2

MavMan’s picture

:( That's sad.

If Drupal is able to upgrade between minor versions why not between major versions?!

And that old story that the site Admin's has to know what he is doing, test the site before upgrade and that, if he's the Admin, then he'll find it easy to manually upgrade it, does not invalidate this. Really, it doesn't stick!!!

There is absolutely no point at having to do this manually except that, if this is done manually, it will scare newbies away (it's more of a tool to separate us from newbies).

I think that Drupal does not need to scare anyone away... But maybe that's just me...

I'm sorry, i do not intent to criticize anyone (specially not you MediaFormat). i'm just tired of this mambo-jambo talk about the "it's better to do in manually" old talk...

jcnventura’s picture

@MavMan first, Drupal is not currently able to update between major versions. There's some migration support, but it's clearly something that shouldn't be done without extensive testing. To do it from inside the site would in 99.99999% of all cases result in a broken site. Simply because D9 will break compatibility with D8 (that's why it's called D9 and not simply 8.x+1), and any modules using old APIs will stop working. You'd need a site running with 100% Drupal core and 0% custom or contrib code or themes.

Upgrades between 8.0.x, should be possible (as these will be bug fixes and security releases), and maybe an opt-in to enable updates between 8.x versions.

As to security paradise.. Please realize that to have this possible, your server will need to be able to re-write your PHP files. If you lock down Drupal, that shouldn't be possible, and no attacker should then be able to inject whatever code they please into your site. This is clearly not 'security paradise'. I'd call it 'security limbo'.. You'd be able to upgrade your site by pressing a button (or even automatically), but your site would be always vulnerable to code injection.

MavMan’s picture

Hi jcnventura,

Then how are WP and Joomla guys doing it...?

How can they do this and we can't???

jcnventura’s picture

They are embracing "Security limbo" and making people bask in the warm embrace of the god of cluelessness.

But I honestly agree that 8.0.x->8.0.x+1 and 8.x to 8.x+1 should eventually be done. Drupal 8 is not capable of doing this in core. No doubt there will be contrib modules implementing such services, as there are for Drupal 7 already.

It would probably be one of the first settings to be disabled by larger agencies, but for small clients running on hosts that don't allow them to harden their sites, this would be better than nothing.

philbar’s picture

I started a GoFundMe campaign for this issue.

There's a catch 22 for this issue. The people who are able to code this are technical enough not to need this feature. Those who would benefit greatly from this feature are not technical enough to code it. I'm hoping funding will resolve it. I put $20 in. Anyone else want to contribute?

Feedback from developers is also welcome.

MediaFormat’s picture

Great initiative, thank you philbar!

MavMan’s picture

@philbar

To clarify: The feature being funded is destined to allow updates (e.g.: 8.1->8.2) or Upgrades (E.g.: 8.x->9) ?

Tks

philbar’s picture

@MavMan, the funding will be tied directly to this issue. It depends on the core developers.

Keep in mind, Drupal 8 starts a shift in development cycles (#2135189).

Drupal Core Development Cycle

philbar’s picture

Issue summary: View changes
philbar’s picture

Issue summary: View changes
jcnventura’s picture

@MavMan: please understand that you need an AI-level system to upgrade between D8 to D9. It's not a question whether such an update may or not break your site. It's GUARANTEED to break it. Core update, if ever, will only be between minor versions.

And the $200 funding just out of the blue? This is something on the level of a feature branch.. Just the discussion on this thread has already used more than that..

philbar’s picture

Issue summary: View changes
philbar’s picture

I've decided to end the funding campaign and refund the money, since there wasn't the support I was hoping regarding the funding campaign.

This was more of an experiment for an idea I thought could take hold in the Drupal community.

I donated $20 and someone else matched it. $40 isn't much incentive and there is no guarantee this issue would even be fixed within the year since this issue is over 5 years old. I don't want to be responsible for managing the funding without more certainty this model would be applied to other Drupal issues. I thought GoFundMe would hold the funds indefinitely, but they have rules were they don't retain funds longer than 30 days (to avoid banking regulations, since they don't want to be a bank). The the funds ended up mixed with my personal funds. I can keep track of it in the short term, but I don't want to manage it long term. Because of this, the funds have been refunded and the campaign ended.

delta’s picture

I think composer and the drupal-template in 8.x kind of resolve this issue. (you can update the drupal core with composer). If it's just a matter of upgrading code and database: composer update && drush updb will do the job.

However, IMHO upgrading minor (and major..) version always need to be done first on a testing environment because every drupal site is kind of unique and a bug can throw only on combination of custom code, contrib modules, server configuration, etc..

So..., a "safe" automated update can be done only on an industrialised drupal product (with good tests and their own custom update-testing-deploy script). Even in that case a testing environment would be used to try first the deployment of each version...

Regarding composer update && drush updb, upgrading core look like upgrading a composer library today, and the upgrade manager, that is asked here to be responsible of the core update, is for managing the module of the "drupal" library...

(If you look at Jenkins for example, update of core are shell commands :O, updater/manager of the plugins are inside the UI.)

Maybe this issue is asking the wrong question. Maybe it's more a question of support for a safe way to update drupal core :

just my 2 cents..

dww’s picture

If you have shell access and are comfortable with composer, you can already use drush for this. The feature request is to be able to do this from *within* the Drupal web UI, not the shell.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

fatkinson’s picture

I was wondering what was going on with the auto-update feature in Drupal 8?
I heard that this was going to be a feature when Drupal 8 was released.
I was very disappointed after installing Drupal 8 when it was released to find out that the feature was not included. So I've done nothing more with Drupal 8.
I get so fed up every time I have to upgrade revisions of Drupal 7 every few weeks as the updates come out. Which is why I was looking for the Drupal 8 being released.
I was told that the update feature would come 'later'. Presumably, that meant it would show up in a Drupal 8 update.
Haven't heard a word since then.
An update would be nice.

MakeOnlineShop’s picture

@fatkinson EXACTLY the same feeling here.

Wasting time updating something that should be done is so ridiculous that I hope to leave Drupal soon, and I guess that we are not alone !

But maybe this is what they want, or they would have done something smart 10 years later.

Drupal was ok 10 years ago but they must be really stupid not to have improved the update process 10 years later...

I know that some are passionate about technical useless things, but some others are just interested in using drupal because it is powerful, not in having to cry every time an update is needed !

cilefen’s picture

@fatkinson @MakeOnlineShop: Where did you hear this was a planned feature of Drupal 8? I have been involved with the project for a few years and I did not ever hear of it.

Drupal 8 can add new features in point releases, so something like this could be added before Drupal 9. For my own self, I do not want the site code to be writable by the web server so I may never use this.

chx’s picture

Oh sure we had discussions for this feature stretching back a decade or more. We could extend the classloader to support loading from phpstorage for fun and profit.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

MavMan’s picture

What's the state of this feature?

Is it abandoned??? We're kind of "waiting" for this to migrate to Drupal...

MediaFormat’s picture

David_Rothstein’s picture

I don't see why this issue would need to depend on #509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages.

Looks like it was added to the issue summary in https://www.drupal.org/node/606592/revisions/view/9121946/9122132, but with no explanation as to why.

MavMan’s picture

If it doesn't depend on it... then it's inclusion is only delaying the feature. Why not just get on with this?! It's a very important feature (even if it only upgrades between minor versions, it's better than nothing...)

mbaynton’s picture

I've been working on it for a month or two now. I'm more interested in providing an avenue for sites to opt in to receive fully automated security updates (#2367319: Implement automatic background updates for highly critical security issues), but being able to update core over http is pretty much a prerequisite for that.

fatkinson’s picture

I was told at Drupalcon in Austin that this feature would be in Drupal 8.
I almost jumped for joy when I heard it. When Drupal 8 was released, I immediately installed Drupal 8 on an inactive Web site I owned and logged in to find out that update button wasn't there. When I made a post about it, I was told it would be included with a future release.
I stopped development of Drupal on my sites because I spend more time upgrading firmware to my various sites.
I do not plan to use Drush because of the arrangements I have with the hosting provider I use. So core updates through Drush is not a solution that would appease me.
This was the single most important improvement to Drupal as far as I was concerned. Without it, Drupal 8 fell short.
I suspect others feel the same way. There is just too much time spent updating the core.

MavMan’s picture

@fatkinson - i feel the same way!

webchick’s picture

I have absolutely no idea why someone would've said this feature would be included in Drupal 8. This feature was never touted on any official communication.

There also seems to be some pretty big misunderstanding in recent comments of the "do-ocracy" nature of open source projects such as Drupal. There is no company/organization/dedicated workforce behind Drupal to complain to when things aren't moving fast enough. The developer community here is a collection of individuals who work on what they are motivated to work on, whether that's because it's a particular problem they have, because they like the fun of the challenge of it, or because they're being paid to work on it, either via their boss or via a customer.

So if you care about any particular feature getting done, and it's not getting done fast enough to your liking through the open source community, then the way forward isn't to make demands of others to work on it (this tends to demotivate people from working on your problem). It's to put some of your own skin in the game, either with code, with $, with mockups, and so on.

Since @mbaynton seems motivated to work on this, a good first step might be finding out what help he might need and see where you can pitch in.

fatkinson’s picture

Regardless, it is a feature that is badly needed. I am not going to spend hours updating firmware on all of my Web sites every time there is a security patch or a bug fix. And I suspect others feel the same way!
Are the developers going to continue blindly down the path they are?
Or are they going to recognize the need and act!
For me, Drupal development is on hold!

Dom.’s picture

@fatkinson: this is exactly the kind of reply why you could read #80 again :)

fatkinson’s picture

I read it, sir!
And my statement remains unchanged.
If they don't develop this, people like me are going to continue to drop Drupal.
I tried to develop a module for Drupal 7 once.
I spent a great deal of time and money developing a module for Drupal.
I did evertyhing they asked me to do.
Every time I finished their required changes, they saddled me with more required changes.
That module was never completed and is still sitting on the Drupal site unfinished.
So my coding is not good enough to help them since I could never satisfy them.
Nevertheless, I won't be using Drupal much longer if this this doesn't happen.

mbaynton’s picture

I'd be happy to provide more details about my project if I got any offers for assistance. There's a public git repo (30 days behind my private one already?!? :O), and a docker image that runs the unit tests and, as of a week or so ago, even some integration tests against an FTP daemon. Help is helpful.

WorldFallz’s picture

If they don't develop this, people like me are going to continue to drop Drupal.

@fatkinson -- Respectfully, but I don't think you have read #80 a second time. There is no "they". There is only 'we'. Please, read webchick's comment again-- then maybe even a third time. She's already said it so clearly and succinctly it doesn't need to be repeated in a different comment.

This is the type of hyperbolic threat people often make to commercial entities when they've 'paid' physical money for a product that doesn't meet their needs. In an open source community, this type of statement will very often garner nothing but negative reactions on the part of those that actually contribute to the project.

The analogy I like to use is that if one of your neighbors mowed your lawn for free, would you complain to them that they didn't edge or mow low enough for your liking (most people would consider that ungrateful or even extremely rude)? Or would you simply thank them and either finish the job yourself or hire someone to do it the way you like it?

This really is no different. You're using and benefiting from the work of others for free.

If Drupal doesn't meet your needs then by all means find a product that does. Unlike a commercial product where you very likely have made a substantial financial purchase, there's nothing to keep you using Drupal other than it does what you need it to do.

Or, even better, come aboard and help make it the product you need it to be! Everyone is welcome and everyone is able to contribute in whatever way they are able.

fatkinson’s picture

Am I benefitting?
It's creating too much maintenance work which should be automated.
I am sorry you took it as a threat. It was meant to let you know what I consider important in Drupal development.
You can think of it as a threat or as an opportunity.
Every time I turn around, I am getting notices that I need to install updates. When I do, it takes a great deal of my time.
I am at the point where I ignore the messages any more because it is just too much work.

tobias-hd’s picture

A possibly completely stupid question of a non-php programmer:

Since drush also seems to be written in php, and provides ready-to-use update functionality for core, isn't it an option to copy/port "drush pm ..." functionality to drupal, and "just" build an UI for that?

I understand that there is an additional webserver in between - does this "break" drush coding for drupal?

Tobias

fatkinson’s picture

You obviously didn't read the comments I made about my hosting provider.
I don't get shell access as part of the package. If I want it, I have to pay extra for it for *each* Web site that I want to add it to.
Others are in the same boat.
This feature is badly needed.

mbaynton’s picture

Thanks for the periodic reminders that there's a need to write some more code to this end, @fatkinson. This is definitely the demographic/use case I hope to help with. But no open-source coding on my birthday is my rule I think :). I will try to channel the energy later on...

sarmiliboyz’s picture

From my point of view, Drupal is segmented for mid-end user on this feature (people who can get shell access, or people with knowledge/not novice user, or people/organization with more money to develop and maintain Drupal). For newbie people or "low budget" organization, Wordpress is still more interesting than drupal since they can upgrade the site core from UI, even Wordpress can automatically upgrade itself when the admin is away.

Having a core update on the admin UI is a big advantages on Drupal site since it will attract more newbie or low-end budget organization, and with that, Drupal user will be more increasing. It will be a long term investment for Drupal.

But since Drupal is a community project, and there is no company to complain for the bug/feature request. I think it will be better if we can make a crowd funding for a feature request that people ask to the community so there will be people interest to develop the request. Either the funding is backed on Drupal Association or on the 3rd party service is no problem for me. So if people really want to add some feature while there is no progress, no people working on it, create a funding and make the feature happen.

fatkinson’s picture

Understand.
I think Drupal has a lot of potential.
But this is a big obstacle.
It's one I would like see Drupal overcome.

MavMan’s picture

I, personally, am holding back on implementing Drupal until this is solved...

But once i get to the point i can't wait no more it won't matter how much better or secure Drupal is... i'll be heading the Joomla way just because of this feature.

This is not a critic... it's just for developers to have an idea of how much important this feature is for me... and how much important it can be for others...

fatkinson’s picture

Over the years, I have heard from my service oriented employers that if one customer is complaining that there are hoards more customers that are silent but just as unhappy.
When multiple customers are actually complaining, multiply that number by the number of customers complaining to realize how many unhappy customers you actually have.

cilefen’s picture

Issues like this one disclosed yesterday underscore that doing this properly is difficult, and can potentially make all participating sites hackable.

The vulnerability we describe below may have allowed an attacker to use the WordPress auto-update function, which is turned on by default, to deploy malware to up to 27% of the Web at once.

Hacking 27% of the Web via WordPress Auto-Update

MavMan’s picture

But i'm not in favor of the feature being enabled by default.
I truly do think it should be off by default and warn anyone who tries to turns it on.

But it should, ultimately, be a user choice.

(and i don't see wordpress getting any less users because of this kind of disclosures. In fact, has far as i know, wordpress continuous on getting more and more users. (even bad publicity, is still publicity)

David_Rothstein’s picture

This issue does not propose adding auto-update functionality, so I think the last two comments belong on a different issue?

One-click updates do have some risk too, of course, but the reality is that updates.drupal.org is already a pretty big target since there are contrib modules (e.g. Views) that are installed on the vast majority of Drupal sites and that can already be updated via one click in the Update Manager, so adding one-click updates for core on top of that is not going to increase the risk significantly. (That is why I suggested above that #509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages is not a prerequisite for this issue.)

cilefen’s picture

dww’s picture

Issue tags: +composer

@mbaynton re: #80: That's great you've been working on this! THANKS!! If you post a link to your public git repo in a comment (or better yet, update the issue summary) someone might help. Having been intimately involved in getting the Update Manager (such as it is) into core in the first place, and ostensibly the maintainer of it for a while, I might be able to take a look and at least answer questions, bounce ideas, give you my opinions, etc. Maybe I'd even be inspired to actually help write some of this code again. ;)

@fatkinson re: #93: Yes, for everyone loudly and repeatedly complaining that other people aren't doing enough free work for them, there are hundreds or thousands of others with the same opinion who aren't speaking up. That doesn't make people who might work on this feature (like mbaynton or me) actually want to work on it any harder. Please understand that people who are donating their time to contribute *really* don't like being told we're not working hard enough for the ones who are anxiously waiting for us to find some "free" time to solve their problems.

@webcheck: First of all, hi! :) Sorry I disappeared. I miss interacting with you (and many others around here).

Re: #80: Very well said! Mostly true. ;) However, it's also a little misleading. There are a lot of people who get paid part/full time by their boss(es) to work on parts of Drupal, including core, including yourself. ;) While companies like Acquia benefit from having a large user-base and community around Drupal, and while this feature would probably help expand that user-base, it's not *directly* in their financial interest to provide *this* feature. If it was, they would have funded someone to work on it long ago. Not to pick too much on Acquia, but they're either interested in having small-time clients pay for Acquia hosting (in which case this feature would be disabled) or they want bigger clients that are going to hire Acquia as consultants for how to architect a large/complex site (in which case there are going to be serious sysadmin resources that will be upgrading/deploying Drupal via the shell). Same goes for basically every other Drupal shop with enough resources to fund core development -- this simply isn't a problem those shops need to solve for themselves or their clients. In some cases, solving this problem is actually a threat to their bottom line, since maybe people wouldn't pay for a hosted solution like Pantheon or Acquia if it was easier to upgrade Drupal directly from itself.

Given that, back to @fatkinson, you're therefore barking up the wrong tree. You're expecting the commercial entities that have the resources to fund this to operate against their own financial interests. That's not how capitalism "works". (Personally, I don't think it "works" at all, but that's another story). So, *this* issue has to be treated as a true Open Source effort, where it's simply done out of the goodness of the hearts of people who believe in it and want to share their efforts for the greater good. Therefore, you have to actually contribute to a positive atmosphere of support, camaraderie, gratitude, encouragement, and Getting Things Done if you want to see progress here. There's no company you can complain to like a disgruntled customer. If you keep acting like that, you're only going to piss off/scare off/alienate the (very small) community of people who might actually solve this some day. You may think you're telling us things we don't already know, but you're fooling yourself. We (in the larger sense of the people who contribute to Drupal, regardless of how/why) know there's a need for this feature. We know you're not the only one who thinks so. We know it's in fact a major pain point for lots of people. But we all have to pay the bills, and landlords (especially in the Bay Area, where I live) don't accept "I'm doing a bunch of work for the benefit of a huge number of people, even though none of them can/will pay me for it" in exchange for our (exorbitant) rent. If this issue isn't going to pay my rent, one of the only remaining motivations is that it's fun and rewarding. But having to answer to comments like yours makes it neither. Definitely not fun. And it actually can lead me to believe (although this is very wrong) that the only people who would reap the rewards of all that effort are entitled jerks who expect me to do free work on their behalf. Thankfully I know you're just a vocal minority acting this way, and *TONS* of people would be genuinely grateful and happy if this worked. I hope you join the ranks of the helpful, or at least grateful. And even if you don't, I hope that mbaynton, myself, and anyone else positively contributing to this issue remembers that we're not doing it for the fatkinsons of the world, but the vast numbers of people who will greatly benefit if we get this working.

Cheers,
-Derek

p.s. Given how likely it is that we're going to want to solve this by basically having authorize.php be able to run composer for us, given that there's a group of composer enthusiasts/experts who pay attention to that tag (I think), and given that I still don't personally consider myself enough of a composer ninja to proceed down that path without their help, I'm tagging this "composer" since I hope we Do The Right Thing(tm) and I think both this issue and the composer team (whatever that means) will benefit from cooperation and communication in here.

mbaynton’s picture

FileSize
614.67 KB

@dww, cool, help would definitely be appreciated if you do feel so inspired and do not find my approach to solving this too wild and eccentric. Whether or not you decide to jump on board, perhaps this is a good time to give an update on what I'm up to.

As I've said before, my motivations for doing this really revolve around #2367319: Implement automatic background updates for highly critical security issues, so the direction I'm going is substantially informed by a desire to reach that goal. I also believe (as evidenced by the extensive dialog in that issue) that large portions of the "securely update a PHP web application" task turn out to be nontrivial but also turn out to be essentially the same thing whether one wants to update Drupal, WordPress, Joomla, MediaWiki, ... So, rather than having every project reimplement a solution, what I'm doing is more One Updater To Rule Them All than a patch that depends on Drupal. I wrote some additional details about exactly how I see that looking here.

Ultimately, the architecture I envision is my updater component -- I've taken to calling it "Curator" -- packaged as a single .phar that lives on webservers alongside the sites that wish to be updated. The meat of the .phar's functionality will be accessed via API endpoints, enabling project (e.g. Drupal) infrastructure to push automated updates out (more on why I believe pushed updates are superior to periodic polling here), but also allowing site owners to perform manually-initiated updates in browser by hitting the same endpoints from an AngularJS user interface that the .phar conveniently also serves up.

With that 10,000 ft overview in mind, here's the github: https://github.com/curator-wik. There are two main repositories:

What has been accomplished:

  • A filesystem access abstraction layer a little different from Flysystem or plain-old stream wrappers in that it allows operations to be routed to different I/O mechanisms depending on whether they're reads or writes, while always giving the rest of the app a single path to refer to a file. This is basically to avoid performing all my reads through something slower than a mounted filesystem just because I may need to perform my writes through ftp/ssh/etc. That stuff lives in the curator repo under dist/src/FSAccess.
  • A way to do some basic key/value persistent storage by leveraging the filesystem access and writing it out to a file -> dist/src/Persistence. I eventually do imagine a Drupal module that works with the .phar to provide a tighter integration, including defining a db schema for persisting settings that way, but priorities.
  • Getting the GUI off the ground and getting it embeddable into/served out of the .phar.
  • Lots of rigging up of tests and CI setups.

Immediate next priorities as I see them are:

  • Implement a batch processor conceptually similar to Drupal's.
  • Start defining/implementing APIs! A number of them don't need batch processing, like inputting file upload method and credentials, so this can happen now.
  • Start hitting the APIs with the GUI!

I fixed a few bugs yesterday and successfully built the project from source trees into a .phar using the instructions on my README.md's, on a few different computers. Only gotcha I'm aware of is that to do all the scss compilation and other build steps for the GUI you need a newer node.js than some linux distributions' defaults. You should also be able to stick the .phar attached in the .tgz on any PHP 5.4+ server and "run" the thing, such as it currently is. If the server is not configured to interpret .phar's as php, you can use the attached .php file to get around that.

dww’s picture

Cool, thanks for the update!

First reaction: "wow, that's different." ;) Sounds like you're trying to reimplement composer, among other things.

I'm also not sure how push notifications from Drupal.org will work -- is d.o supposed to maintain a database of all known Drupal sites and how to contact them? My privacy-minded side freaks out at the prospect of that. I know many site owners will not opt-in to a system where Big Brother is storing connection details about every Drupal site on Earth. How does each site authenticate that the push is actually coming from d.o and not a forged attack? What happens if the network link between d.o and site X is down at the time d.o wants to notify? Does that mean it's d.o's problem to remember every site that got the notification and keep retrying the ones that missed the memo?

Furthermore, I'm quite skeptical that you're going to get all these different systems to support One Updater To Rule Them All(tm). So, if you over-abstract the problem into a general solution that only one organization adopts, you're probably going to have more pain for no gain.

All that said, maybe we need a different place to discuss all the implementation details of your proposal, since I fear this issue is quickly going to mushroom into 500 comments if we try to do it right here. Maybe create a sandbox project (so you'll have your own issue queue) and we can continue over there?

And *that* said, I definitely applaud your ambition to try to solve this (nasty) problem! Hurray for thinking big and outside the box. No reason to lock ourselves into a solution that continues down the path we already started, if an entirely different path will work better in the end.

Cheers,
-Derek

mbaynton’s picture

Sounds like you're trying to reimplement composer, among other things.

I hope not. Unless/until Drupal and similar software add Composer to their system requirements and make it a necessary part of creating a new installation, I don't see Composer's functionality as essential. Of course, as a developer, I recognize the benefits of building a site with Composer, but probably most of the people doing it that way are going to prefer the CLI to get stuff done anyway.

I don't think I'd be opposed to finding a way to integrate Composer's core services into Curator eventually, but it shouldn't be necessary to allow upgrading core in-browser.

I'm also not sure how push notifications from Drupal.org will work

I'll choose not to open that can of worms just now. It's a subject more appropriate for the other d.o issue anyway, and much has been said about it already over there. I spent quite a while exploring the questions you raise last July/August, and I feel there are satisfactory answers to most of your questions, for most people.

Furthermore, I'm quite skeptical that you're going to get all these different systems to support One Updater To Rule Them All(tm)

Could you elaborate on the problems you foresee? The way I see it, CMSs/wikis/forums/etc all get upgraded by

  1. Cause the updated source files to be written to disk where the webserver interprets or serves them
  2. Redirect the user to that app's in-browser post-update runner.

...and if they don't support getting upgraded using that procedure, then Curator probably won't work.

For #1, it can be surprisingly hard, but how is it different from one app to another?
For #2, you make map of (app => path to update.php), and write some sniffing of files/directory structure to determine which app is being updated. That sniffing ought to be the extent of the code that targets specific applications/systems.

Thanks for sharing your perspective @dww, it's important to have multiple points of view.

dww’s picture

Unless/until Drupal and similar software add Composer to their system requirements and make it a necessary part of creating a new installation, I don't see Composer's functionality as essential

That seems to be the direction we're headed, there are people who advocate for that, and issues like this are one of the reasons it hasn't yet happened. If authorize.php and install.php ("securely") invoked composer for you, there'd be little reason not to let it handle a bunch of our problems for us, even if end users didn't have to know.

It's a subject more appropriate for the other d.o issue anyway, and much has been said about it already over there.

Do you mean #2367319: Implement automatic background updates for highly critical security issues ? Ok, time permitting (!) I'll read through that monster thread, too. ;) I remain skeptical, but I'm curious to see what you propose.

Could you elaborate on the problems you foresee?

Client-side, I don't foresee many problems at all. Your assessment there is accurate.

I'm talking about server-side: the infrastructure needed by each software project to support push notifications, and/or publish release metadata in a format your tool can understand, etc. E.g. whatever happens at updates.drupal.org to make your thing work. I have high confidence we can make changes to updates.drupal.org if/when necessary, but I don't personally have any inroads to get similar changes to happen for any of the other things you propose to be able to update with your system. I suppose folks like Crell are on various committees/bodies that attempt to coordinate and standardize various things, at least in the PHP world, so maybe you could get some traction that way. But ultimately, you're going to spend a *lot* of time convincing each project to adopt your solution, agree to make infrastructure/server changes to enable it, implement those changes, test them, maintain them, etc.

Meanwhile, it seems Composer has become the de-facto standard for at least some of this stuff (e.g. how to advertise your release metadata, and existing infrastructure that maintains a database of releases that anyone can query to try to find the latest versions of things, where to find a copy, etc).

I suppose I'll reply at #2367319 with my thoughts on push notifications for this, although ultimately, my point is we need an issue in the drupal.org infrastructure queue about this (unless lots of infra folks are already participating in #2367319).

Thanks,
-Derek

mbaynton’s picture

@dww, thanks for giving my approach more consideration! Sorry, yes I meant #2367319: Implement automatic background updates for highly critical security issues, I'm surprised it wasn't already marked related!

I am of two minds on the Composer issue. On the one hand, I think composer should be regarded as a tool that is used by developers to create releases, not as a tool for release deployment. In a monolithic system (vs. one composed of modules like Drupal), that would be the end of it in my mind -- wrong tool for the job. On the other hand, in the context of Drupal, identifying a workable set of external dependencies and their versions that satisfies all modules on a given site might really need Composer's talents, as more and more sites/modules are built that way.

Regardless, I don't think downloading composer packages should be the only way for Curator to source the updated code that it can install. In the interests of making visible progress easier to achieve, I'd like to stick to reading more straightforward upgrade packages that contain the literal code to be patched for now. All the file management and batch processing infrastructure I'm working on should be useful to any future efforts to make Curator Composer-aware.

Regarding adoption by d.o. and other projects: you'll see when you do get a chance to look over #2367319: Implement automatic background updates for highly critical security issues that I made a valiant effort to get buy-in on a specific approach before it actually existed, and that more or less got nowhere. So, I am forced to go with the "build it and (hope) they will come" approach. I think the task of writing server software to perform en-masse pushed updates to a registered set of sites is a sensible next step, though, because once that software exists, hashing out the details of its deployment at one place or another becomes much more achievable. You wouldn't argue that drupal.org and wordpress.org need to each have their own custom http server daemons because they have to serve drastically different content; both projects instead figured out how to use existing software to meet their needs. Once a pushed update server is existing software, my hope is that it would similarly be adopted.

mbaynton’s picture

And, I should add, while it'd obviously be my preference for projects like Drupal to eventually provide releases officially through Curator, the entire thing could be sustained without official buy-in. The software doesn't care who is running the servers. Consequently, it'd be totally reasonable for not only the in-browser core updates but also automatic updates to be fully functional contrib things for a while and just see what happens.

David_Rothstein’s picture

I may have some time to work on a new patch for this issue during the next couple weeks. It's not guaranteed (depends on what client work might come up) so if someone else wants to look into it in the meantime, don't let me stop you (just leave a quick note here saying that you're working on it). But hopefully I will be able to.

This is the type of hyperbolic threat people often make to commercial entities when they've 'paid' physical money for a product that doesn't meet their needs. In an open source community, this type of statement will very often garner nothing but negative reactions on the part of those that actually contribute to the project.

The analogy I like to use is that if one of your neighbors mowed your lawn for free, would you complain to them that they didn't edge or mow low enough for your liking (most people would consider that ungrateful or even extremely rude)? Or would you simply thank them and either finish the job yourself or hire someone to do it the way you like it?

I'm not sure that's a fair analogy. In recent years the Drupal Association (which is supposed to represent and grow the community) has solicited and spent a lot of money directly on core development. Questioning how that money was spent seems reasonable to me.

A better analogy might be if you live in a neighborhood that has a common area, and the neighborhood has a homeowners' association that helps maintains the common area but individual residents also volunteer their time to improve it. Complaining to your neighbors about their volunteer efforts is rude, but questioning how the homeowners' association spends its money isn't.

In any case, I personally agree with everyone else that this is a very important feature that Drupal needs to keep the majority of its current users happy, as well as to attract new users - so I'd like to see it happen :)

David_Rothstein’s picture

Issue tags: +needs backport to D7

On the technical side of things:

  1. I agree that what @mbaynton is working on is interesting, and also that it's beyond the scope of this issue (since it's basically about replacing the entire Update Manager, whereas the feature proposed in this issue is possible to add with the current Update Manager). So I like the idea of a separate issue and/or sandbox project.
  2. Similarly, @dww:

    Given how likely it is that we're going to want to solve this by basically having authorize.php be able to run composer for us...

    It is not obvious to me why you see that as a possible requirement. I actually filed a related issue a long time ago (#2538090: Allow the Update Manager to automatically resolve Composer dependencies) but the goal there was to allow Update Manager to be able to install modules that have Composer dependencies (i.e. that require code beyond what is hosted on drupal.org). However for Drupal core, all the code is included in the tar/zip file that is downloaded from drupal.org, so it seems to me like we should just be able to use that file.

  3. I agree with #87 that it would be useful to look at how Drush does this. I am particularly curious if they deal with the possibility (first mentioned in #4) that the new version of core could have deleted files compared to the old version, or if they just ignore that and unpack the tar file right over the existing installation.

    I agree with others above (#4, #7, etc) that if ignoring the file deletion problem winds up being the simplest technical implementation, it is fine to go ahead with that. Within Drupal 7, as well as within an individual branch of Drupal 8 (e.g. within Drupal 8.2.x) I think we can guarantee that no file deletions will happen (or at least no important file deletions will happen). That's probably not true between Drupal 8 branches (e.g. Drupal 8.2.x => 8.3.x) because of external library updates, the possibility of removing "experimental" modules, etc. However, a core update feature that can be used for everything except branch jumps within Drupal 8 (which only happen every 6 months) would still be very useful.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

DigitalFrontiersMedia’s picture

I agree with #87 that it would be useful to look at how Drush does this. I am particularly curious if they deal with the possibility (first mentioned in #4) that the new version of core could have deleted files compared to the old version, or if they just ignore that and unpack the tar file right over the existing installation.

I believe Drush makes a backup because I've had Core updates fail before for various reasons and, if I recall correctly, it rolled it back to the previous version. I don't think it does this once it gets to the database updates; once those have occurred, it's up to you to roll back to a database back-up and a previous commit to get things back if the database updates have screwed something up.

DigitalFrontiersMedia’s picture

I've written a module (https://www.drupal.org/project/dfm_autoupdater) that implements a shell_exec() that could be used and/or modified to do this via UI though it requires SSH access on the host and the shell tools of choice for updates, e.g. drush, composer, git, etc. Not sure if this is 100% applicable here but maybe it will add to the discussion/ideas. This topic (updating Drupal), in general, has been a long-standing interest of mine (and obviously many).

mbaynton’s picture

I've been continuing to work on this issue, and have been focused for the past few months on a goal of getting my code to perform a complete core upgrade, at least inside an integration test by 1 month to DrupalCon. I am happy to say that with commit 9181e8 yesterday, I've succeeded in that goal.

There are still important elements to be developed, but the project has long since passed the point where there's any risk of my becoming bored and abandoning it. This work will definitely result in an avenue to performing in-browser core updates, and it will be sooner than later.

Although it's not strictly "Drupal development" in that I can't point to a component or file of Drupal's that I'm enhancing, and in that I ultimately hope it could be useful to projects that aren't Drupal as well, I hope others with an interest in this nearly 8 year old issue will be open to taking a look at the momentum I've built as a path of least resistance to getting this done. A guiding principle of the project is that it doesn't require any additional server system requirements beyond Drupal's.

In the month remaining, I'll try to make it more presentable for informal demos to any interested parties in Baltimore. If you follow this issue and would like to see it, let me know and we'll schedule a time to meet. If you won't be at DrupalCon Baltimore, I should also be able to post interesting-looking demos online by around that time as well. In particular, the complete upgrade integration test demonstrates that the server-side code required to affect an upgrade via HTTP requests is there & working, but I still need to write the JS to replicate it via in-browser UI. No UI = not much of a demo.

A few cool technical elements I've incorporated include a framework for batch processing of tasks that's built from the ground up to support concurrent execution, this is used to write out files or run ftp uploads for several changed files simultaneously. We've also made real progress on making the update package be a set of patch files plus delete and rename information, instead of shipping complete files in the update data. My colleague has developed a utility in C linked to libgit2 and some related libraries that generates the patch data between two git commits in a patch format that is more easy to work with in php code. It mostly works but some diffs still give it problems.

The user experience I'm going for is for a user to be able to see an available core update through existing Drupal administrative UI, with some added buttons/links on that page that transport them away from Drupal-handled routes to my code, which gives them basically a progress bar while it performs the code modifications. It then sends them back to Drupal's update.php.

fatkinson’s picture

This will be a big step in the right direction!

cilefen’s picture

MavMan’s picture

Finally!!! The "Final Missing Piece" to make Drupal really the best!!!
Thanks @mbaynton for this.

I have a question, however: To clarify What kind of upgrades will this be able to do? "8.x -> 8.y" or "8.x -> 9.0" ?

Will be watching this bug report closely :D

mbaynton’s picture

@MavMan, you've probably seen Making Drupal Upgrades Easy Forever, where it says

Drupal 9.0 should be almost identical to the last Drupal 8 release, minus the deprecated code.

Based on that, 8.x -> 9.0 sounds like it might even be doable, assuming

  • My project (Curator Web Install Kit) becomes and remains enough of a concern by the time there is a 9.x for me and others to be keeping its associated D8 module up-to-date with the core APIs.
  • The same is true for all the other modules on your site, and you can update all of them to ones that'll be able to handle a 9.x core first as well.
  • You have a pinch of luck.

Who knows, by then, maybe Curator will even have some update rollback capability...

quietone’s picture

I noticed this issue because it has 'upgrading core' in the title. But when I skim the issue this seems to be about updating from a minor version to another minor version and not upgrading a major version. See Updating and upgrading Drupal core. I've been reviewing the upgrade documentation recently so getting the words correct is on my mind. So, is this issue about update, upgrades or both?

MavMan’s picture

@mbaynton I was not aware of that blog post!!! Thanks for sharing. :D

It does look very promising, but... will all those upgrades be doable thru a native Drupal Web GUI (like in Joomla/Wordpress)??? Or to have a GUI we'll need Curator Web Install Kit???

MavMan’s picture

@quietone AFAIK this issue is about upgrading (which ends up including updates. If it can do upgrades, for sure updates will be easier). So, i think the answer is actually "both". But i'm not an expert... so i could be wrong...

fatkinson’s picture

Any progress on this issue?

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

fatkinson’s picture

Any progress on this issue?

dww’s picture

There's been more recent discussion in the issue marked as related to this:

#2845379: Do not leave non-experts behind: do not require Composer unless a GUI is also included

Please follow (but don't start spamming) that issue if you're interested.

Cheers,
-Derek