Minor release to update core?

According to the documentation for hostmaster-migrate:

The command [...] will fetch the latest stable Drupal release, so it can simply be run again when a new security release of Drupal is made available.

So we shouldn't need to provide minor releases of Aegir (although it would be nice); folks can simply run this command.

However, as per #2607972: How to cleanly update hostmaster ? and the attempt I made yesterday, this doesn't actually work. I was on Drupal 7.42 before, and on 7.42 after, even though the command should have moved me to 7.43.

So either:

1. This worked previously, but doesn't now, so we have a regression, or
2. This never worked, and the documentation is wrong.

To answer #3 (thanks helmo), this only works if on unstable. Stable will fetch the original core release.

I don't think #4 is a good solution given that it just broke my system:

[...]
Deploying site from /var/aegir/backups/aegir.dev-trusty.example.com-20160505.175437.tar.gz                                                     [success]
Replacing the existing site at /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com                                           [success]
Created aegirdevtrustyex database                                                                                                              [success]
Successfully extracted the contents of /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore                           [success]
chgrp(): No such file or directory FileSystem.php:448                                                                                          [warning]
filegroup(): stat failed for /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/files FileSystem.php:227            [warning]
Could not change group ownership of files in /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/files [warning]
to www-data (chgrp to www-data failed on /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/files)
chgrp(): No such file or directory FileSystem.php:448                                                                                          [warning]
filegroup(): stat failed for /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/private/files FileSystem.php:227    [warning]
Could not change group ownership of private files in                                                                                           [warning]
/var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/private/files to www-data (chgrp to www-data failed
on /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/private/files)
chgrp(): No such file or directory FileSystem.php:448                                                                                          [warning]
filegroup(): stat failed for /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/private/temp FileSystem.php:227     [warning]
Could not change group ownership of temp files in                                                                                              [warning]
/var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/private/temp to www-data (chgrp to www-data failed
on /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore/private/temp)
Swapping out the /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com.restore and                                             [success]
/var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com directories was successful.
[...]
Platforms path /var/aegir/platforms is writable.                                                                                                   [success]
'drush' cache was cleared.                                                                                                                         [success]
The drush command 'hosting-resume' could not be found.  Run `drush cache-clear drush` to clear the commandfile cache if you have installed new     [error]
extensions.
Drush was not able to start (bootstrap) the Drupal database.                                                                                       [error]
Hint: This may occur when Drush is trying to:
 * bootstrap a site that has not been installed or does not have a configured database. In this case you can select another site with a working
database setup by specifying the URI to use with the --uri parameter on the command line. See `drush topic docs-aliases` for details.
 * connect the database through a socket. The socket file may be wrong or the php-cli may have no access to it in a jailed shell. See
http://drupal.org/node/1428638 for details.

Drush was attempting to connect to:
 Drupal version         :  7.43
 Site URI               :  aegir.dev-trusty.example.com
 Database driver        :  mysql
 Database hostname      :  localhost
 Database port          :  3306
 Database username      :  aegirdevtrustyex
 Database name          :  aegirdevtrustyex
 PHP configuration      :  /etc/php5/cli/php.ini
 PHP OS                 :  Linux
 Drush script           :  /usr/local/bin/drush
 Drush version          :  8.1.0
 Drush temp directory   :  /tmp
 Drush configuration    :  /var/aegir/hostmaster-7.x-3.4-7.43/sites/aegir.dev-trusty.example.com/drushrc.php
                           /var/aegir/hostmaster-7.x-3.4-7.43/sites/all/drush/drushrc.php /var/aegir/.drush/drushrc.php
 Drush alias files      :  /var/aegir/.drush/platform_hostmaster.alias.drushrc.php
                           /var/aegir/.drush/test.dev-trusty.example.com.alias.drushrc.php
                           /var/aegir/.drush/test2.dev-trusty.example.com.alias.drushrc.php
                           /var/aegir/.drush/server_localhost.alias.drushrc.php /var/aegir/.drush/hm.alias.drushrc.php
                           /var/aegir/.drush/server_master.alias.drushrc.php /var/aegir/.drush/platform_hostmaster7x34743.alias.drushrc.php
                           /var/aegir/.drush/hostmaster.alias.drushrc.php
 Drupal root            :  /var/aegir/hostmaster-7.x-3.4-7.43
 Drupal Settings File   :  sites/aegir.dev-trusty.example.com/settings.php
 Site path              :  sites/aegir.dev-trusty.example.com

And then if I try surfing to the site, I get:

PDOException: SQLSTATE[28000] [1045] Access denied for user 'aegirdevtrustyex'@'localhost' (using password: YES) in lock_may_be_available() (line 167 of /var/aegir/hostmaster-7.x-3.4-7.43/includes/lock.inc).

Going to try the following instead:

1. Copy the hostmaster platform to a new directory.
2. Delete the hostmaster site within it.
3. drush pm-updatecode drupal
4. hostmaster-migrate to new platform

Hopefully that'll work better.

Turns out that wasn't necessary. This is what I came up with, and it proved successful:

• Become the Aegir user.
• Upgrade Drupal core via Drush.
• Verify the upgraded platform.
• Verify the Hostmaster site.

I added the details of this to the docs. Please review the commit.

Maybe we can simply recommend this process instead of pushing out minor updates? It's still a bit risky, but it's less risky than running with a known security vulnerability. And I'm quite sure that Drush maintains its own backups so these can be restored if there's a problem.

Agreed. I didn't even realize it worked that way. 3.1 should always be the same!

For reference, here is the issue where we switched to using the distro tarball: https://www.drupal.org/node/2002114. I'm still astonished that distributions need new releases to track each Drupal core release.

I've asked for guidance on best practices from the d.o infrastructure team: #2726767: Best practice to update core version in distribution?. Let's see what they have to say.

Do note that our distro has a no-core tarball: https://www.drupal.org/node/2665636. This appears to bundle all the contrib components, and so would effectively pin all versions.

I don't see any convenient way to use this in Drush Make, but being a co-maintainer, I could help build out a proper solution for this.

In the mean time, perhaps we could just deploy the no-core tarball as a library, using some combination of download, subdir, and directory_name and/or destination: http://docs.drush.org/en/master/make/#library-options?

FYI: https://github.com/drush-ops/drush/issues/2189

That feature request was simpler to implement than I had thought. The next version of Drush 8 should include it. Basically, it will allow us to do:

core: 7.x
api: '2'
projects:
  drupal:
    type: core
  hostmaster:
    type: profile
    do_recursion: false
    variant: projects

Of course, that's the newer YAML makefile format, but I think that'd be a worthwhile conversion too.

At any rate, that will use the hostmaster-7.x-3.4-no-core.tar.gz tarball, built atop the latest Drupal, which should resolve this issue cleanly.

I've updated the change to Drush such that the [do_recursion] = false is implicit when [variant] = "projects". As a result, only the variant parameter is required. Since that'll be ignored by older versions of Drush Make, it should be safe to use it, so long as we don't add do_recursion = false. So:

core = 7.x
api = 2

; This makefile fetches the latest release of Drupal from Drupal.org.
projects[drupal][type] = "core"

; The release.sh script updates the version of hostmaster.
projects[hostmaster][version] = "3.4"
projects[hostmaster][type] = "profile"
projects[hostmaster][variant] = "projects"

I added an issue in the docs repo to fix the documentation for core upgrades before an Aegir release. See https://github.com/aegir-project/documentation/issues/22.

@helmo: What's a good way to test #19?

Sounds good to me.

Confirming that the latest Drupal core was installed from the unstable repository.