fix release process and scripts [#2002114]

Problem/Motivation

Since we have split hosting in a separate module (#1912134: split hosting in its standalone module), we have some issues with the release process. We basically need to update the versions of the hosting and eldir module in the drupal-org.make file by hand at every release. That is time consuming and error prone.

Proposed resolution

Current situation

aegir.make: specifies drupal core, chains into hostmaster.make
hostmaster.make: pins hosting and eldir to the git tags, downloads from git, includes drupal-org.make
drupal-org.make: details all contrib modules, pins hosting and eldir versions
build-hostmaster.make: specifies core, includes drupal-org.make

Proposed fix

aegir.make: specifies the "hostmaster distribution" as core for releases, specifies core and includes hostmaster.make on git
hostmaster.make: pins hosting and eldir as from git, includes drupal-org.make
drupal-org.make: pins all contrib modules except eldir and hosting
build-hostmaster.make: specifies core, includes drupal-org.make

Notice how in the new way, no makefile in "hostmaster" will need any pinning. The reason this works is that we make the eldir and hosting releases before hostmaster is released - i *think*.

We could have two makefiles in provision, aegir-dev.make would be like the current one, and aegir-release.make would be like this:

core = 6.x
api = 2

projects[hostmaster][type] = "core"
projects[hostmaster][patch][] = "http://drupal.org/files/common.inc_6.28.patch"
projects[hostmaster][version] = "6.x-2.0-rc5"

Then release.sh could just flip a aegir.make symlink between the two files and fix the version in aegir-release.make?

Remaining tasks

Make the changes in git, test a release?

User interface changes

None?

API changes

None?

#2023113: [meta] 2.0 release

Original report by [username]

One thing that could help us is if we could specify the distribution version within the aegir.make file. RIght now this is not possible because we can't replace core in Drush make 5. But drush make 6 allows you to use a distribution as "core", so we could use the tagged release or the nightly builds there. Then in turn we wouldn't need to specify release numbers at all in the makefiles, as they would get built as snapshots on drupal.org in the right time.

But this requires drush 6.

Also, this wouldn't work for dev builds, since the development builds of distributinos are only nightly right now on d.o. I talked to marvil07 about this and apparently this should be fixed with the D7 deployment of drupal.org, so we could simply use tarballs all the time.

Comments

Comment #1

anarcat CreditAttribution: anarcat commented 24 May 2013 at 18:47

some more explanation of the problem

13:52:18 <@anarcat> ergonlogic: i reverted the alpha2 hardcode in hostmaster.make
13:52:31 <@anarcat> so from now on, aegir alpha2 installs will actually install git head :/
13:52:39 <@anarcat> i really need to get a hold of the drush people
13:52:48 <@anarcat> we're now 3 people at the aegir/drush sprint
13:52:54 <@anarcat> i suspect hangover
13:53:24 <@anarcat> ergonlogic: oh, and i think we should add a build-hostmaster.make after all...
13:53:29 <@ergonlogic> anarcat: ok... I thought alpha2 was in drupal-org.make
13:53:49 <@ergonlogic> and hostmaster.make had the git overrides
13:53:51 <@anarcat> ergonlogic: it's actually useful to have that stuff to rebuild platforms by hand for testing purposes
13:53:53  -!- Rob_C__ is now known as Rob_C
13:53:57  -!- MrScumbag <lsw@conference/drupalcon/x-nnrdgyqswvynrqui> has joined #aegir
13:54:01 <@anarcat> ergonlogic: hostmaster.make has git overrides, but they were tagged
13:54:09 <@anarcat> because aegir.make sources hostmaster.make
13:54:14 <@anarcat> remember? :)
13:54:16 <@anarcat> this is all fucked up
13:54:22 <@anarcat> i really need to find that drush 6 patch
13:55:00 <@ergonlogic> I'm still confused about all that
13:55:10 <@anarcat> so here's the thing
13:55:12 <@anarcat> we got that silly aegir.make
13:55:16 <@anarcat> that thing sets up core
13:55:28 <@anarcat> it can't use the d.o distribution, because drush 5 make is broken (fixed in drush 6)
13:55:34 <@anarcat> so it downloads core, then downloads hostmaster
13:55:39 <@anarcat> and chains into hostmaster.make
13:55:46 <@ergonlogic> right
13:55:48 <@anarcat> which sources drupal-org.make
13:55:56 <@ergonlogic> oh, right
13:56:08 <@anarcat> that's pretty much it i guess
13:56:42 <@anarcat> our plan was to have drupal-org.make have the release tags and hostmaster.make have the git version, so that a git install that chains into hostmaster would install from git but a build from the
                    distribution would use tags
13:57:01 <@ergonlogic> which depends on drush 6
13:57:07 <@anarcat> yes!
13:57:09 <@anarcat> exactly

i now believe we'll have to depend on drush 6, see #2002804: [meta] depend on Drush 8.x-6.x? for that.

Comment #2

anarcat CreditAttribution: anarcat commented 31 May 2013 at 22:09

It seems we won't be able to depend on Drush 6 as this will mean depending on PHP 5.4 on Debian, which drops support for Drupal 6.

So we need to backport this: #1991764: Support using a distribution as core.

Comment #3

ergonlogic

he/him

English

Montréal, Québec 🇨🇦

CreditAttribution: ergonlogic commented 31 August 2013 at 18:01

Priority:

Normal

» Critical

I'm raising this to 'critical', as our release process is long, and error-prone. Also, our Jenkins server appears unstable, see: #2078531: ci.aegirproject.org loses jobs after every reboot and #2078533: ci.aegirproject.org cannot send notifications to #aegir

Comment #4

helmo CreditAttribution: helmo commented 24 September 2013 at 14:44

So do we have to wait for the drush 5.10 release? Or could be use drush 5.x already.

Comment #5

anarcat CreditAttribution: anarcat commented 24 September 2013 at 14:57

i think we need drush 5.10, and not only for this, but also for the flaky error reporting (#1931000: Missing drush backend output in frontend log).

Comment #6

anarcat CreditAttribution: anarcat commented 24 September 2013 at 15:26

i pinged the drush people again, we're just waiting on the maintainer to push the release. it will probably happen somewhere this week. :)

note that drush is on github now, i wonder if this changes our release process?

Comment #7

anarcat CreditAttribution: anarcat commented 10 October 2013 at 15:48

5.10 debian packages now in the unstable archive, so we can resume work on this.

Comment #8

cweagans

He/Him

Boise, ID, USA

CreditAttribution: cweagans commented 15 October 2013 at 17:14

What are the remaining tasks here? Anything I can help with?

Comment #9

anarcat CreditAttribution: anarcat commented 15 October 2013 at 20:24

the next step is to start hacking at the makefiles to generate a full tarball for the hostmaster distribution that will include core and all the necessary contrib modules then deploy that in hostmaster-install instead of the bits by bits setup. we also need to fix release.sh and related scripts and documentation. see the original post and irc chat above for more information.

Comment #9.0

anarcat CreditAttribution: anarcat commented 16 October 2013 at 15:30

Issue summary:

View changes

issue summary

Comment #9.1

anarcat CreditAttribution: anarcat commented 16 October 2013 at 15:34

Issue summary:

View changes

sample makefile

Comment #9.2

ergonlogic

he/him

English

Montréal, Québec 🇨🇦

CreditAttribution: ergonlogic commented 16 October 2013 at 16:09

Issue summary:

View changes

Updated issue summary.

Comment #10

anarcat CreditAttribution: anarcat commented 16 October 2013 at 17:42

Status:

Active

» Fixed

i actually committed all this into git. during my tests using drush make, things went pretty smoothly and in fact performance has increased by almost two orders of magnitude:

anarcat@desktop008:provision[6.x-2.x]$ time ~/src/drush/drush.php make aegir-release.make /tmp/aegir-release
Beginning to build aegir-release.make.                                                                                                                                                                                              [ok]
hostmaster-6.x-2.0-rc4 downloaded.                                                                                                                                                                                                  [ok]
hostmaster patched with common.inc_6.28.patch.                                                                                                                                                                                      [ok]
Generated PATCHES.txt file for hostmaster                                                                                                                                                                                           [ok]

real    0m1.651s
user    0m0.716s
sys     0m0.572s
anarcat@desktop008:provision[6.x-2.x]$ time ~/src/drush/drush.php make aegir-dev.make /tmp/aegir-dev
Beginning to build aegir-dev.make.                                                                                                                                                                                                  [ok]
drupal-6.28 downloaded.                                                                                                                                                                                                             [ok]
drupal patched with common.inc_6.28.patch.                                                                                                                                                                                          [ok]
Generated PATCHES.txt file for drupal                                                                                                                                                                                               [ok]
hostmaster cloned from http://git.drupal.org/project/hostmaster.git.                                                                                                                                                                [ok]
Checked out branch 6.x-2.x.                                                                                                                                                                                                         [ok]
Found makefile: hostmaster.make                                                                                                                                                                                                     [ok]
hosting cloned from http://git.drupal.org/project/hosting.git.                                                                                                                                                                      [ok]
Checked out branch 6.x-2.0-rc4.                                                                                                                                                                                                     [ok]
eldir cloned from http://git.drupal.org/project/eldir.git.                                                                                                                                                                          [ok]
Checked out branch 6.x-2.0-rc2.                                                                                                                                                                                                     [ok]
hosting_platform_pathauto-6.x-2.0-beta2 downloaded.                                                                                                                                                                                 [ok]
 >> Project views_bulk_operations contains 2 modules: actions_permissions, views_bulk_operations.
views_bulk_operations-6.x-1.16 downloaded.                                                                                                                                                                                          [ok]
 >> Project views contains 3 modules: views_export, views, views_ui.
views-6.x-3.0 downloaded.                                                                                                                                                                                                           [ok]
 >> Project modalframe contains 2 modules: modalframe_example, modalframe.
modalframe-6.x-1.7 downloaded.                                                                                                                                                                                                      [ok]
jquery_update-6.x-2.0-alpha1 downloaded.                                                                                                                                                                                            [ok]
jquery_ui-6.x-1.5 downloaded.                                                                                                                                                                                                       [ok]
install_profile_api-6.x-2.2 downloaded.                                                                                                                                                                                             [ok]
openidadmin-6.x-1.2 downloaded.                                                                                                                                                                                                     [ok]
admin_menu-6.x-1.8 downloaded.                                                                                                                                                                                                      [ok]
jquery_ui downloaded from http://jquery-ui.googlecode.com/files/jquery-ui-1.7.3.zip.                                                                                                                                                [ok]

real    1m3.873s
user    0m6.108s
sys     0m2.472s

Yes, you read that right: 1.651s vs 63.873s! This is probably worse because I am on a networked filesystem with an office connexion, but still...