[meta] Provide a beta to beta/rc upgrade path

Adding some markup to the summary for scannability, adding SAs to the must "should", and adding library upgrades to the "could".

@catch: I don't see at all how security issues would affect the D8 upgrade path? Most don't have API changes? Or do we consider throwing new exceptions early in the kernel API changes? Maybe I'm missing something, could you elaborate why?

@klausi: from #11:

A 'best effort' upgrade path also implies at least minimal effort to stop test sites getting pwned if they're publicly accessible.

In other words, once we say that beta X will have an upgrade path to future betas, then some early adopters/testers will start building real or semi-real sites with them. So would be nice to have those sites not be sitting targets of publicly disclosed vulnerabilities.

I think we should not support real sites running on D8 beta. That would mean we have to develop and release D8 patches when we fix security issues internally.

Sounds more like "D8 upgrade path" is the new "critical" category, after resolving all of those people can "use" D8 in production?

Klausi, I basically agree with your point of view, but I am currently in a situation where I need to evaluate D8 for future projects, whether or not to replace D7. An upgrade path between beta's would help a lot in this case, but I understand that not too much development time should go into this. I would much rather see it being spent on fixing the critical issues indeed.

We have a few projects coming up, which will be long-lasting projects, and I would very much like to know whether to use D7 (and risk an upgrade within a few years), or advice to go D8 within a few months. For this, I will need to evaluate quite a lot of features that we currently use in D7. We have a beta-2 install running now, and are running into a few issues. We can update to beta-3 and see if some of these things are fixed, but no working upgrade path would basically mean I will have to install it all over, right?

I've been trying to get a handle on which issues are blocking a beta-to-beta upgrade path, but the 'D8 Upgrade Path' tag seems to be a mishmash of all upgrade issues (with most of them being D7->D8). Since issues such as solidifying the views schema won't impact D7 upgrades, does it make sense to add a new tag to the ones that will impact beta-to-beta?

Added links from #22 to summary

I've been digging about for info on how the upgrade path stuff is going. Can someone point me to an update somewhere? Or, if one doesn't exist, I'd be happy to hear a braindump and write something up for others?

AFAIK https://www.drupal.org/project/head2head has a beta2beta sub module

Maybe this is offtopic, but since Migrate now is in core would it then be possible to include something like a .migrate as complement to the .install to handle update hooks?

I'm currently learning ruby and rails and like the way it has isolated database schema changes to be handled with their migrate feature. I find it very logical to separate storage and code changes the way it's done in Rails...

@catch. Sounds like an excellent plan to me, using 8.x as pivot to start using migrate instead of update hooks. That way the api can probably be made more clean than have to also keep supporting both D6 and D7 ways of doing things.

Because #1815826: Add "Plan" category to categorize what is called "meta issues" in core right now

Posted #2507899: [policy, no patch] Require hook_update_N() for Drupal 8 core patches beginning June 29, which as proposed would be the next step prior to a complete upgrade path in a later beta.

I guess this one shouldn't be postponed anymore.

Added #2527816: Logic error in SqlContentEntityStorage::countFieldData() attempts to drop `name` column which prevents any update hooks from running if the users_field_data table is detected as having schema changes.

At 11 (!) criticals, it's possible that the next beta is our last beta. Therefore, we should make this a "beta target" for one to try and get in before the next beta (tentatively July 22) so we could have at least one release with beta-to-beta upgrade paths.

If the next beta is our last, then the only "supported" upgrade path will be beta to rc, so updating title. Of course, it's possible the next one won't be the last one, so keeping the title flexible as to that.

I'm a little confused where this issue sits now that beta13 did actually ship with an upgrade path that even seemed to actually work in my limited manual testing.

We do still have 2 (possibly 3?) outstanding upgrade path blockers, though:

* #2537928: MySQL index normalization only works on table create - RTBC, lots of +1s from real-world users. Probably will be in shortly after post-beta commit freeze is over.
* #2540416: Move update.php back to a front controller
* #2538108: Add hook_post_update_NAME() for data value updates to reliably run after data format updates - Overlaps with the above. Only major but people in there asking why it's not critical.

So... do we close this, because we already did it? Or do we re-title it somehow and then leave it open until those other issues are fixed? Or?

#2542748: Automatic entity updates can fail when there is existing content, leaving the site's schema in an unpredictable state is also definitely a blocker for closing this, I'd say.

Been following the progress in this issue with great interest as it is a biggie. I have also started to play around and test a few ideas in D8 and today I had the opportunity to test how b12-b14 updating worked.

Site involved: Installed from D8b12 (some time ago) using Swedish as language setting.

Glad to report it seem to have worked just fine, no error message on screen och in dblog.

[Have reposted the below into its own issue: #2557097: The update result UI report needs clarifications

However, this report after the "update finished" is very confusing.

Particularly those three highlighted is not helpful at all, instead would probably make me quite nervous (for real site) even when the there are no failures listed below or nothing logged.

Is there no way for Drupal to verify that all known updates queued up have been performed and that the storage server (MySQL in my case) have not choked on any of them?

Right now the text tells me Drupal 8 can only guess and I can only hope that the lack of listed failures and no logged errors mean it actually worked as expected.

I would be happy to work with someone who knows the code involved here, to develop a better report than the one in the screenshot above. Feel free to ping me on IRC, I usually leave my computer logged in on #drupal-contrib and other channels.

I'm pretty sure that message is either identical or changed very little from D7, @tsvenson I would say create a new issue to look into that. Thanks!

I have created an issue for #42, #2558765: Improve readability UX for "Update report", including catch & display errors

@catch, @alexpott, @effulgentsia, @webchick, and I discussed the outstanding D8 upgrade path criticals today. The following two bugs with update functionality likely need to be resolved before we can consider this meta fixed:

• #2555665: When index is added for content_translation_uid, the corresponding stored schema definition is not updated
• #2558247: Remove rebuildAll() and module install from UpdatePathTestBase (may need retitle/rescope, but the exposed problems are definitely critical)

Then the following issues related to upgrade path test coverage remain critical to fix before a release candidate:

• #2555183: Fix the filled update tests, they are broken
• #2554151: Test content/configuration in update database dump

On this

So...

Used this dump: https://www.drupal.org/files/issues/beta12-filled.sql_.gz

Went to update.php:

Results looked good:

No errors when logged in:

Hadn't seen that post @plach -- this was with the same dump as in the tests though so would be good if you could do a double check with a fresh one.

Don't know how that tag got removed, must have happened in the cross post

I created a new installation on beta12, installed 3 languages, enabled translation for all entity types and config translation. Then I created some data for the various entity types and this was the status before the upgrade:

After checking out HEAD the theme was broken and I to clear caches to fix it:

Additionally the front page menu link was disabled:

When trying to run updates without clearing caches I needed to manually refresh the update page, otherwise it would be stuck after the first batch step. I was able to successfully complete the update this way, but I had to manually clear caches at the end of process for the site to be displayed correctly.

I restored the pre-update DB dump, cleared caches and updates run smoothly:

The end result looks correct, aside from the home page menu link that is still disabled after running the updates:

I'm attaching the beta12 DB dump in case people wants to try this by themselves.

@plach I imported your dump, cleared caches and went to update.php and didn't have those issues. So maybe some template issue?

I did have the disabled front page link.

As to the required cache clear after updating files to HEAD (without even having run update.php), I'm surprised that's all it took to have a semi-working site again :)

As I said, the cache clear fixes things. The problem arises when going straight to update.php without clearing caches.

Regarding the menu link disabling: #2548009: Overridden menu links lose parent, expanded and enabled (are disabled) status on cache clear seems to be exactly that issue.
This IS not a problem of the update path though, you should not clear caches before running update.php, well its pure randomness that you could still access the site, to be hoenst :)

The JS problem is now in a new issue: #2560521: Javascript does not work on beta 12 => HEAD update.php

you should not clear caches before running update.php, well its pure randomness that you could still access the site, to be hoenst :)

I know, it was just an attempt to proceed :)

Just tried out beta13 -> HEAD (with the JS fix applied) for a real site.

a) The update is quite slow, well we are adding more indexes and fields to the node tables for example. This is problematic / slow if you have actual content.
b) Everything works as expected

catch agreed that with the existing RTBCs #2560521: Javascript does not work on beta 12 => HEAD update.php #2548009: Overridden menu links lose parent, expanded and enabled (are disabled) status on cache clear and #1031122: postgres changeField() is unable to convert to bytea column type correctly we should good to go on this issue.

Before closing this, I'd wait for the manual testing to succeed: #2560521: Javascript does not work on beta 12 => HEAD update.php is fixing the assets, but I still see the menu item disappearing, even after #2548009: Overridden menu links lose parent, expanded and enabled (are disabled) status on cache clear has been committed.

1. I loaded the beta12 db dump I posted in #52
2. I checked out beta12
3. the menu page shows the link enabled
4. I checked out HEAD and applied #2560521-45: Javascript does not work on beta 12 => HEAD update.php
5. I run the update and I got lots of notices like the following on the selection page
   

     Notice: unserialize(): Error at offset 0 of 48 bytes in Drupal\Core\Menu\MenuTreeStorage->prepareLink() (line 622 of core/lib/Drupal/Core/Menu/MenuTreeStorage.php).
   

6. When the update is over the menu link is disabled

After quite some debugging I realized the beta12 DB dump was affected by #2548009: Overridden menu links lose parent, expanded and enabled (are disabled) status on cache clear and that we had a buggy static menu link override disabling the front page link ready to be applied at the first menu rebuild. In fact just clearing caches on beta12 was enough to disable the link.

This was the content of the core.menu.static_menu_link_overrides config entry before running the update process:

a:1:{s:11:"definitions";a:2:{s:18:"contact__site_page";a:5:{s:7:"enabled";b:1;s:9:"menu_name";s:6:"footer";s:6:"parent";s:0:"";s:6:"weight";i:0;s:8:"expanded";b:0;}s:20:"standard__front_page";a:5:{s:6:"weight";i:-50;s:9:"menu_name";s:4:"main";s:6:"parent";s:0:"";s:8:"expanded";b:0;s:7:"enabled";b:0;}}}

The update process just triggers a menu link rebuild, which correctly disables the menu item. The error is in the config and is introduced while working with the beta12 codebase. Hence, we should be good here :)

I tried running a smoke test of update.php, not as complete as #52.

Here's what I tried, all on MAMP (so, MySql):

First, install beta-12 via web UI, Standard Profile. Create an article node. Git checkout 8.0.x. Run /update.php via browser. This worked great for me. The node tables got the expected indexes and NOT NULL constraints. The "Home" link in Main navigation remained Enabled.

Then, I started over from scratch, installed beta-12 via web UI, Standard Profile. Create an article node, with a menu link, giving it a weight of -10. Went to edit the Main navigation menu, and reordered Home to be first (so it ended up with weight=-50 and the node/1 link with weight=-49). Git checkout 8.0.x. Run /update.php via browser. This ended up fataling with a WSOD and the following in my php_error.log:

PHP Fatal error:  Class 'Doctrine\Common\Annotations\SimpleAnnotationReader' not found in /d8/core/lib/Drupal/Component/Annotation/Plugin/Discovery/AnnotatedClassDiscovery.php on line 72

At @catch's suggestion, I tried the 2nd scenario from #62 again, with an apache restart prior to installing beta-12, and got a different fatal error at the end:

PHP Fatal error:  Maximum execution time of 240 seconds exceeded in /d8/core/lib/Drupal/Core/Cache/ApcuBackend.php on line 188

At @catch's suggestion, I tried the 2nd scenario from #62 again, with an apache restart prior to installing beta-12, and got a different fatal error at the end:

What means end, is this the step where we call drupal_flush_caches?

The hang in #63 was on the screen with a progress bar saying "Completed 12 of 12" and trying to redirect to the "Finished" page. During that, BareHtmlPageRenderer->renderBarePage() is called which does library discovery, which calls color_library_info_alter(), which does a config get, which because it's using the chained backend, tries to do a ApcuBackend->set().

However, I restarted my computer and tried the whole scenario again, and this time it worked. Everything completed just fine. But I was left with the "Home" link disabled, which maybe is as expected per #61.

Also, I confirmed #61. Just doing this:

- Install beta-12 via web UI, Standard Profile.
- Create an article node, with a menu link, giving it a weight of -10.
- Edit the Main navigation menu, and reorder Home to be first (so it ends up with weight=-50 and the node/1 link with weight=-49).

Results in the core.menu.static_menu_link_overrides config record having enabled: false for the Home link. The link only appears to still be on the site due to the lack of a menu rebuild. So there's nothing wrong with the update.php process in this regard, only a bug with beta-12 itself, fixed recently by #2548009: Overridden menu links lose parent, expanded and enabled (are disabled) status on cache clear.

So, what's left to do/test before closing this issue?

I also ran the same scenario as #62.2, but starting from beta-13 and again starting from beta-14. Both worked smoothly. Between that, and #59 - #61, I'm marking this RTBC, and assigning to @catch to decide on when it can be closed (e.g., whether to block on #1031122: postgres changeField() is unable to convert to bytea column type correctly and/or other issues).

Since the problems uncovered in manual testing are now captured in other issues, marking this one as duplicate. Yay!