drush updb errors on System and Layout Builder modules when going 10.3.1 to 11.0.0.-rc1

Changes are made on on 11.x (our main development branch) first, and are then back ported as needed according to our policies.

I think this was caused by a merge conflict resolution failure in #3437162: Move twig_debug and other development toggles into raw key/value - it looks like I removed the post update from 10.3, then a couple of weeks later we also removed it in #3439769: Remove all the pre-10.3.0 updates hooks from 11.x, which means we're looking for a post update that could never possibly have run.

sdc is still in 11.x as obsolete, so the answer is to add back the update in 11.x, and probably to 10.4 too, but let's start with 11.x

Oh and also probably the reason our upgrade path testing didn't catch this is because the database dumps were also created around the same period, so they will probably have the post update applied, before it was accidentally removed. Another reason to do #3403649: Rework database update tests so we don't have to ship database dumps in git.

Thanks catch.

I can confirm that the MR fixes this update problem. I followed the steps in the issue summary and for completeness also tested on 11.x.

The steps to reproduce are pretty clear. Just for kicks, I decided to install the demo_umami profile. When I tried the update, I saw two error messages:

 [error]  The installed version of the /Layout Builder/ module is too old to update. 
Update to a version prior to 11.0.0 first (missing updates: 
layout_builder_post_update_default_expose_field_block_setting).
 
 [error]  The installed version of the /System/ module is too old to update. Update to 
a version prior to 11.0.0 first (missing updates: 
system_post_update_sdc_uninstall).

I tried again: this time, I checked out the feature branch instead of 11.0.0-rc1. One error is gone, but one remains:

 [error]  The installed version of the /Layout Builder/ module is too old to update.  
Update to a version prior to 11.0.0 first (missing updates:  
layout_builder_post_update_default_expose_field_block_setting).

I think we should expand the scope of this issue to handle both.

The layout builder issue is similar but also different.

In #3043330: Reduce the number of field blocks created for entities (possibly to zero) we added a config-based feature flag for field blocks - this was committed to 10.3.x during development.

In #3432874: Replace "Expose all fields as blocks to Layout Builder" configuration with a feature flag module we switched from the config-based feature flag to a feature flag module, this commit removed the existing upgrade path for the config change (because it never made it into a release).

In #3439769: Remove all the pre-10.3.0 updates hooks we remove the pre-10.3.0 upgrade hooks, this was only four days after the switch from config to module, so we managed to still 'remove' a post update that never existed.

The new 10.x database dumps were committed in the same week via #3414563: Add new 10.3.x database dump fixtures, without modules deprecated for removal in 11.x - these must include the post update having been run, (i.e. created during the short window then the update existed) which is why the upgrade path tests didn't pick up the problem.

The answer for layout builder is just to remove that post update from the list of removed post updates because as far as stable releases are concerned, it never existed.

With the updated MR, both errors are gone when I run database updates. That is half of RTBC.

I want to confirm the history of these database updates before I am willing to call this issue RTBC. But maybe someone else has already looked into it and can do that before I finish.

For ease of reference (and editing the output of Git to produce links):

$ git hist -S layout_builder_post_update_default_expose_field_block_setting
...
* 42c76c8f545 2024-08-02 | Remove removed post update that never made it into a release. [catch]
...
* 360a9c58550 2024-04-19 | Issue #3439769: Remove all the pre-10.3.0 updates hooks by quietone, catch: Remove all the pre-10.3.0 updates hooks [catch]
...
* 41e2fd078ea 2024-04-15 | Issue #3432874: Replace "Expose all fields as blocks to Layout Builder" configuration with a feature flag module by acbramley, catch, alexpott: Replace "Expose all fields as blocks to Layout Builder" configuration with a feature flag module [catch]
...
* 1561edc2b82 2024-03-07 | Issue #3043330: Reduce the number of field blocks created for entities (possibly to zero) by acbramley, nterbogt, phenaproxima, Sam152, tim.plunkett, quietone, smustgrave, jibran, larowlan, azinck, catch, alexpott, tedbow, kim.pepper: Reduce the number of field blocks created for entities (possibly to zero) [Alex Pott]

and

$ git hist -S system_post_update_sdc_uninstall
...
* b8037acd209 2024-04-03 | Issue #3409456: Remove SDC deprecated code before 11.0.0 by plopesc, smustgrave, quietone, e0ipso, catch: Remove SDC deprecated code before 11.0.0 [Alex Pott]

The MR for this issue adds one line containing system_post_update_sdc_uninstall and removes another, so it is missed by git -S system_post_update_sdc_uninstall.

I agree with @benjifisher that the latest MR fixes the problem with layout_builder and Umami. Hopefully, @benjifisher will RTBC so that I can commit this.

I do not know why the priority and status changed.

OK, I confirmed that the system update function was added in #3409456:

$ git hist -S 'system_post_update_sdc_uninstall()'
...
* a43e166e53d 2024-08-02 | Remove the sdc uninstall post update. [catch]
...
* 360a9c58550 2024-04-19 | Issue #3439769 by quietone, catch: Remove all the pre-10.3.0 updates hooks [catch]
...
* b8037acd209 2024-04-03 | Issue #3409456 by plopesc, smustgrave, quietone, e0ipso, catch: Remove SDC deprecated code before 11.0.0 [Alex Pott]

I get the same result with 11.0.x.

It was added and removed in the same issue in the 10.4.x branch:

$ git hist -S 'system_post_update_sdc_uninstall()' 10.4.x
...
* 9283d590cf5 2024-04-08 | Issue #3437162 by plopesc, smustgrave, Berdir: Move twig_debug and other development toggles into raw key/value [catch]
* d31ade0f626 2024-04-08 | Issue #3437162 by plopesc, smustgrave, Berdir: Move twig_debug and other development toggles into raw key/value [catch]

and I get the same results with 10.3.x.

This update function is not in the 10.2.x branch nor in the 10.3.0, 10.3.1 tags, so I agree that it has never been in a released version.

I guess it was just an accident that it was removed in #3439769. Is that the failed conflict resolution mentioned in #5?

I am satisfied with that. I am going to look at the layout_builder one now.

I am also adding some related issues.

I guess it was just an accident that it was removed in #3439769. Is that the failed conflict resolution mentioned in #5?

Yes looking at the commit history, I originally had correctly added the update after resolving conflicts, but then thought better of it and removed it - but because it was at 10.4.x issue it should have stayed in there (or alternatively been 11.0-only, but then not removed from 11.0). Since we need the update to run on sites that have sdc installed, and 10.3 is already out, bringing it back is easiest.

On the 11.x and 11.0.x branches, it looks as though layout_builder_post_update_default_expose_field_block_setting() was added in #3043330: Reduce the number of field blocks created for entities (possibly to zero) and removed in #3432874: Replace "Expose all fields as blocks to Layout Builder" configuration with a feature flag module–(replaced by layout_builder_post_update_enable_expose_field_block_feature_flag()):

    $ git hist -S 'layout_builder_post_update_default_expose_field_block_setting()' 
    ...
    * 41e2fd078ea 2024-04-15 | Issue #3432874 by acbramley, catch, alexpott: Replace "Expose all fields as blocks to Layout Builder" configuration with a feature flag module [catch]
    ...
    * 1561edc2b82 2024-03-07 | Issue #3043330 by acbramley, nterbogt, phenaproxima, Sam152, tim.plunkett, quietone, smustgrave, jibran, larowlan, azinck, catch, alexpott, tedbow, kim.pepper: Reduce the number of field blocks created for entities (possibly to zero) [Alex Pott]

I get the same thing, but with different commit hashes, on the 10.3.x and 10.4.x branches.

That update function was never in the 10.2.x branch.

I also checked the 10.3.0 and 10.3.1 tags. Both have layout_builder_post_update_enable_expose_field_block_feature_flag() and not layout_builder_post_update_default_expose_field_block_setting(), so there is no point listing the latter in layout_builder_removed_post_updates().

I am just a little confused about the title of #3439769: "Remove all the pre-10.3.0 updates hooks". We are also removing the update hooks that are only in 10.3.0 and 10.3.1. I checked the release notes for 11.0.0-rc1, and I see

Sites should update to the latest release of 10.3 prior to updating to Drupal 11

So sites are supposed to upgrade to 10.3, run these update functions, and then upgrade to 11. That is what I tested. So it is right to remove these functions, and it is only the issue title that is confusing me.

I notice that the component is the database update system, but the fix is in the `system` and `layout_builder` modules. I guess we can live with that.

I am just a little confused about the title of #3439769: "Remove all the pre-10.3.0 updates hooks". We are also removing the update hooks that are only in 10.3.0 and 10.3.1. I checked the release notes for 11.0.0-rc1, and I see

Ah yeah it's 'all the update hooks added up until the point that 10.3.0 was tagged but not any after 10.3.0 was tagged'. The mis-matches here were because we wanted to get 11.0.0 beta released before 10.3.0 was released, we actually had to remove the 11.x update hooks, and update the database dump fixtures, before 10.3.0 was tagged. It's a bit too much time travel.

@tobybellwood, thank you for reporting this!

@benjifisher, thanks for the thorough checking. And yes, the issues regarding the update path should be in the 'database update system' component.

Committed 6b50d02 and pushed to 11.x.

Committed a6e453e and pushed to 11.0.x.

Thanks!

Since #3439769: Remove all the pre-10.3.0 updates hooks had two problems that we know about (both fixed in this issue), I decided to confirm that there are not any more.

I found all the modules affected by #3439769: git show --name-only 360a9c58550, then extract the module names.

Then I repeated the testing steps, installing those modules:

git checkout 10.3.1
composer require drush/drush
drush si
drush en big_pipe block_content ckeditor5 comment contact datetime_range dblog editor file filter forum help history layout_builder locale media migrate mysql node path_alias pgsql responsive_image statistics system taxonomy text tracker update user views
git restore composer*
git checkout 11.0.0
composer require drush/drush drupal/forum drupal/statistics
git clone -C modules git@git.drupal.org:project/tracker.git
drush updatedb

I had to install three modules that had been removed from core. I am not sure why I had trouble with tracker: maybe I should have cleared the composer cache.

The good news is that I did not get any message about missing update functions.

@benjifisher thanks for the extra testing!