Problem/Motivation

Depends on the API being added in #3066801: Add hook_removed_post_updates(), should not be any patch conflicts with it.

How to check for updates added since 8.8.0-rc1:

git log --since 22/11/2019 8.9.x core/modules/*.install
git log --since 22/11/2019 8.9.x core/modules/*.post_update.php
git log --since 22/11/2019 9.0.x core/modules/*.install
git log --since 22/11/2019 9.0.x core/modules/*.post_update.php

Don't remove these:


action_post_update_move_plugins

action_post_update_remove_settings

hal_post_update_delete_settings

layout_builder_post_update_override_entity_form_controller

rest_post_update_delete_settings

serialization_post_update_delete_settings

system_post_update_entity_revision_metadata_bc_cleanup

system_post_update_extra_fields_form_display

system_post_update_uninstall_classy

system_post_update_uninstall_entity_reference_module

system_post_update_uninstall_simpletest

text_post_update_add_required_summary_flag_form_display

workspaces_post_update_remove_association_schema_data

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

As a reminder, Drupal 8 sites on 8.7 or earlier must update to either 8.8 or 8.9 before updating to Drupal 9 with update.php. All Drupal 8 post-update functions have been removed in 9.0.0-beta1. (Regular update functions were already removed in 9.0.0-alpha1.)

CommentFileSizeAuthor
#48 3106666-48.patch134.91 KBjungle
#48 interdiff-45-48.txt1.54 KBjungle
#45 interdiff-38-45.txt1.38 KBjungle
#45 3106666-45.patch135.53 KBjungle
#42 interdiff-38-42.txt1.81 KBjungle
#42 3106666-42.patch135.96 KBjungle
#38 3119027-combined.patch138.34 KBcatch
#38 3106666-37.patch135.37 KBcatch
#38 3106666-interdiff-34-37.txt1.24 KBcatch
#34 3106666-interdiff.txt3.43 KBcatch
#34 3106666-32.patch136.61 KBcatch
#32 3066801-116.patch19.46 KBcatch
#32 3066801-interdiff.txt804 bytescatch
#29 3106666-30.patch133.17 KBcatch
#29 3106666-interdiff.txt1.24 KBcatch
#29 3106666-combined.patch151.17 KBcatch
#27 3106666-25.patch131.94 KBcatch
#27 3106666-combined.patch148.22 KBcatch
#25 3106666-25.patch131.94 KBcatch
#25 3106666-combined.patch148.22 KBcatch
#22 3106666-interdiff.txt4.27 KBcatch
#22 3106666-22.patch133.01 KBcatch
#22 3106666-combined.patch149.29 KBcatch
#21 3106666-combined.patch151.71 KBcatch
#19 3106666-13.patch135.43 KBcatch
#19 3106666-combined.patch151.7 KBcatch
#18 3106666-13.patch135.43 KBcatch
#17 3106666-3066801.patch16.28 KBcatch
#14 3106666-13.patch135.43 KBcatch
#13 3106666-interdiff.txt2.46 KBcatch
#13 3106666-13.patch134.26 KBcatch
#12 3106666-12.patch134.26 KBcatch
#12 3106666-interdiff.txt2.37 KBcatch
#10 3106666-10.patch134.27 KBcatch
#10 3106666-interdiff.txt7.08 KBcatch
#9 3106666.patch127.68 KBcatch
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

catch created an issue. See original summary.

catch’s picture

mpdonadio’s picture

Potentially handling an individual case in #3110393: Remove @deprecated views code datetime_range.module, where some deprecated code is coupled to a post_update.

catch’s picture

Updates to not remove because they are 9.0.x-only:

hal_post_update_delete_settings

serialization_post_update_delete_settings

catch’s picture

Title: Remove post updates up to 88** » [pp-1] Remove post updates added prior to 8.8.0
Status: Active » Postponed
Gábor Hojtsy’s picture

Tagging as beta requirement per @xjm in #3108254-17: [META] Drupal 8-9 upgrade path clean-up.

catch’s picture

Now postponed on #3066801: Add hook_removed_post_updates() (still one issue).

catch’s picture

Issue summary: View changes
catch’s picture

Still requires the other patch before this can actually be committed, but this should be it I think.

catch’s picture

Was over-zealous removing use statements from system.post_update.php, hence the test failures.

Also forgot to do workspaces post updates in the last patch, not happening in #3108416: Remove workspace_update_8803() but the decision there means we can remove all the post updates from workspaces.

catch’s picture

catch’s picture

Wrong hook name in contextual module.

catch’s picture

And system..

catch’s picture

Updating for the new format on #3066801: Add hook_removed_post_updates(), no interdiff because it affects nearly every added line in the patch.

dww’s picture

+++ b/core/modules/action/action.post_update.php
@@ -5,22 +5,13 @@
+    'action_post_update_move_plugins' => '9.0.0',

Not quite as elegant, but technically shouldn't these all be 9.0.0-beta1 instead? Or are we going to assume that if someone gets into the error state and is told "you must upgrade to the last release before 9.0.0", that everyone will assume "last official release, not including alphas/betas/rcs, etc"?

If we want to use 9.0.0 for these, maybe we want a clearer message about it in #3066801: Add hook_removed_post_updates() to avoid any possible confusion / headaches?

catch’s picture

For Drupal 9, no one is ever going to see this message because they will hit the schema message from #3098475: Add more strict checking of hook_update_last_removed() and better explanation first.

But yeah we should never be telling people to update to a release earlier than a beta, that could be interpreted as telling them to update to an alpha.

catch’s picture

Here's a combined patch just to make sure none of our upgrade path testing will run into the hook_requirements().

catch’s picture

Status: Postponed » Needs review
FileSize
135.43 KB

Re-uploading #13 now that this passes.

We can't actually commit this until #3106666: Remove post updates added prior to 8.8.0 is in but it's reviewable now so changing status.

catch’s picture

Ugh #17 wasn't a real combined patch :(

The last submitted patch, 19: 3106666-combined.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

catch’s picture

Proper combined patch this time, but looks like this is going to fail due to post updates not being properly recorded in the filled database (or a bug in the hook_removed_post_updates() implementations).

catch’s picture

OK fixing some real bugs in the patch here now.

Text and system post update removals were too aggressive, took out some from #3092714: Config entity updater misbehaves when updating multiple entity types which needed to stay in - added these back.

migrate_drupal added a post update that was namespaced incorrectly so could never run. It was only to clear a cache so we can just remove it entirely as if it never existed.

Image post update - missing an array value in the hook implementation.

Locale - cruft in the hook implementation.

At least this shows we have extensive test coverage for the hook implementations via existing upgrade path tests.

The last submitted patch, 21: 3106666-combined.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

The last submitted patch, 22: 3106666-combined.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

catch’s picture

Actions fail is due to #3053656: Rename action.post_udate.php to action.post_update.php so that the upgrade path runs correctly - we added the post update over a year ago, but it didn't actually run properly until post-8.8.0 so it needs to stay in.

New patch just reverts all changes to action.post_update.php

Nice how the system_requirements() keeps us honest.

The last submitted patch, 25: 3106666-combined.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

catch’s picture

The combined patch should be green now. No changes to the actual patch here but re-uploading for clarity.

The last submitted patch, 27: 3106666-combined.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

catch’s picture

One more test failure to resolve, hidden by the last one.

alexpott’s picture

Title: [pp-1] Remove post updates added prior to 8.8.0 » Remove post updates added prior to 8.8.0

#3066801: Add hook_removed_post_updates() is in 9.0.x so this can roll on.

alexpott’s picture

Status: Needs review » Needs work

field.post_update.php needs updating. I think all the post updates in that file can be remvoed.

I think we could consider removing layout_builder_post_update_override_entity_form_controller() because it's pointless on a major version upgrade - even though it has been added since.

catch’s picture

Status: Needs work » Needs review
FileSize
804 bytes
19.46 KB

#31.1 - done. I think I just missed the file.

#31.2 - don't think we can do that. If we remove it and add it to a hook implementation, sites will hit the error. If we remove it and don't add it to the hook implementation, we don't protect against adding a post update with the same name in a later release (which would then not run on sites updated from 8.8.0 because they've already run something that looks like it).

alexpott’s picture

RE #31.2 Good point - we could file an issue to remove it altogether - it's not been in a release. But plenty of people have applied the patch that adds it in prod - so let's leave it well alone.

catch’s picture

#32 was completely the wrong patch and interdiff...

tim.plunkett’s picture

Status: Needs review » Needs work

Excluding the ones explicitly mentioned in the IS, here are the remaining post_updates left after the patch:

// non-test
action_post_update_move_plugins
system_post_update_entity_revision_metadata_bc_cleanup
system_post_update_extra_fields_form_display
text_post_update_add_required_summary_flag_form_display

// test-only
module_test_post_update_test
update_test_failing_post_update_first
update_test_postupdate_post_update_first
update_test_postupdate_post_update_second
update_test_postupdate_post_update_test_batch
update_test_postupdate_post_update_test0
update_test_postupdate_post_update_test1
workspace_update_test_post_update_check_active_workspace

I think some of them should be added to the IS, but at least the action one looks removable.

Berdir’s picture

I don't want to hold this issue up on that, but #3119027: Use filled dump for RestSettingsDeletionUpdateTest would be a cleaner fix for RestSettingsDeletionUpdateTest, we wouldn't need any changes in that test anymore in this issue, I can also reroll that once this is in.

catch’s picture

Issue summary: View changes
Status: Needs work » Needs review

#35:

actions_post_update_move_plugins() is not removable. We added it a long time ago, but it never ran until #3053656: Rename action.post_udate.php to action.post_update.php so that the upgrade path runs correctly which landed post 8.8.0. It does look removable and this caught me out too until #3106666-25: Remove post updates added prior to 8.8.0, added to issue summary.

system_post_update_extra_fields_form_display
text_post_update_add_required_summary_flag_form_display

These are both due to #3092714: Config entity updater misbehaves when updating multiple entity types which also landed post-8.8.0. Also added to issue summary.

system_post_update_entity_revision_metadata_bc_cleanup

this was added to core after the issue summary was last updated, added too.

All the test post updates need to stay in of course.

catch’s picture

#3119027: Use filled dump for RestSettingsDeletionUpdateTest now has a patch courtesy of berdir.

So... here's a patch without the changes to RestSettingsDeletionUpdateTest

Also uploading a combined patch with that issue to ensure no surprises. That is actually less changes to the test than were done in this issue, so it might also be OK to merge the two.

The last submitted patch, 38: 3106666-37.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

catch’s picture

#3119027: Use filled dump for RestSettingsDeletionUpdateTest is in.

Kicked off a retest of #38 which should now pass.

The last submitted patch, 38: 3106666-37.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

jungle’s picture

FileSize
135.96 KB
1.81 KB

Fix coding standards according to https://www.drupal.org/pift-ci-job/1608391

xjm’s picture

Thanks @jungle for helping with so many of the beta blockers. We have a set of coding standards rules that have to pass for patches to be committed, defined in phpcs.xml.dist. If those rule don't pass on a changed file, the patch can't be committed. However, it's a smaller ruleset than the full Drupal standard, because it only includes rules that core complies with globally.

However, coding standards cleanups to unchanged lines not in the passing ruleset are out of scope unless the issue is specifically about coding standards fixes. (See https://www.drupal.org/core/scope#creep for more information.) I'm able to commit this The reason is that it adds more lines for the reviewer to read in an already large patch. It also can cause conflicts with other patches.

And... apparently the test results are using a different ruleset than that, because I'm able to commit #38 locally with no error, which means the coding standards fixes are out of scope. Sorry about that ! Whitespace fixes are usually innocuous enough, but this change should be reverted from the patch (even though it's fixing a valid coding standards bug):

+++ b/core/modules/system/system.post_update.php
@@ -96,7 +97,7 @@ function system_post_update_entity_revision_metadata_bc_cleanup() {
-  foreach ($last_installed_definitions as $entity_type_id => $entity_type) {
+  foreach ($last_installed_definitions as $entity_type) {

What you could do is file a followup issue to check all of core for failures of this particular rule, which we'd then follow with an issue to add that to the enabled core ruleset. (This can be a big project!) Reference here: https://www.drupal.org/core/scope#coding-standards

Thanks!

jungle’s picture

Assigned: Unassigned » jungle

Thanks, @xjm for your detailed explanation. Understood and learnt it again :), I am reverting it next. @alexpott did share that link to me once on another issue I worked on.

jungle’s picture

Assigned: jungle » Unassigned
FileSize
135.53 KB
1.38 KB

Started from #38 again, skip #42 please.

alexpott’s picture

@jungle to run the current core coding standards the easiest way is do $ composer run phpcs -- ./core -ps from the command line. Note you can change the ./core to target only the paths you've changed (so it's a bit quicker).

alexpott’s picture

I think that was should leave workspaces_post_update_remove_association_schema_data() in? I was added in #3098427: Manipulating the revision metadata keys before running the BC layer breaks the BC layer so this update was added after 8.8.0. It was first released in 8.8.3 and the hook_update_removed is 8803 - which was present in 8.8.2. So I *think* (but not 100% sure) that workspaces_post_update_remove_association_schema_data() has to stay. As we allow updating from 8.8.2 workspaces to 9.0.0.

jungle’s picture

FileSize
1.54 KB
134.91 KB

Thanks, @alexpott for your tips. Bring back workspaces_post_update_remove_association_schema_data ()

catch’s picture

Oof yes workspaces_post_update_remove_association_schema_data() does need to stay in.

It's linked to an update that we've removed in 9.0.x, but it's an addition from the latest fixes to that update that can run on already-upgraded sites to clean up the cruft (and will run fine by itself without the context of the update).

Test coverage caught all the other wrongly-removed post updates, but it doesn't catch this one because we'd need an explicit dump of 8.8.3 (i.e. where the workspaces schema version is high enough to not run into hook_update_last_removed(), but the post update has not yet run).

We ran into similar confusion on the original issue, not that I remembered it working on this, see #3098427-77: Manipulating the revision metadata keys before running the BC layer breaks the BC layer for the sequence of events.

xjm’s picture

Status: Needs review » Needs work

NW for #49.

catch’s picture

Status: Needs work » Needs review

#48 resolves #49, I was just +1ing.

We never add upgrade path test coverage for patch releases, and we can't really do that sustainably until we refactor our upgrade path tests to install core + create content in one version, then upgrade to another (theoretically possible with build tests).

tim.plunkett’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

Adding workspaces_post_update_remove_association_schema_data to the IS along with the others.

The only oddity remaining is that drupal_migrate_post_update_clear_migrate_field_plugin_cache is removed outright, with no corresponding entry in a hook_removed_post_updates implementation.
I see it's because it was supposed to be migrate_drupal, not drupal_migrate.

In that case, I can confirm that 100% of the removed updates are represented in a removed hook, and that no other additions are made.
Thanks!

xjm’s picture

Assigned: Unassigned » xjm

Reviewing the thing.

xjm’s picture

Alright! I read the whole patch, including looking at the removed implementations, and also read side-by-side to compare the removed code with the added hook implementations. I also verified that the removed helper functions in LB were indeed not used outside the post-update.

One thing did not make sense on a unprepped read of the patch:

+++ /dev/null
@@ -1,15 +0,0 @@
-function drupal_migrate_post_update_clear_migrate_field_plugin_cache() {
-  // Empty post-update hook.

There's no hook_removed_post_updates() for this. Going to look in the issue for why...

xjm’s picture

Okay, that's explained in #22:

migrate_drupal added a post update that was namespaced incorrectly so could never run. It was only to clear a cache so we can just remove it entirely as if it never existed.

Kind of like a test to see if the reviewer is paying attention. ;)

Going to do one more read-through of added lines only, and then apply it locally and start checking post-updates.

xjm’s picture

So the only thing locally that I can't account for from the IS is:
workspace_update_test_post_update_check_active_workspace

It's mentioned in #35 but there's no response from @catch about that particular one, nor is it mentioned in #49 as being related to the infamous Workspaces 8.8.3 fix. It's a test implementation, for a test that is testing....🥁.... (This is me reading the code and blaming....) #3102059: Make tests of the update system use UpdatePathTestTrait instead of UpdatePathTestBase ? 🤔

Reading the diff from that commit, the same test used to be in....
core/modules/workspaces/tests/src/Functional/Update/WorkspacesUpdateTest.php

And the bit there dates to #3093879: Ensure that there's no active workspace while running database updates. Which doesn't seem to be the fated post-8.8.0 issue; it was committed in November before 8.8.0.

xjm’s picture

Assigned: xjm » Unassigned
xjm’s picture

@Berdir confirmed that the test that uses workspace_update_test_post_update_check_active_workspace() is meant as a permanent regression test (versus being related to any particular update that needs to be tested).

Onwards to credits...

  • xjm committed 4c45368 on 9.0.x
    Issue #3106666 by catch, jungle, xjm, alexpott, tim.plunkett, Berdir:...
xjm’s picture

Issue summary: View changes
Issue tags: +9.0.0 release notes

Committed and pushed to 9.0.x. Yay!

We should mention this in the beta1 release notes (since it's a change from the alphas) and then it's just rolled in with the general note about update hooks in the 9.0.0 notes. I attached the change record from the hook_update_N() removals to this issue as well.

Added a release note to the IS.

xjm’s picture

Status: Reviewed & tested by the community » Fixed

I've been told this is the issue status you Drupalists use for committed patches. ;)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.