Followup #2999721: [META] Deprecate the legacy include files before Drupal 9

Problem/Motivation

Avoid handling of update.in file include and leave that task for autoloader.

This ticket can cover
update_already_performed
update_retrieve_dependencies
update_replace_permissions

Proposed resolution

Convert functions of update.inc to a class.

Remaining tasks

  • Convert functions.
  • Replace calls.
  • Properly deprecate functions.
  • Add legacy tests.

User interface changes

none

API changes

Deprecation of the update_* functions from update.inc file.
New Update class.

Data model changes

none

Issue fork drupal-3012523

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

voleger created an issue. See original summary.

voleger’s picture

Issue summary: View changes
StatusFileSize
new65.74 KB

First iteration

voleger’s picture

Assigned: voleger » Unassigned
Status: Active » Needs work
goodboy’s picture

StatusFileSize
new66.11 KB

Change call update_do_one to Update::doOne in triggerBatch()

voleger’s picture

+++ b/core/modules/system/src/Controller/DbUpdateController.php
@@ -602,7 +602,7 @@ protected function triggerBatch(Request $request) {
+        $operations[] = [['Update', 'doOne'], [$update['module'], $update['number'], $dependency_map[$function]]];

replace 'Update' with Update::class to pass the full class name to the callable array.

goodboy’s picture

StatusFileSize
new66.15 KB

Change to Update::class

voleger’s picture

StatusFileSize
new668 bytes
new679 bytes
new4.7 KB
new69.58 KB

Attached interdiff for #4 and #6.
Replaced rest off the calls and updated docs.
Let's see if we ready for the further tasks.

goodboy’s picture

StatusFileSize
new10.63 KB
new77.47 KB

Added deprecations

voleger’s picture

StatusFileSize
new78.87 KB
new1.4 KB

Add initial legacy test

goodboy’s picture

StatusFileSize
new1.46 KB
new78.81 KB

Fix deprecations errors

goodboy’s picture

StatusFileSize
new80.28 KB

Add legacy tests for update_fix_compatibility() and update_check_incompatibility()

voleger’s picture

StatusFileSize
new2.04 KB

Interdiff

voleger’s picture

StatusFileSize
new80.42 KB
new630 bytes

This functionality depends on the system module. Let's enable the module in the tests.

goodboy’s picture

StatusFileSize
new2.34 KB
new81.83 KB

Add tests for 3 functions. Intediff has #13 and #14.

goodboy’s picture

StatusFileSize
new83.92 KB
new3.87 KB

Add new test methods and fix exists methods

goodboy’s picture

StatusFileSize
new85.43 KB
new3.08 KB

Add new test methods and fix exists methods

goodboy’s picture

StatusFileSize
new3.35 KB
new86.66 KB

Add new test methods and fix exists methods

voleger’s picture

StatusFileSize
new87.24 KB
new3.83 KB

Fixed tests.
Also figured out that update_replace_permissions() used immutable configs to update them later. So I had fixed this issue too. Looks like there are no usages of that function and that function is not covered with tests.

voleger’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new87.38 KB
new3.38 KB

All functions covered in legacy tests. One more thing that I changed in the tests is the update number to lower and not existent to avoid the potential match with future update number.

voleger’s picture

Issue summary: View changes
voleger’s picture

Issue tags: +Kill includes, +@deprecated
andypost’s picture

Status: Needs review » Needs work

Initial review, thanks for great work!

- lots of \Drupal:: calls inside loops - needs refactor
- probably this class needs split into static methods (batch related) and service to properly inject dependencies, guess multiple classes
- usage needs better analysis, if there will be a service it needs injection
- naming - requirements could use "get" or "build"... like #2620304: htaccess functions should be a service

will check deeper later

+++ b/core/lib/Drupal/Core/Update/Update.php
@@ -0,0 +1,714 @@
+    $extension_config = \Drupal::configFactory()->getEditable('core.extension');
...
+      $themes = \Drupal::service('theme_handler')->rebuildThemeData();
+      $modules = system_rebuild_module_data();
...
+    $system_schema = drupal_get_installed_schema_version('system');
...
+    $requirements = \Drupal::moduleHandler()->invokeAll('requirements', ['update']);
...
+    \Drupal::keyValue('system.schema')->set($module, $schema_version);
...
+    require_once \Drupal::root() . '/core/includes/module.inc';
+    system_list_reset();
...
+    module_load_include('php', $module, $module . '.post_update');
...
+          \Drupal::service('update.post_update_registry')->registerInvokedUpdates([$function]);
...
+    $modules = drupal_get_installed_schema_version(NULL, FALSE, TRUE);
...
+      $updates = drupal_get_schema_versions($module);
...
+        $last_removed = \Drupal::moduleHandler()->invoke($module, 'update_last_removed');
...
+      $updates = drupal_get_schema_versions($module);
...
+    foreach (\Drupal::keyValue('system.schema')->getAll() as $module => $schema) {
...
+    $role_names = \Drupal::service('config.storage')->listAll($prefix);
...
+      $config = \Drupal::configFactory()->getEditable("user.role.$rid");

All this dependencies makes me think about service, OTOH there still procedural things used that means it will be hard to test this new class

voleger’s picture

Just reroll.

voleger’s picture

StatusFileSize
new87.57 KB
voleger’s picture

StatusFileSize
new88.15 KB
new862 bytes

Add new usage.

voleger’s picture

StatusFileSize
new88.41 KB
new876 bytes

Updated docs and removed include of the file.

andypost’s picture

Checked new class and I really like to split static methods from real service
Also new class should not use module_load_include() and others functions which supposed to be deprecated in other "include conversions" - so better to add protected methods for them with @todo to related conversion issue

+++ b/core/lib/Drupal/Core/Update/Update.php
@@ -0,0 +1,715 @@
+    $extension_config = \Drupal::configFactory()->getEditable('core.extension');
...
+    $system_schema = drupal_get_installed_schema_version('system');
...
+    $requirements = \Drupal::moduleHandler()->invokeAll('requirements', ['update']);
...
+    \Drupal::keyValue('system.schema')->set($module, $schema_version);
+    \Drupal::service('extension.list.profile')->reset();
+    \Drupal::service('extension.list.module')->reset();
+    \Drupal::service('extension.list.theme_engine')->reset();
+    \Drupal::service('extension.list.theme')->reset();
+    drupal_static_reset('drupal_get_installed_schema_version');
...
+      drupal_set_installed_schema_version($module, $number);
...
+    module_load_include('php', $module, $module . '.post_update');
...
+          \Drupal::service('update.post_update_registry')->registerInvokedUpdates([$function]);
...
+    $modules = drupal_get_installed_schema_version(NULL, FALSE, TRUE);
...
+        $last_removed = \Drupal::moduleHandler()->invoke($module, 'update_last_removed');
...
+      $updates = drupal_get_schema_versions($module);
...
+    return $number <= drupal_get_installed_schema_version($module);
...
+    foreach (\Drupal::keyValue('system.schema')->getAll() as $module => $schema) {
...
+      module_load_install($module);
...
+    $role_names = \Drupal::service('config.storage')->listAll($prefix);
...
+      $config = \Drupal::configFactory()->getEditable("user.role.$rid");

All this services & functions will need injection means this class can't be static

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

andypost’s picture

It needs reroll but maybe postpone it on other conversions?

voleger’s picture

Good option. In that case, we need to clarify the issues which the issue have to be postponed on.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

andypost’s picture

+++ b/core/includes/update.inc
@@ -109,14 +82,15 @@ function update_check_requirements() {
+  @trigger_error("update_set_schema() is deprecated in Drupal 8.7.0-dev and will be removed before Drupal 9.0.0. Use  \Drupal\Core\Update\Update::setSchema() instead. See https://www.drupal.org/node/3013060", E_USER_DEPRECATED);
+  Update::setSchema($module, $schema_version);

It slightly related to #2908886-65: Split off schema management out of schema.inc

andypost’s picture

andypost’s picture

+++ b/core/tests/Drupal/KernelTests/Core/Update/UpdateLegacyTest.php
@@ -0,0 +1,242 @@
+   * Tests update_check_incompatibility() function.
...
+    $this->assertTrue(update_check_incompatibility('incompatible_module'));

I filed separate issue to remove the function without replacement #3150726: Deprecate update_check_incompatibility as unused

andypost’s picture

The next hunk to decouple is update_system_schema_requirements(), update_check_requirements(), _update_fix_missing_schema() - the only usage found is \Drupal\system\Controller\DbUpdateController::handle() so this requirements could be moved to system module

EDIT there's \Drupal\Core\Updater\Module::getSchemaUpdates() so part of conversion could be moved to extension lists

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

longwave’s picture

Opened #3210900: Deprecate update_set_schema() for removal as I don't think that function is needed at all.

andypost’s picture

Re #38 - there's usage of update_check_requirements() in drush https://github.com/drush-ops/drush/blob/dbdb6733655231687d8ab68cdea6bf9f...

Probably better to file separate issue as well

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

voleger’s picture

Title: [PP-3] Convert the update.inc file functions to a class » [PP-2] Convert the update.inc file functions to a class
voleger’s picture

Version: 9.4.x-dev » 10.0.x-dev
andypost’s picture

Title: [PP-2] Convert the update.inc file functions to a class » [PP-1] Convert the update.inc file functions to a class
danielveza’s picture

Status: Postponed » Active

According to #43, this was postponed on a number of other issues. All those issues are now committed, so I think work can continue on this one.

andypost’s picture

Title: [PP-1] Convert the update.inc file functions to a class » Convert the update.inc file functions to a class
Status: Active » Needs work

Patch needs update as there is only left - update requirements and batch to run'em

elber’s picture

Hi @andypost can you give more details about it ?

voleger’s picture

Assigned: Unassigned » voleger

Updating the patch

voleger’s picture

Version: 10.0.x-dev » 10.1.x-dev
andypost’s picture

Version: 10.1.x-dev » 11.x-dev

voleger’s picture

Assigned: voleger » Unassigned
Status: Needs work » Needs review

Opened MR, rebased initial work against 11.x branch. Ready for the review.

smustgrave’s picture

Status: Needs review » Needs work

Reran the tests and the 1 failure seems legit.

Did not review.

voleger’s picture

Status: Needs work » Needs review

Addressed review comments

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs Review Queue Initiative

Wonder if the CR needs updating searching for

update_fix_compatibility
update_check_incompatibility
update_set_schema
update_replace_permissions

I don't find them in updates.inc maybe deprecated and removed already?

andypost’s picture

Added suggestion and tests should compare result of old vs new function calls

CR needs clean-up as since 8.7 core already deprecated and removed some functions, for example via related

quietone made their first commit to this issue’s fork.

quietone’s picture

I removed the todo in \Drupal\Core\Updater\Module::getSchemaUpdates because it didn't have an associated issue. Also, I can find no usages of the method either. That needs some investigation.

quietone’s picture

quietone’s picture

Status: Needs work » Needs review

The MR is updated and tests are passing. Time to get another round of review.

quietone’s picture

I have updated the CR,

smustgrave’s picture

Status: Needs review » Needs work

Left some comments on the MR.

quietone’s picture

Status: Needs work » Needs review

I am not sure about the testing being asked for. The testing being asked for is to compare result of the old function execution with the replacement result. However, there are no explicit tests of this legacy code. Any testing of it is implicit in other update tests which rely on this code. And we have all tests passing and I think that is the best we can do.

Then, that means that UpdateLegacyTest is only testing that the exception is thrown. I am not sure we need to do that in this case either. So, I discussed this with catch and he did say that UpdateLegacyTest can be removed.

Therefor, I have removed that test.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

All green and my feedback was addressed by #66, so marking this.

xjm’s picture

Status: Reviewed & tested by the community » Needs review

This will be good to clean up. That said, though, the diff is massive -- 11 files, +868, −464. That is a total of 1332 changed lines, five to ten times as many lines as a reviewer can effectively review. (And it's exponential decay -- that means having this all lumped into one change set means a reviewer will find 1-5% of the bugs they would find if the scoping were better.) Especially since it's moved lines, which git does not deal well with.

Is there any way we could split this up into stages?

smustgrave’s picture

I see your point @xjm but think the high number of changes is due to whole functions being moved. But if it's a blocker I imagine each function could be broken out to individual tickets.

xjm’s picture

Moved functions are harder to review than changed ones. So I really would suggest splitting this into logical steps. What has to happen first? What's second? Etc. You can use the original MR to make sure everything will still end up in the same final state, but a little git add -p etc. could make this easier to get in.

smustgrave’s picture

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.