Problem/Motivation

Modules can add base fields to entity types using hook_entity_base_field_info(). Entity displays that use these fields don't have a dependency on these modules and on uninstallation of the module that provides them, the base fields are not properly cleaned up.

This was discovered in #2850973: ConfigEntityInterface::onDependencyRemoval() called with incorrect dependency list and is relevant to the uninstall of the scheduler module.

This is a major issue because it is a data integrity issue. Entity displays have configuration for base fields that no longer exist.

Proposed resolution

Add dependency in \Drupal\Core\Entity\EntityDisplayBase::calculateDependencies() and remove field \Drupal\Core\Entity\EntityDisplayBase::onDependencyRemoval() on module uninstall.

Remaining tasks

Needs tests
Needs upgrade path

User interface changes

None

API changes

None

Data model changes

New dependencies information in entity displays.

Comments

alexpott created an issue. See original summary.

alexpott’s picture

Status: Active » Needs review
StatusFileSize
new2.62 KB
new5.56 KB

He's a bug fix and test for it. Now to write the upgrade path :(

chr.fritsch’s picture

Since this is a major bug, shouldn't it go into 8.4.x?

alexpott’s picture

@chr.fritsch well there's going to be an upgrade path and generally we try not to add upgrade paths to patch releases.

Status: Needs review » Needs work

The last submitted patch, 2: 2924256-2.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
Issue tags: +Needs upgrade path tests
StatusFileSize
new3.85 KB
new3.41 KB
new7.87 KB

Ah I see - we don't need to add a dependency when the module that provides the base field is the same one that provides the entity type. This is because that is already taken care of by

    // Create dependency on the bundle.
    $bundle_config_dependency = $target_entity_type->getBundleConfigDependency($this->bundle);
    $this->addDependency($bundle_config_dependency['type'], $bundle_config_dependency['name']);

And therefore the module will be in the dependency chain and is not worth adding again.

Patch attached also has the start of some work on the upgrade path tests...

The last submitted patch, 6: 2924256-6.test-only.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

The last submitted patch, 2: 2924256-2.test-only.patch, failed testing. View results

alexpott’s picture

Issue tags: -Needs upgrade path tests
StatusFileSize
new1.86 KB
new207.36 KB

Here's an update path test. New database needed because there are other updates like system_post_update_add_region_to_entity_displays() that would do the necessary saves to running with an old database is not good.

Interdiff doesn't contain the dumped db because it's not that helpful unless you can decompress stuff just by looking at it :)

amateescu’s picture

Status: Needs review » Needs work

Patch is looking pretty good, here's a review:

  1. +++ b/core/lib/Drupal/Core/Entity/EntityDisplayBase.php
    @@ -298,12 +299,19 @@ public function calculateDependencies() {
    +        // If the field is provider by a module different than the module that
    

    "is provider" -> "is provided" ;)

  2. +++ b/core/modules/system/system.post_update.php
    @@ -81,3 +81,15 @@ function system_post_update_classy_message_library() {
    +    // calculateDependencies() will add any missing base field dependencies.
    

    Let's also mentioned the class name, not just the generic method name :)

  3. +++ b/core/modules/system/tests/src/Functional/Update/UpdateEntityDisplayBaseFieldDependenciesTest.php
    @@ -0,0 +1,32 @@
    +  public function testUpdate() {
    +    $this->runUpdates();
    +  }
    

    This is not really useful since it doesn't do any assertion that a dependency has been added, so we need an assertion before running the updates on a set of dependencies and another assertion after the update process to check that the set of dependencies has been updated.

  4. +++ b/core/tests/Drupal/KernelTests/Core/Entity/EntityDisplayBaseTest.php
    @@ -150,4 +160,35 @@ public function testOnDependencyRemoval() {
    +  public function testBaseFields() {
    

    How about renaming this to testBaseFieldDependencies()?

Here's an update path test. New database needed because there are other updates like system_post_update_add_region_to_entity_displays() that would do the necessary saves to running with an old database is not good.

How about using an update fixture that makes the system act like system_post_update_add_region_to_entity_displays() already ran instead of providing yet another db dump :)

You can find an example where we're already doing this in core/modules/hal/tests/fixtures/update/drupal-8.rest-hal_update_8301.php.

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new2.29 KB
new207.42 KB

Thanks for the review @amateescu

  1. Fixed
  2. Fixed
  3. There are assertions - that was the code in #6 - #9 added some.
  4. Fixed

Run the update gz - it's not just about the updates that have run - the gzip also has entity_test and entity_test_extra installed. It has state to make \entity_test_extra_entity_base_field_info() actually work. It has entity displays (both form and view) for the entity_test entity type configured and it has the necessary entity updates run. To me that's enough to just a dump. I think maybe we should always add a dump for each minor release and it would be awesome if there was some tool that would help use make partial files like core/modules/hal/tests/fixtures/update/drupal-8.rest-hal_update_8301.php because at the moment the process seems very fragile and prone to making slightly sites fake sites as it is easy to miss something from a partial.

amateescu’s picture

Ohh, right, I missed the interdiff from #9. Those assertions look exactly like what I would expect from this test :)

+++ b/core/modules/system/tests/src/Functional/Update/UpdateEntityDisplayBaseFieldDependenciesTest.php
@@ -18,7 +18,7 @@
+      __DIR__ . '/../../../../tests/fixtures/update/drupal-8.4.2.bare.standard.entity_test_extra_with_base_fields.php.gz',

I don't think we should have this dump specific to version 8.4.2. I discussed this scenario with @catch some time ago and the outcome of that discussion was that we should try to allow upgrades across multiple minor releases at a time (directly from 8.1.x to 8.4.0 for example) on a best-effort basis, meaning that we can't guarantee that it will work but we also should *not* actively prevent that upgrade from working.

In our case, the bug that we're fixing here was not introduced in 8.4.2, it probably exists before 8.0.0 so the dump that we should be using is one that has 8.0.0 (maybe rc1 even) and uses the standard profile, because chances are that you can easily find an existing form/view display that is missing dependencies of a base field which should be fixed by this update.

The reason I'm not keen on adding db dump so easily because they're essentially a black box, no one but the person who created them actually knows what they contain (sometimes even that person forgets :D), so they are realistically one-off blobs of data that we keep piling on without much enticement for re-usability :)

alexpott’s picture

you can easily find an existing form/view display that is missing dependencies of a base field

I don't think so. This is because you only get extra dependencies when base fields are added by an another module than the one that provides the entity as the entity display already has a dependency on the entity type / bundle.

alexpott’s picture

I thought maybe I spoke to quick re #13 because of things like path_entity_base_field_info() but actually the entity displays already have a dependency on path because the field type it is provided by path. So the situation is we need hook_entity_base_field_info() where the added fields are not provided by the same module and I don't think we have that in any core install - as example in real code would be content_moderation_entity_base_field_info() but no profile is configured to use that module yet. Hmmm... that said maybe content_translation_entity_base_field_info() might cause this. Maybe drupal-8.filled.standard.php.gz has something for us. That said I'm not really sure how this actually gets us around the black box issue because drupal-8.filled.standard.php.gz is an unknown too really and just hacking the run updates to not call system_post_update_add_region_to_entity_displays() puts the system in a state it could never actually be in.

alexpott’s picture

StatusFileSize
new1.3 KB
new207.96 KB

Unfortunately there are no entity view displays in the Drupal 8 filled standard dump so that's not an option. So to open up the black box I've added documentation to the test to explain exactly what is in the dump and how it was made.

alexpott’s picture

StatusFileSize
new1.66 KB
new207.95 KB

Slightly better comment.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

jonathan1055’s picture

Issue summary: View changes

Thanks for your work on this. I can't recall whether I did any manual testing when the issue was first started, but I will check it again.

I re-queued the patch in #16 but it needs a re-roll.

I'm trying to sort out #3038746: Remove deprecated EntityDefinitionUpdateManagerInterface::applyUpdates() from hook_install and hook_uninstall and having the fields remain after uninstall is making the task harder to test.

jonathan1055’s picture

I have manually re-tested the patch in #16 on 8.5.11 (just the first two files, not the addition to testing) and it appiles OK and does the job required. After uninstalling the module the two base fields are immediately removed from core.entity_form_display.node.article.default (checked using config inspector). The fields are also removed from the content type 'manage form display' tab.

I have not tested on 8.6 or 8.7 as the patch was created for 8.5.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new132 bytes

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.