Problem/Motivation

The problem is identical to the one described (and fixed) in #2655162: Fatal error when updating or deleting an entity type after removing its class/definition from the codebase, but it seems that revisionable entity types need special care.

Here's the call stack for the exception:

Drupal\Core\Entity\EntityTypeManager->getDefinition()
Drupal\Core\Entity\EntityFieldManager->buildBaseFieldDefinitions()
Drupal\Core\Entity\EntityFieldManager->getBaseFieldDefinitions()
Drupal\Core\Entity\ContentEntityType->getRevisionMetadataKeys()
Drupal\Core\Entity\Sql\DefaultTableMapping::create()
Drupal\Core\Entity\Sql\SqlContentEntityStorage->getCustomTableMapping()
Drupal\Core\Entity\Sql\SqlContentEntityStorage->getTableMapping()
Drupal\Core\Entity\Sql\SqlContentEntityStorage->initTableLayout()
Drupal\Core\Entity\Sql\SqlContentEntityStorage->__construct()
156, Drupal\Core\Entity\Sql\SqlContentEntityStorage::createInstance()
Drupal\Core\Entity\EntityTypeManager->createHandlerInstance()
Drupal\Core\Entity\EntityTypeListener->onEntityTypeDelete()
Drupal\Core\Entity\EntityDefinitionUpdateManager->uninstallEntityType()

In short: EntityDefinitionUpdateManager instantiates a SQL storage handler, which in turn instantiates a DefaultTableMapping, which needs to get the revision metadata keys of the entity type, so it calls ContentEntityType::getRevisionMetadataKeys() which tries to retrieve the (live, in-code) base field definitions, and EntityFieldManager::buildBaseFieldDefinitions() tries to get the entity type definition from the entity type manager, which results in an exception.

Proposed resolution

TBD.

Remaining tasks

Find a fix, review, etc.

User interface changes

Nope.

API changes

Nope.

Data model changes

Nope.

Release notes snippet

TBD.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

amateescu created an issue. See original summary.

amateescu’s picture

Status: Active » Needs review
FileSize
1.47 KB

Here's a test-only patch for now.

amateescu’s picture

My initial attempt to fix this is to call getActiveFieldStorageDefinitions() in ContentEntityType::getRevisionMetadataKeys(), but that results in some failing in update tests, as can be seen in #3062434-14: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index.

Status: Needs review » Needs work

The last submitted patch, 3: 3067024-3.patch, failed testing. View results

amateescu’s picture

Status: Needs work » Needs review
FileSize
4.96 KB
2.24 KB

Found the problem! block_content_update_8003() must run before system_update_8400(), otherwise the revision_created and revision_user fields added by block_content_update_8003() are not moved to the revision table.

andypost’s picture

plach’s picture

This looks great to me!

I was initially concerned about switching from live definitions to active ones under the hood, but then I checked all the invocations of the ::getRevisionMetadataKeys() method and it seems that the only places that would actually require the live definitions (EntityFieldManager::buildBaseFieldDefinitions() and RevisionLogEntityTrait::revisionLogBaseFieldDefinitions()) are using the flag to skip fetching definitions to avoid recursion. This seems a strong indicator that the change is definitely in the right direction.

This change might affect contrib/custom update code relying ::getRevisionMetadataKeys(), as we saw with system_update_8400(), but updates should always rely on active definitions to be predictable, so this change would actually allow to surface sneaky update path bugs.

I have only a minor remark:

+++ b/core/modules/system/tests/src/Functional/Entity/Update/MoveRevisionMetadataFieldsUpdateTest.php
@@ -60,6 +60,11 @@ public function testSystemUpdate8400() {
+      // Only run the following assertions for the test entity types.

I assume that the reason here is that we only have test data for test entity types, but let's make this comment more explicit.

amateescu’s picture

Thanks for the review, @plach!

Updated the comment and also changed the check to use in_array() so it's a bit more clear :)

plach’s picture

Status: Needs review » Reviewed & tested by the community

Great, thanks!

effulgentsia’s picture

+++ b/core/lib/Drupal/Core/Entity/ContentEntityType.php
@@ -86,7 +86,7 @@ public function getRevisionMetadataKeys($include_backwards_compatibility_field_n
-      $base_fields = \Drupal::service('entity_field.manager')->getBaseFieldDefinitions($this->id());
+      $base_fields = \Drupal::service('entity_field.manager')->getActiveFieldStorageDefinitions($this->id());

This issue is currently assigned to 8.7.x, not 8.8.x. It's fixing a major bug, so it's reasonable to want that in 8.7.x, but are we sure this is a safe change to make in a patch release?

hchonov’s picture

+++ b/core/modules/system/tests/src/Functional/Entity/Update/MoveRevisionMetadataFieldsUpdateTest.php
@@ -38,7 +38,7 @@ public function setDatabaseDumpFiles() {
-    foreach (['entity_test_revlog', 'entity_test_mul_revlog'] as $entity_type_id) {
+    foreach (['block_content', 'entity_test_revlog', 'entity_test_mul_revlog'] as $entity_type_id) {

@@ -60,6 +60,12 @@ public function testSystemUpdate8400() {
+      // The database dump contains data just for the test entity types, so the
+      // following assertions can only run for them.
+      if (!in_array($entity_type_id, ['entity_test_revlog', 'entity_test_mul_revlog'], TRUE)) {
+        continue;
+      }

By adding and later excluding block_content we only want to ensure, that its revision metadata keys are placed in the correct tables. This is a good addition, however it makes me think why don't we do that check for all entity types in the system instead of only for three of them?

--------

Found the problem! block_content_update_8003() must run before system_update_8400(), otherwise the revision_created and revision_user fields added by block_content_update_8003() are not moved to the revision table.

This means, that if in custom/contrib there is a similar update it should necessary run before system_update_8400(), right? Otherwise an inconsistent state might be the side effect of the change.

--------

This issue is currently assigned to 8.7.x, not 8.8.x. It's fixing a major bug, so it's reasonable to want that in 8.7.x, but are we sure this is a safe change to make in a patch release?

I've checked all the places in core, where we retrieve either a single revision metadata key (which always uses the BC layer) or all revision metadata keys by explicitly requesting the BC layer.

Now with the patch in system_update_8400() we will be retrieving the live base field definitions, but we will extract the revision metadata keys through the BC layer based on the active base field definitions. I don't see a problem with this beside that we might find a revision metadata key, which does not exist in the live base field definitions, but only in the active ones. However this does not represent an issue, because we do an intersect. If we are to change the update it will be only because of consistency. I've just wanted to share this with you, so that you can consider such use cases and think if there might be some tricky edge cases.

I, personally, support the change and cannot think of any problems that it could cause beside that updates adding revision metadata keys now should necessary run before system_update_8400(). Maybe we could add a post update, in which we check that everything went well and all revision metadata keys are placed in the correct tables? If not, then inform that something went wrong? Or as an alternative maybe we could check in \Drupal\Core\Entity\EntityDefinitionUpdateManagerInterface::installFieldStorageDefinition() if we are installing a revision metadata key/field after system_update_8400() has been executed and check if the fields are placed in the correct tables?

--------

After having written all that, I don't really understand why if block_content_update_8003() runs after system_update_8400() the revision metadata fields will land in the wrong tables? What happens if somebody installs Drupal 8.8.0 and adds a similar update as block_content_update_8003()?

Is it because in \Drupal\Core\Entity\Sql\DefaultTableMapping::create() with the patch applied we retrieve the revision metadata keys from the active instead from the live base field definitions?

amateescu’s picture

I've looked into it with fresh eyes and found that the problem is not caused by system_update_8400(), but by system_update_8501() in combination with the BC layer for revision metadata keys. This is what happens:

- system_update_8501() runs and at some point calls $definition_update_manager->updateEntityType($installed_entity_type);

- before calling updateEntityType(), $installed_entity_type has only one entry in the revision_metadata_keys array: ['revision_default' => 'revision_default']

- EntityTypeListener::onEntityTypeUpdate() instantiates a storage handler for the given $entity_type object

- \Drupal\Core\Entity\Sql\SqlContentEntityStorage instantiates a table mapping object in its constructor, through initTableLayout()

- DefaultTableMapping::create() calls $entity_type->getRevisionMetadataKeys(), and the BC layer populates the revision_metadata_keys property of the $entity_type object with a value for the revision_log_message key

- going back to EntityTypeListener::onEntityTypeUpdate(), after the storage instantiation described above, the $entity_type object has two entries in revision_metadata_keys now: revision_default and revision_log_message

- the entity type definition with two revision metadata keys gets saved in the last installed repository

- at some later time block_content_update_8003() runs, and, when the field definitions are installed, the storage schema instantiates a table mapping object to determine where to add the new columns, and the BC layer from ContentEntityType::getRevisionMetadataKeys() is not triggered anymore because the entity type object has an entry for the revision_log_message revision metadata key, which means the table mapping is not aware that the new revision_created and revision_user fields are revision metadata keys and installs them in the wrong table

The tl;dr version is: the BC layer for revision metadata keys can subtly alter an entity type definition at runtime (by design, sadly), and that altered definition can end up in the last installed repository as a side-effect of simply instantiating a storage handler for that entity type.

Given all that, I think a better fix would be to pass a cloned entity type definition to the table mapping.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 12: 3067024-12.patch, failed testing. View results

amateescu’s picture

Title: Revisionable entity types can not be uninstalled if their code doesn't exist anymore » Revisionable entity types using SqlContentEntityStorage can not be uninstalled if their code doesn't exist anymore
Issue tags: -blocker

Ok, it seems that the only way to fix this mess is to remove the BC layer for revision metadata keys, and that won't be easy to do.

Therefore I went with a different approach for uninstalling a revisionable entity type in #3062434-22: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index, so this issue is not a blocker anymore.

hchonov’s picture

What about controlling by a flag or something similar whether to retrieve the live or the active field storage definitions for the BC layer? When an entity type is to be uninstalled then we don't care about the live field storage definitions. Maybe we could simply check if the entity class exist - not a beautiful solution, but a one that would work without any breaks I think.

We might have to do this fallback in the EntityFieldManager so that no matter how one is about to access the field storage definitions of a missing but not yet uninstalled entity type then the active field storage definitions will be returned. Wouldn't such a general solution solve even more potential problems for a missing entity type?

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.9 was released on November 6 and is the final full bugfix release for the Drupal 8.7.x series. Drupal 8.7.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.8.0 on December 4, 2019. (Drupal 8.8.0-beta1 is available for testing.)

Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

amateescu’s picture

Title: Revisionable entity types using SqlContentEntityStorage can not be uninstalled if their code doesn't exist anymore » Add test coverage for uninstalling revisionable entity types whose code doesn't exist anymore
Version: 8.9.x-dev » 9.1.x-dev
Priority: Major » Normal
FileSize
1.47 KB

The entity metadata keys BC layer has been removed in 9.x, so this problem only exists in 8.9.x now. I think we should let it be and just add the extra test coverage in 9.0.x and 9.1.x to ensure that we don't break this again in the future.

Here's the test-only patch from #2, which doesn't fail anymore on 9.x.

amateescu’s picture

Status: Needs work » Needs review

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

More testing the better.

longwave’s picture

Requeued tests against 10.x, if they pass I don't see why this can't go in.

amateescu’s picture

3-years-ago me should've known that you always need to pair a negative assertion like this with a positive one, otherwise it's meaningless since the table might have never existed in the first place.

  • catch committed ea27636 on 10.0.x
    Issue #3067024 by amateescu, hchonov, plach: Add test coverage for...
  • catch committed 346d609 on 10.1.x
    Issue #3067024 by amateescu, hchonov, plach: Add test coverage for...
  • catch committed 3e5e417 on 9.5.x
    Issue #3067024 by amateescu, hchonov, plach: Add test coverage for...
catch’s picture

Version: 10.1.x-dev » 9.5.x-dev
Status: Reviewed & tested by the community » Fixed

Agreed with the extra assertions so +1 RTBC. Committed/pushed to 10.1.x and cherry-picked back to 9.5.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.