Applying entity schema updates fail when you install a module that changes the entity type definition (e.g. storage handler) and adds new base field definitions at the same time (like I try to do with Multiversion module).

The reason why it fails is because when installing the new entity schema it already creates it with all the base fields (including the new ones). Then when EntityDefinitionUpdateManager tries to add the new base field it fails because they already exists.

Comments

dixon_’s picture

Status: Active » Needs review
StatusFileSize
new6.52 KB

The first idea that came to my mind was to simply handle the exception gracefully when creating the field fails because it already exists (e.g. SchemaObjectExistsException). The exception is still logged as a warning just to make it more transparent what's going on. But this is obviously not a very pretty solution, so we probably should find a nicer way.

Status: Needs review » Needs work

The last submitted patch, 1: 2342543-entity-definition-updates-fail-1.patch, failed testing.

catch’s picture

Priority: Major » Critical
Issue tags: +D8 upgrade path
dixon_’s picture

Status: Needs work » Needs review
StatusFileSize
new3.1 KB

Couldn't figure out the circular container references when passing the logger. So I restored to using \Drupal::logger() instead which generally seems accepted throughout core anyhow.

dixon_’s picture

Status: Needs review » Needs work

Just talked to @effulgentsia and we discussed a more proper way to solve this. We would essentially let SqlContentEntityStorageSchema::onEntityTypeCreate() be a wrapper around a new protected method, something like createEntityTypeSchema(), that's able to install the definition you give it.

That way while creating the schema for the new storage handler, in SqlContentEntityStorageSchema::onEntityTypeUpdate(), we can pass it the last installed definition, so that the fields that were added in the same process can be applied without problem.

jeqq’s picture

Status: Needs work » Needs review
StatusFileSize
new3.15 KB

Plain re-roll for #4.

xjm’s picture

Issue tags: +Entity Field API
dixon_’s picture

Just talked to @effulgentsia and we discussed a more proper way to solve this. We would essentially let SqlContentEntityStorageSchema::onEntityTypeCreate() be a wrapper around a new protected method, something like createEntityTypeSchema(), that's able to install the definition you give it.

That way while creating the schema for the new storage handler, in SqlContentEntityStorageSchema::onEntityTypeUpdate(), we can pass it the last installed definition, so that the fields that were added in the same process can be applied without problem.

I've been playing around with this implementation, and the problem here is that since the updates are being applied in two steps...

  1. the entity type updates
  2. the field updates

...there's no way for step 1 to know if any new fields are going to be added since that information isn't passed along through the event. So we simply don't have enough information to know if we need to base the updates on the last definition or not (in order for subsequent field updates to not collide).

Further, this approach fail in cases where there are other things like index updates that actually do need to be created as part of step 1.

So rather than passing the necessary information down through the event (which would mean additions and changes to the API) we can instead simply bubble up information needed to take the right decisions. This change requires a lot smaller/simpler changes to the API. Basically all we need to know is if the schema handler recreated the entity schema as part of the step 1 update. Because if the schema was recreated it means that any new field storage definitions already got created as part of this process. In this case we should not carry out the updates as part of EntityDefinitionUpdateManager::applyUpdates(), because that would result in conflicts.

Attached is a patch that demonstrates this approach. This patch implements only the minimum of what's required to solve the problem. For consistency we could perhaps make onEntityTypeCreate and onEntityTypeDelete bubble up (e.g. return) it's results as well. But since those cases are a lot more straight forward I'm not sure that's really needed...

dixon_’s picture

This bug is blocking a stable release of the Multiversion contrib module.

dixon_’s picture

StatusFileSize
new7.23 KB
new2.52 KB

Updated the patch with some docs, comment fixes and made ::onEntityTypeUpdate() a bit more consistent by always returning a status.

fago’s picture

Status: Needs review » Needs work
  1. +++ b/core/lib/Drupal/Core/Entity/EntityDefinitionUpdateManager.php
    @@ -108,14 +115,17 @@ public function getChangeSummary() {
    +      // definition was not already recreated, because any any new field storage
    

    two any

  2. +++ b/core/lib/Drupal/Core/Entity/EntityTypeListenerInterface.php
    @@ -27,6 +27,11 @@ public function onEntityTypeCreate(EntityTypeInterface $entity_type);
    +   *   Returns EntityDefinitionUpdateManager::DEFINITION_UPDATED if the storage
    ...
    +   *   was deleted and recreated or NULL if no action was taken.
    ...
       public function onEntityTypeUpdate(EntityTypeInterface $entity_type, EntityTypeInterface $original);
     
    

    While the general architecture of the solution would be fine to me, the implementation is a bit problematic as the schema handler currently implements a listener concept. It's weird for a listener to provide additional information via return values. In particular if the return values stem from an "unrelated" different interface :)

    Also, we've the current problem/glitch that we do not invoke onFieldStorageDefinitionCreate() when creating the entity type either - so I'm wondering whether shouldn't just always compile a complete $change_list and pass it own to all listeners. So listeners can decide whether to act based upon the complete change_list? That would allow us to account for any other change combination issues we might run into in future as well.

    Another option would be to just override the onEntityUpdate() method in the EntityStorageSchemaInterface ?

fago’s picture

Issue tags: +Ghent DA sprint

Discussed with plach, yched and berdir on how we could solve that.

Came up with a possible way with plach: It's the same class that consumes both events, so we could statically keep the events processed before and then skip the field event once the entity event was there before. I could see this a bit troublesome only if on a single page load multiple changelists are compiled instead of a single one - what probably mostly happens in tests, but might happen due to custom update scripts also. To mitigate we'd have to provide changelist ids, set them on event objects and make sure listeners receive the events.

But then, plach came up with another way simpler solution: Just move around the around ordering and apply field updates first - so let's try that.

fago’s picture

Assigned: dixon_ » fago
fago’s picture

Status: Needs work » Needs review
StatusFileSize
new2.85 KB

Tried re-ordering things and took over the test case - looks like it doesn't solve the problem, not sure why yet.

fago’s picture

StatusFileSize
new4.13 KB

silly me. this should work.

The last submitted patch, 14: d8_entity_definition_update.patch, failed testing.

plach’s picture

+++ b/core/lib/Drupal/Core/Entity/EntityDefinitionUpdateManager.php
@@ -135,6 +127,12 @@ public function applyUpdates() {
+      // Process entity type definition changes.
+      if (!empty($change_list['entity_type']) && $change_list['entity_type'] == static::DEFINITION_UPDATED) {
+        $entity_type = $this->entityManager->getDefinition($entity_type_id);
+        $original = $this->entityManager->getLastInstalledDefinition($entity_type_id);
+        $this->entityManager->onEntityTypeUpdate($entity_type, $original);
+      }
     }

This fix was originally introduced in #1916790: Convert translation metadata into regular entity fields. There I added also a comment:

// Process entity type definition changes after storage definitions ones,
// as in some conditions a change in the field storage definition can also
// be detected as an entity type definition change. This avoids the change
// to be applied twice (resulting in an unrecoverable error).
fago’s picture

Assigned: fago » Unassigned
StatusFileSize
new1008 bytes
new4.37 KB

Thanks. Refactored the comment a bit to take account of this issue and updated patch.

plach’s picture

+++ b/core/modules/system/src/Tests/Entity/EntityDefinitionUpdateTest.php
@@ -478,6 +478,25 @@ public function testDefinitionEvents() {
+    // Entity type updates creates base fields as well, thus make sure doing

Typo, should be "create". Aside from that this looks good to me.

fago’s picture

Thanks, fixed.

plach’s picture

Status: Needs review » Reviewed & tested by the community

Let's get this in.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

This issue addresses a critical bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 4738109 and pushed to 8.0.x. Thanks!

  • alexpott committed 4738109 on 8.0.x
    Issue #2342543 by fago, dixon_, jeqq: Applying entity schema updates...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.