Last installed field storage definitions not used when loading from / saving to dedicated tables

Problem/Motivation

I tried to solve a considerably simple task on a site:

1. Add N new fields to an entity type
2. Change cardinality of the same (base) field on N entities of M entity types

These steps were part of one single deploy, one change request. It took me more time than I expected to be able to test and write the following update hooks:

<?php

declare(strict_types = 1);

use Drupal\Core\Field\BaseFieldDefinition;
use Drupal\Core\StringTranslation\TranslatableMarkup;
use Drupal\Core\Utility\UpdateException;
use Drupal\my_entity_1\Entity\MyEntity1;

/**
 * Implements hook_update_dependencies().
 */
function my_entity_1_update_dependencies(): array {
  return [
    'my_entity_1' => [
      // This update hook must run after this update hook, otherwise
      // update fails, because one of the entity storages tries to read from
      // the new individual ip addresses db table when it does not exist yet.
      8001 => [
        'my_deploy' => 8001,
      ],
    ]
  ];
}

/**
 * Adds "conditional fields" related fields to MyEntity1 entity.
 */
function my_entity_1_update_8001(array &$sandbox): void {
  $update_manager = \Drupal::entityDefinitionUpdateManager();
  /** @var \Drupal\Core\Field\FieldDefinitionInterface[] $new_fields */
  $all_base_fields = MyEntity1::baseFieldDefinitions(\Drupal::entityTypeManager()->getDefinition('my_entity_1'));
  $new_fields = [
    'new_field_1',
    'new_field_2',
    'new_field_3',
    'new_field_4',
  ];

  foreach ($new_fields as $field_name) {
    $update_manager->installFieldStorageDefinition($field_name, 'my_entity_1', 'my_entity_1', $all_base_fields[$field_name]);
  }
}

<?php

/**
 * Change cardinality of the IP addresses field.
 */
function my_deploy_update_8001(): void {
  // We must do both entity updates here otherwise one of the entity updates
  // fails because the entity storage tries to read data from the individual
  // ip addresses db table when it is not available yet.
  /** @var \Drupal\my_entity2\Storage\MyEntity2Storage $my_entity2_storage */
  $my_entity2_storage = \Drupal::entityTypeManager()->getStorage('my_entity2');
  $my_entity2_entity_type = \Drupal::entityTypeManager()->getDefinition('my_entity2');
  /** @var \Drupal\my_entity3\Storage\MyEntity3Storage $my_entity3_storage */
  $my_entity3_storage = \Drupal::entityTypeManager()->getStorage('my_entity3');
  $my_entity3_entity_type = \Drupal::entityTypeManager()->getDefinition('my_entity3');

  $my_entity2_stored_ip_addresses = \Drupal::database()->select($my_entity2_entity_type->getRevisionTable(), 'r')
    ->fields('r', [$my_entity2_entity_type->getKey('revision'), 'ip_addresses'])
    ->condition('r.ip_addresses', NULL, 'IS NOT NULL')
    ->execute()
    ->fetchAllKeyed();
  $my_entity3_stored_ip_addresses = \Drupal::database()->select($my_entity3_entity_type->getRevisionTable(), 'r')
    ->fields('r', [$my_entity3_entity_type->getKey('revision'), 'ip_addresses'])
    ->condition('r.ip_addresses', NULL, 'IS NOT NULL')
    ->execute()
    ->fetchAllKeyed();

  // Store and remove previous values so we can change the cardinality.
  foreach ($my_entity2_storage->loadMultipleRevisions(array_keys($my_entity2_stored_ip_addresses)) as $entity) {
    $entity->set('ip_addresses', NULL);
    $entity->save();
  }
  foreach ($my_entity3_storage->loadMultipleRevisions(array_keys($my_entity3_stored_ip_addresses)) as $entity) {
    $entity->set('ip_addresses', NULL);
    $entity->save();
  }

  /** @var \Drupal\Core\Entity\EntityDefinitionUpdateManagerInterface $entity_definition_update_manager */
  $entity_definition_update_manager = \Drupal::entityDefinitionUpdateManager();
  foreach (['my_entity2', 'my_entity3'] as $entity_type) {
    /** @var \Drupal\Core\Field\BaseFieldDefinition $field_storage_definition */
    $field_storage_definition = $entity_definition_update_manager->getFieldStorageDefinition('ip_addresses', $entity_type);
    $field_storage_definition->setCardinality($field_storage_definition::CARDINALITY_UNLIMITED);
    $entity_definition_update_manager->updateFieldStorageDefinition($field_storage_definition);
  }

  // Get the refreshed version of entity storages after the update.
  $my_entity2_storage = \Drupal::entityTypeManager()->getStorage('my_entity2');
  $my_entity3_storage = \Drupal::entityTypeManager()->getStorage('my_entity3');

  // Set the ip addresses values, again.
  /** @var \Drupal\my_entity2\Entity\MyEntity2Interface $entity */
  foreach ($my_entity2_storage->loadMultipleRevisions(array_keys($my_entity2_stored_ip_addresses)) as $entity) {
    $entity->set('ip_addresses', $my_entity2_stored_ip_addresses[$entity->getRevisionId()]);
    $entity->save();
  }
  // Set the ip addresses values, again.
  /** @var \Drupal\my_entity3\Entity\MyEntity3Interface $entity */
  foreach ($my_entity3_storage->loadMultipleRevisions(array_keys($my_entity3_stored_ip_addresses)) as $entity) {
    $entity->set('ip_addresses', $my_entity3_stored_ip_addresses[$entity->getRevisionId()]);
    $entity->save();
  }
}

I would like to highlight the following comments:

// This update hook must run after this update hook, otherwise
// update fails, because one of the entity storages tries to read from
// the new individual ip addresses db table when it does not exist yet.

// We must do both entity updates here otherwise one of the entity updates
// fails because the entity storage tries to read data from the individual
// ip addresses db table when it is not available yet.

(IOW, I could not update the two fields in two separate update hooks, in two different modules.)

The "entity updates fails because the entity storage tries to read data from the individual ip addresses db table" refers to the following error:

>  [notice] Update started: my_entity1_update_8001
>  [notice] Update completed: my_entity1_update_8001
>  [notice] Update started: my_entity2_update_8001
>  [error]  SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal.my_entity2_revision__ip_addresses' doesn't exist: SELECT t.*
> FROM
> {my_entity2_revision__ip_addresses} t
> WHERE (revision_id IN (:db_condition_placeholder_0, :db_condition_placeholder_1)) AND (deleted = :db_condition_placeholder_2) AND (langcode IN (:db_condition_placeholder_3, :db_condition_placeholder_4, :db_condition_placeholder_5))
> ORDER BY delta ASC; Array
> (
>     [:db_condition_placeholder_0] => 106
>     [:db_condition_placeholder_1] => 107
>     [:db_condition_placeholder_2] => 0
>     [:db_condition_placeholder_3] => en
>     [:db_condition_placeholder_4] => und
>     [:db_condition_placeholder_5] => zxx
> )
>
>  [error]  Update failed: my_entity2_update_8001
[error]  Update aborted by: my_entity2_update_8001
[error]  Finished performing updates. 

This error was trigged by the code part that tried to clear the existing field values by saving NULL as values.

In my understanding, this happened because one entity update reset the entity type manager's internal (static) cache and the next update using the new field definitions (cardinality N instead of 1) that has not been applied yet on the database level.

And the story was not closed by figuring out the "correct order" of update hooks, because these updates still failed on our test server with the same database exception when we did a deploy. Why? Because just like other companies , we run the following deploy sequence:

$ composer install --no-dev
$ drush cr
$ drush updb -y
$ drush cim -y

Note the initial drush cr which also clears all service definitions, including the entity type manager's definition cache and you know the rest...

Questions

• Is there a problem with these update hooks? (IOW, am I doing something wrong)
• Is there an actual problem with the entity update system?
• Is there a problem with the sequence of these deploy commands? - I guess no, because without drush cr the deploy fails if you move/rename service classes, etc.
• Does anybody else have/had similar issues?