Problem/Motivation

This was discovered in #2841291: Fix NOT NULL handling in the entity storage and 'primary key' changes when updating the storage definition of an identifier field.

The stored field schema does not contain:

  • primary keys, if the field is part of the primary key
  • unique keys, unless they are declared by the field type itself (the UUID field does this, for example)
  • indexes, unless they are declared by the field type or the field itself (for example composite indexes are currently missing)

This means that the field schema is not sufficient when updating field storage definitions. We instead need to fetch the entity schema which is less than ideal and potentially problematic due differing states of the entity type and field storage definitions.

Proposed resolution

Let's add them!

CommentFileSizeAuthor
#47 interdiff_46-47.txt4.39 KBvsujeetkumar
#47 2929120-47.patch40 KBvsujeetkumar
#46 2929120-46.patch39.7 KBvsujeetkumar
#42 2929120-32.patch40.88 KBajaypratapsingh
#32 2929120-32.patch40.88 KBtstoeckler
#32 2929120-31-32-interdiff.txt12.6 KBtstoeckler
#31 2929120-31.patch29.42 KBtstoeckler
#31 2929120-29-31-interdiff.txt5.89 KBtstoeckler
#29 2929120-29.patch35.96 KBtstoeckler
#29 2929120-27-29-interdiff.txt3.28 KBtstoeckler
#27 2929120-26.patch32.68 KBtstoeckler
#27 2929120-25-26-interdiff.txt2.41 KBtstoeckler
#25 2929120-25.patch32.5 KBtstoeckler
#21 2929120-21--for-robots.patch46.8 KBtstoeckler
#21 2929120-21--for-humans.txt36.51 KBtstoeckler
#21 2929120-20-21-interdiff.txt1.8 KBtstoeckler
#20 2929120-20--for-robots.patch48.28 KBtstoeckler
#20 2929120-20--for-humans.txt35.04 KBtstoeckler
#20 2929120-19-20-interdiff.txt1.57 KBtstoeckler
#19 2929120-19--for-robots.patch48.6 KBtstoeckler
#19 2929120-19--for-humans.txt34.34 KBtstoeckler
#19 2929120-17-19-interdiff.txt5.34 KBtstoeckler
#2 2929120-2-interdiff-2928906.txt5.68 KBtstoeckler
#2 2929120-2.patch30.61 KBtstoeckler
#8 2929120-8--for-humans.patch31.43 KBtstoeckler
#8 2929120-8--for-robots.patch47.3 KBtstoeckler
#11 2929120-8-11-interdiff.txt3.54 KBtstoeckler
#11 2929120-11--for-humans.txt58.29 KBtstoeckler
#11 2929120-11--for-robots.patch73.02 KBtstoeckler
#14 2929120-11-14-interdiff.txt4.48 KBtstoeckler
#14 2929120-14--for-humans.txt32.49 KBtstoeckler
#14 2929120-14--for-robots.txt47.12 KBtstoeckler
#15 2929120-15--for-robots.patch47.12 KBtstoeckler
#17 2929120-15-17-interdiff.txt4.81 KBtstoeckler
#17 2929120-17--for-humans.txt32.9 KBtstoeckler
#17 2929120-17--for-robots.patch47.53 KBtstoeckler
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

tstoeckler created an issue. See original summary.

tstoeckler’s picture

Status: Active » Needs review
FileSize
5.68 KB
30.61 KB

Here we go. Extracted this from #2928906: The field schema incorrectly stores serial fields as int. Interdiff is relative to #4 over there and should hopefully fix the tests. Could be that I messed something up in the split though...

Status: Needs review » Needs work

The last submitted patch, 2: 2929120-2.patch, failed testing. View results

mayurjadhav’s picture

Issue tags: +DrupalMumbaiCodeSprint

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

tstoeckler’s picture

Title: The field schema is missing field-level primary keys, unique keys and indexes » [PP-1] The field schema is missing field-level primary keys, unique keys and indexes
Status: Needs work » Postponed

Looked at the test failures, but similarly to #2928906: The field schema incorrectly stores serial fields as int this requires #2938951: The primary key is not correctly re-created when updating the storage definition of an identifier field because we are updating the field storage definitions as part of the update, so we need to recreate the primary keys.

tstoeckler’s picture

tstoeckler’s picture

Status: Postponed » Needs review
FileSize
31.43 KB
47.3 KB

OK, so here's a re-roll on top of #33 from #2938951: The primary key is not correctly re-created when updating the storage definition of an identifier field.

Once we get a green here, we can start tackling those todos from that issue that point here to prove that they are warranted.

tstoeckler’s picture

+++ b/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorageSchema.php
@@ -1384,8 +1369,8 @@ protected function createSharedTableSchema(FieldStorageDefinitionInterface $stor
+              if (!empty($schema[$table_name]['primary key']) && in_array($name, $entity_schema[$table_name]['primary key'], TRUE)) {

Ahh borked the re-roll here. This should be $schema in both cases, and we should remove $entity_schema altogether here. Should in theory not affect green-ness, though. We'll see...

Status: Needs review » Needs work

The last submitted patch, 8: 2929120-8--for-robots.patch, failed testing. View results

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
3.54 KB
58.29 KB
73.02 KB

This fixes #9 and one other thing I borked up in the re-roll + bumps the update functions to 8.6

tstoeckler’s picture

index 0000000000..5a86d59e2c
--- /dev/null

--- /dev/null
+++ b/2929120-2.patch

index 0000000000..740c90403a
--- /dev/null

--- /dev/null
+++ b/test.php

Sorry, that's stupid. Will leave that out of the next one.

Status: Needs review » Needs work

The last submitted patch, 11: 2929120-11--for-robots.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
4.48 KB
32.49 KB
47.12 KB

This should fix at least some of the fails.

Of course, I ran into #2928906: The field schema incorrectly stores serial fields as int here. Adding more duct-tape for now in order to keep those separate.

tstoeckler’s picture

FileSize
47.12 KB

D'uh, wrong extension. Here you go, little bot.

Status: Needs review » Needs work

The last submitted patch, 15: 2929120-15--for-robots.patch, failed testing. View results

tstoeckler’s picture

The last submitted patch, 17: 2929120-17--for-robots.patch, failed testing. View results

tstoeckler’s picture

So this revealed a bug, yay ;-): When we change the serial field to an int during an update of a field definition we never actually change it back. So we should do that. ...except not for user entities of course, as those do not use serial field at all. Hah, this is so much fun... ;-)

Unless I broke anything else - which is entirely possible - this should be green. Will then attempt to remove some of the hacks introduced by #2938951: The primary key is not correctly re-created when updating the storage definition of an identifier field.

tstoeckler’s picture

tstoeckler’s picture

Now testing my theory that the ugly workaround in SqlContentEntityStorageSchema::processBaseTable() is only needed due to this.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Gogowitsch’s picture

Issue summary: View changes
jibran’s picture

Title: [PP-1] The field schema is missing field-level primary keys, unique keys and indexes » The field schema is missing field-level primary keys, unique keys and indexes
Status: Needs review » Needs work
tstoeckler’s picture

Here's a re-roll. This still contains some cruft which can be removed, but let's see if this is green first.

Status: Needs review » Needs work

The last submitted patch, 25: 2929120-25.patch, failed testing. View results

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
2.41 KB
32.68 KB

So the problem was that when we were merging field schemas, we were passing a schema array with multiple indexes to addIndex(), but only with the fields that are needed for one of them. So we need to make sure that only a single index is part of the $index_schema array.

Hope this will be green, will try to remove the primary key stuff if so, as I'm pretty sure that it's no longer needed.

Status: Needs review » Needs work

The last submitted patch, 27: 2929120-26.patch, failed testing. View results

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
3.28 KB
35.96 KB

Maybe this time?

jibran’s picture

Title: The field schema is missing field-level primary keys, unique keys and indexes » The stored field schema does not contain field-level primary keys, unique keys and indexes

Updated the title.

tstoeckler’s picture

FileSize
5.89 KB
29.42 KB

OK, this time with less cruft. The interdiff does not show the (re-)removal of (the previously introduced here) AssertSchemaTrait as that is no longer needed because I accidentally removed it with PhpStorm 😡. But the patch should be significantly smaller now.

tstoeckler’s picture

FileSize
12.6 KB
40.88 KB

Alright, this now covers all existing entity types. I guess we want to add a change record, but in terms of the patch this should be RTBC-able/reviewable now, at least from my perspective.

Berdir’s picture

I didn't look at the changes in-depth yet, but the fact that entity types like comment and node need their own custom update functions and EntityStorageSchema changes seems very problematic to me. That seems like an non-BC API change and even if we'd decide to allow/require that, it's really quite hard to deal with this in contributed modules.

Even if you add an update function, you need to ensure that it runs at the right time, meaning, it mustr only run after the 8.7 update (assuming it lands there), so modules need to add a version dependency as well as an update hook dependency on that system update function to ensure the necessary change actually exists, otherwise the updateFieldStorageDefinition() call wouldn't actually do anything.

Maybe we can somehow detect these kind of changes and call the update method automatically on all field storage definitions that need the update? But that is also really tricky, in general changes that are active automatically on an entity type and don't require any kind of configuration/definition change. Imagine that in my_awesome_module, I'm doing a change that requires a custom update function because I'm altering the schema and actually have data. If I'm updating to 8.7 and updating that module at the same time, hen the generic update would jump in and run and obviously fail because there is data. Making it extremely hard for my module to be update-able both before/after core.

Status: Needs review » Needs work

The last submitted patch, 32: 2929120-32.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

larowlan’s picture

Do we still want to do this folks?

ajaypratapsingh’s picture

tested on drupal 9.4.x

ajaypratapsingh’s picture

Status: Needs work » Needs review

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

immaculatexavier’s picture

Status: Needs review » Needs work

#42 Patch Failed to Apply, changing the status to Need Work.

vsujeetkumar’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
39.7 KB

Re-roll patch created for 9.5.x, Please have a look.

vsujeetkumar’s picture

Fixed custom command fail issues.

Status: Needs review » Needs work

The last submitted patch, 47: 2929120-47.patch, failed testing. View results

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.