Helper issue for #1498720: [meta] Make the entity storage system handle changes in the entity and field schema definitions to avoid spamming it with patches to be tested. I hope I didn't break too much stuff.

CommentFileSizeAuthor
#98 et-entity_schema_handling-2298525-97.patch151.37 KBplach
#94 et-entity_schema_handling-2298525-94.patch217.06 KBplach
#94 et-entity_schema_handling-2298525-94.interdiff.txt3.51 KBplach
#90 et-entity_schema_handling-2298525-90.patch217.7 KBplach
#87 interdiff.txt2.6 KBeffulgentsia
#87 et-entity_schema_handling-2298525-87.patch216.56 KBeffulgentsia
#85 interdiff.txt8.85 KBeffulgentsia
#85 et-entity_schema_handling-2298525-85.patch216.23 KBeffulgentsia
#83 interdiff.txt32.45 KBeffulgentsia
#83 et-entity_schema_handling-2298525-83.patch214.94 KBeffulgentsia
#81 interdiff.txt3.71 KBeffulgentsia
#81 et-entity_schema_handling-2298525-81.patch200.62 KBeffulgentsia
#77 et-entity_schema_handling-last-step-do-not-test.patch150.13 KBeffulgentsia
#77 interdiff.txt127.85 KBeffulgentsia
#77 et-entity_schema_handling-2298525-77.patch203.97 KBeffulgentsia
#75 et-entity_schema_handling-2298525-75.patch202.79 KBeffulgentsia
#72 et-entity_schema_handling-2298525-72.patch203.41 KBplach
#72 et-entity_schema_handling-2298525-72.interdiff.txt2.04 KBplach
#71 et-entity_schema_handling-2298525-71.patch203.07 KBplach
#69 et-entity_schema_handling-2298525-69.patch202.48 KBplach
#69 et-entity_schema_handling-2298525-69.interdiff.txt14.33 KBplach
#67 et-entity_schema_handling-2298525-67.patch203.88 KBplach
#67 et-entity_schema_handling-2298525-67.interdiff.txt963 bytesplach
#64 et-entity_schema_handling-2298525-64.patch232.27 KBplach
#64 et-entity_schema_handling-2298525-64.interdiff.txt3.71 KBplach
#63 entity-schema_handling-remainder-full.patch234.31 KBeffulgentsia
#63 entity-schema_handling-remainder-do-not-test.patch155.07 KBeffulgentsia
#60 entity-schema_manager-full.patch100.88 KBeffulgentsia
#59 et-entity_schema_handling-2298525-59.patch229.24 KBplach
#59 et-entity_schema_handling-2298525-59.interdiff.txt4.94 KBplach
#57 et-entity_schema_handling-2298525-58.patch231.89 KBplach
#56 et-entity_schema_handling-2298525-56.patch263.77 KBplach
#54 2298525.54.patch136.45 KBeffulgentsia
#53 et-entity_schema_handling-2298525-53.patch313.48 KBplach
#53 et-entity_schema_handling-2298525-53.interdiff.txt17.14 KBplach
#51 et-entity_schema_handling-2298525-51.patch314.95 KBplach
#49 et-entity_schema_handling-2298525-49.patch319.06 KBplach
#49 et-entity_schema_handling-2298525-49.interdiff.txt19.56 KBplach
#48 et-entity_schema_handling-2298525-48.patch313.9 KBplach
#46 et-entity_schema_handling-2298525-46.patch313.1 KBplach
#45 et-entity_schema_handling-2298525-45.patch310.73 KBplach
#43 et-entity_schema_handling-2298525-43.patch264.56 KBplach
#41 et-entity_schema_handling-2298525-41.patch264.44 KBplach
#39 et-entity_schema_handling-2298525-39.patch302.94 KBplach
#39 et-entity_schema_handling-2298525-39.interdiff.txt9.09 KBplach
#37 et-entity_schema_handling-2298525-37.patch298.27 KBplach
#37 et-entity_schema_handling-2298525-37.interdiff.txt699 bytesplach
#35 et-entity_schema_handling-2298525-35.patch298.26 KBplach
#34 et-entity_schema_handling-2298525-34.patch248.26 KBplach
#34 et-entity_schema_handling-2298525-34.interdiff.txt5.33 KBplach
#33 et-entity_schema_handling-2298525-33.patch252.39 KBplach
#33 et-entity_schema_handling-2298525-33.interdiff.txt13.15 KBplach
#32 et-entity_schema_handling-2298525-31.patch245.09 KBplach
#32 et-entity_schema_handling-2298525-31.interdiff.txt9.24 KBplach
#31 et-entity_schema_handling-2298525-31.patch.patch245.09 KBplach
#31 et-entity_schema_handling-2298525-31.patch.interdiff.txt9.24 KBplach
#29 et-entity_schema_handling-2298525-29.patch245.1 KBplach
#27 25-27-interdiff.txt12.87 KBalexpott
#27 2298525-plach.27.patch247.27 KBalexpott
#25 et-entity_schema_handling-2298525-25.patch247.13 KBplach
#23 et-entity_schema_handling-2298525-23.patch243.32 KBplach
#22 et-entity_schema_handling-2298525-22.patch0 bytesplach
#21 et-entity_schema_handling-2298525-21.patch0 bytesplach
#19 et-entity_schema_handling-2298525-18.patch278.92 KBplach
#16 et-entity_schema_handling-2298525-16.patch273.03 KBplach
#14 et-entity_schema_handling-2298525-14.patch268.98 KBplach
#12 et-entity_schema_handling-2298525-12.patch265.47 KBplach
#10 et-entity_schema_handling-2298525-10.patch265.49 KBplach
#7 et-entity_schema_handling-2298525-7.patch265.5 KBplach
#7 et-entity_schema_handling-2298525-7.interdiff.txt7.06 KBplach
#6 et-entity_schema_handling-2298525-6.patch260.46 KBplach
#4 et-entity_schema_handling-2298525-4.patch256.93 KBplach
#2 et-entity_schema_handling-2298525-1.patch251.9 KBplach
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

plach’s picture

plach’s picture

Status: Active » Needs review
FileSize
251.9 KB

Status: Needs review » Needs work

The last submitted patch, 2: et-entity_schema_handling-2298525-1.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
256.93 KB

Status: Needs review » Needs work

The last submitted patch, 4: et-entity_schema_handling-2298525-4.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
260.46 KB

This merges @alexpott's work from the parent issues. I hope I got it right as I had a lot of conflicts...
(unit tests failures are expected, I need to adjust those tests to match the latest changes)

plach’s picture

The last submitted patch, 6: et-entity_schema_handling-2298525-6.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 7: et-entity_schema_handling-2298525-7.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
265.49 KB

Just a reroll to see whether failures are the same

Status: Needs review » Needs work

The last submitted patch, 10: et-entity_schema_handling-2298525-10.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
265.47 KB

Status: Needs review » Needs work

The last submitted patch, 12: et-entity_schema_handling-2298525-12.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
268.98 KB

Status: Needs review » Needs work

The last submitted patch, 14: et-entity_schema_handling-2298525-14.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
273.03 KB

Status: Needs review » Needs work

The last submitted patch, 16: et-entity_schema_handling-2298525-16.patch, failed testing.

plach’s picture

Issue summary: View changes
Status: Needs work » Needs review
plach’s picture

now with patch

Status: Needs review » Needs work

The last submitted patch, 19: et-entity_schema_handling-2298525-18.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
0 bytes

This should fix the PHPUnit failures we used to have above, now I just need to fix all the new failures :P

plach’s picture

plach’s picture

I definitely need some sleep...

Status: Needs review » Needs work

The last submitted patch, 23: et-entity_schema_handling-2298525-23.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
247.13 KB

This should fix most failures. It incorporates #2279395-54: The PostgreSQL backend does not handle NULL values for serial fields gracefully as this now depends on it.

Status: Needs review » Needs work

The last submitted patch, 25: et-entity_schema_handling-2298525-25.patch, failed testing.

alexpott’s picture

Status: Needs work » Needs review
FileSize
247.27 KB
12.87 KB

Pushed to plach's sandbox - branch is called: 2298525-alexpott

Status: Needs review » Needs work

The last submitted patch, 27: 2298525-plach.27.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
245.1 KB

Rerolled

Status: Needs review » Needs work

The last submitted patch, 29: et-entity_schema_handling-2298525-29.patch, failed testing.

plach’s picture

This should be green again, finally
(pushed to the wip branch, I'm going to go back to a serious branch asap :)

plach’s picture

plach’s picture

And this should restore some previously-removed test coverage.

plach’s picture

plach’s picture

First attempt to see how part2 behaves...

Status: Needs review » Needs work

The last submitted patch, 35: et-entity_schema_handling-2298525-35.patch, failed testing.

plach’s picture

Apparently a tiny fatal in a PHP unit test can break all the test result reporting...

plach’s picture

Status: Needs work » Needs review
plach’s picture

Status: Needs review » Needs work

The last submitted patch, 39: et-entity_schema_handling-2298525-39.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
264.44 KB

Status: Needs review » Needs work

The last submitted patch, 41: et-entity_schema_handling-2298525-41.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
264.56 KB

Status: Needs review » Needs work

The last submitted patch, 43: et-entity_schema_handling-2298525-43.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
310.73 KB
plach’s picture

Status: Needs review » Needs work

The last submitted patch, 46: et-entity_schema_handling-2298525-46.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
313.9 KB
plach’s picture

Status: Needs review » Needs work

The last submitted patch, 49: et-entity_schema_handling-2298525-49.patch, failed testing.

plach’s picture

plach’s picture

Status: Needs work » Needs review
effulgentsia’s picture

FileSize
136.45 KB

Bot check for the next sub-issue I plan on opening.

Status: Needs review » Needs work

The last submitted patch, 54: 2298525.54.patch, failed testing.

plach’s picture

plach’s picture

Status: Needs review » Needs work

The last submitted patch, 57: et-entity_schema_handling-2298525-58.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
4.94 KB
229.24 KB

This should fix test failures.

effulgentsia’s picture

effulgentsia’s picture

Thanks for the reroll in #57. Rerolling #59 on top of #60 will also be a bit tedious, but I'll work on that, and will post it here once I'm done.

plach’s picture

I needed #59 to keep working on one of the last bullets on my todo list: blocking module uninstallation when they provide field storage definitions for which data exists. Patch coming soon :)

effulgentsia’s picture

This is a reroll of #59 on top of #60. The "-do-not-test" one doesn't include any of #60. The "-full" one includes everything, but resolves the various conflicts introduced by #60.

Re #62, great! If your new patch is based on #59, then when you post yours, please post an interdiff relative to #59, and hopefully that interdiff won't be too hard to apply to these patches here.

plach’s picture

And with this we should be "done" with my todo list, at least here. The other two bullets were adding test coverage for the entity schema manager, which can happen in its own issue, and temporarily blocking entity schema layout changes, which is not fully supported yet by storage classes, and requires just a throw somewhere. We should do this only before committing the very final version of the last patch so we can write proper test coverage meanwhile.

Status: Needs review » Needs work

The last submitted patch, 64: et-entity_schema_handling-2298525-64.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
963 bytes
203.88 KB

Another fun reroll + some test fixes

Status: Needs review » Needs work

The last submitted patch, 67: et-entity_schema_handling-2298525-67.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
14.33 KB
202.48 KB

This one should be better

Status: Needs review » Needs work

The last submitted patch, 69: et-entity_schema_handling-2298525-69.patch, failed testing.

plach’s picture

Status: Needs work » Needs review
FileSize
203.07 KB

Just a plain reroll. Working on test failures now.

plach’s picture

The last submitted patch, 71: et-entity_schema_handling-2298525-71.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 72: et-entity_schema_handling-2298525-72.patch, failed testing.

effulgentsia’s picture

Status: Needs work » Needs review
FileSize
202.79 KB

Just a reroll of #72.

Status: Needs review » Needs work

The last submitted patch, 75: et-entity_schema_handling-2298525-75.patch, failed testing.

effulgentsia’s picture

Status: Needs work » Needs review
FileSize
203.97 KB
127.85 KB
150.13 KB

Merged in changes from the history of #2333113: Add an EntityDefinitionUpdateManager so that entity handlers can respond (e.g., by updating db schema) to code updates in a controlled way (e.g., from update.php). Also reordered some functions in SqlContentEntityStorageSchema to match their order in HEAD to make the diff easier to review.

The "-do-not-test" patch is rebased on top of #2333113-58: Add an EntityDefinitionUpdateManager so that entity handlers can respond (e.g., by updating db schema) to code updates in a controlled way (e.g., from update.php), so is the last step once that lands.

Status: Needs review » Needs work

The last submitted patch, 77: et-entity_schema_handling-2298525-77.patch, failed testing.

tim.plunkett’s picture

+++ b/core/lib/Drupal/Core/Entity/EntityManagerInterface.php
@@ -8,11 +8,12 @@
+interface EntityManagerInterface extends PluginManagerInterface, EntityTypeListenerInterface, FieldStorageDefinitionListenerInterface {

This really doesn't seem right to me. Why does EntityManagerInterface know about FieldStorageDefinition anything?

Can't we please create a new class and move all of this stuff out? I don't know at what point EntityManager became the dumping ground for all of this code that only pertains to half of entity types, but this is getting out of control.

EDIT: Ugh, just realized I'm posting to a test-only issue. I'll track down the real one tomorrow...

effulgentsia’s picture

Ugh, just realized I'm posting to a test-only issue. I'll track down the real one tomorrow...

The real issue with that change is #2333113: Add an EntityDefinitionUpdateManager so that entity handlers can respond (e.g., by updating db schema) to code updates in a controlled way (e.g., from update.php). However, realize that EntityManagerInterface already has a ton of methods that deal with field definitions. So if you want to decouple everything field-related from EntityManager, please open a new issue for that rather than holding up that one. However, I'll point out that isFieldable() is a method of EntityTypeInterface, so I don't think it's entirely unreasonable for EntityManager to deal with field definitions in addition to entity type definitions, but if there's a good way to partition EntityManager into smaller parts, I'm ok with that too.

effulgentsia’s picture

Status: Needs work » Needs review
FileSize
200.62 KB
3.71 KB

Removing the system_system_info_alter() change as out of scope for this issue. Seems like can be a separate issue and post-beta. Elsewhere we decided to not add hasData() to entity storage handlers, so if we want to change our mind on that, let's do it in a dedicated issue.

Also fixed a stray merge conflict.

plach’s picture

-1 on the last interdiff: blocking module uninstallation was planned with @alexpott in the last hard problems discussion about this topic we had in Austin. This was meant to prevent modules from breaking while we unify the field purging code.

Status: Needs review » Needs work

The last submitted patch, 83: et-entity_schema_handling-2298525-83.patch, failed testing.

effulgentsia’s picture

Status: Needs work » Needs review
FileSize
216.23 KB
8.85 KB

Merged changes from #2333113: Add an EntityDefinitionUpdateManager so that entity handlers can respond (e.g., by updating db schema) to code updates in a controlled way (e.g., from update.php), which is now RTBC. And fixed the cause of #83's exceptions (system.module's hunk in the interdiff).

Status: Needs review » Needs work

The last submitted patch, 85: et-entity_schema_handling-2298525-85.patch, failed testing.

effulgentsia’s picture

Status: Needs work » Needs review
FileSize
216.56 KB
2.6 KB

Status: Needs review » Needs work

The last submitted patch, 87: et-entity_schema_handling-2298525-87.patch, failed testing.

plach’s picture

Thanks for the rerolls, I will work on those failures later today if no one beats me to it.

plach’s picture

Status: Needs work » Needs review
FileSize
217.7 KB

No luck here, trying some bot debugging

Status: Needs review » Needs work

The last submitted patch, 90: et-entity_schema_handling-2298525-90.patch, failed testing.

effulgentsia’s picture

-1 on the last interdiff: blocking module uninstallation was planned with @alexpott in the last hard problems discussion about this topic we had in Austin. This was meant to prevent modules from breaking while we unify the field purging code.

Makes sense, but seems a different scope than #2337927: SqlContentEntityStorage::onFieldStorageDefinition(Create|Update|Delete)() is broken for base fields, so I opened #2338873: Modules providing non-configurable field storage definitions can be uninstalled, leaving orphaned unpurged data. Maybe having it isolated will also help with troubleshooting those failures?

plach’s picture

I think I found a way to make the test pass, I will post a patch soon.

plach’s picture

Status: Needs work » Needs review
FileSize
3.51 KB
217.06 KB

Ok, this should fix the install/uninstall test failures.

Status: Needs review » Needs work

The last submitted patch, 94: et-entity_schema_handling-2298525-94.patch, failed testing.

plach’s picture

Assigned: Unassigned » plach
effulgentsia’s picture

Status: Needs work » Needs review
FileSize
151.29 KB

Heh, I just happened to do this before seeing your comment, so here's my reroll. Curious if yours ends up being identical or not.

plach’s picture

Assigned: plach » Unassigned
FileSize
151.37 KB

Not sure, mine includes #94...
(for the bot)

plach’s picture

+++ b/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorageSchema.php
@@ -76,14 +101,25 @@ public function __construct(EntityManagerInterface $entity_manager, ContentEntit
-      $entity_type->getKeys() != $original->getKeys() ||

The only difference is here: mine is keeping this line.

The last submitted patch, 97: et-entity_schema_handling-2298525-97.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 98: et-entity_schema_handling-2298525-97.patch, failed testing.

plach’s picture

Status: Needs work » Fixed

The parent issue has been fixed :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.