Problem/Motivation

Dynamic entity reference, unlike core entity reference currently only supports referring to entity types with integer entity IDs. There are two problems with this:

  1. Config entities are out
  2. Core actually supports content entities with string IDs (it's tested with the entity_test_string_id entity type).

Proposed resolution

This is tricky. If we were to store DER.target_id as a string then joining to any "normal" entity on PostgreSQL would throw an error. On MySQL it doesn't but it'd be a serious performance hit (I don't actually have a problem with since I work for a scalability agency and yay, job security! but I do not think the community would be happy about it). Also, sorting would break. Not sure why would anyone want to sort on an entity ID but I thought I will mention that.

So storing an integer is not enough and storing a string is broken in various sometimes subtle ways.

So the idea is to store both. For Entity API, define target_id as string. In the SQL database, since this is a SQL database problem, keep an integer column with the target_id casted to int and use a trigger to maintain it. Views and entity query needs to know about the integer column since they talk directly to SQL but nothing else should. In particular, the field item class should not know about it, it's quite complex already. Another possible solution offered by core is a custom storage which is probably more code than this and also doesn't allow a DER field in the base table. We don't want that.

Drupal never used triggers before and of course the syntax varies between databases but backend_overridable was made for this.

Remaining tasks

  1. Add a views field alias (if possible, I am not sure) to Views so target_id_int never shows up in the Views result rows. Probably won't allow reverting all tests since they directly test a mostly internal structure.
  2. Still in Views, support string ID'd entity types. With the current patch Views only supports integer IDs because it unconditionally switches to the integer column.
  3. Add even more explicit tests. derUpdateTest already checks after update but a test for referring string IDs in shared tables and field configs created (not updated) should be tested too.
  4. Write a core patch so that the field type can rewrite the field column as necessary when entity query runs.

User interface changes

None yet. In a followup we will need to figure out how to refer to config entities. That is always fun. Especially because not all has labels to boot.

API changes

Surprisingly none.

Data model changes

One new column per DER field is added into the database and a trigger per table containing a DER field is added.


Original report:

Cores Entity Reference will set its 'target_id' column schema to either integer or string. By determining the target entity type' primary ID data type.

Dynamic Entity Reference only supports entity types which have a integer for its ID data type.

DERItems 'target_id' schema data type should default to varchar, therefore supporting all entity types. With an opt-in setting for performance, setting the target_id schema to integer. When the field storage is being created in Field UI, show a list of entity types the field will/will not be able to reference depending on the choice.

Comments

dpi created an issue. See original summary.

dpi’s picture

Issue summary: View changes
jibran’s picture

I'm giving it a lot of thought I need to collect some more info about it and http://larowlan.github.io/ds.core-til/#/6/3 so please bear with me. Thanks.

jibran’s picture

Category: Feature request » Task

The idea is to introduce the new DER field type for config entity see #2365331-16: Update DynamicEntityReferenceItem according to EntityReferenceItem for more details. When I converted target_type to integer i thought all the fieldable entities have integer ids. After some research I realized I was wrong.

After discussing with @larowlan we decided to do following.

1) Show all the entities config/content on field storage form. Just like ER field in core.
2) If all traget_types have integer id. We create a schema with integer column.
3) If any one of the traget_type has a string id. We create a schema with varchar column.

If the field has data then it is not a problem because we can't change target types and we don't have to change schema.
If the field has no data and the schema was int before and after re-saving the field it changed to varchar or vise versa then Drupal will update the table automatically as it is done for ER field.

dpi’s picture

Would I be wrong if I said content entities can have string ID's? Im not sure making a config/content distinction is correct.

jibran’s picture

Content entities can have string ID's.

dpi’s picture

The idea is to introduce the new DER field type for config entity

9433787:

DER field can only support either content entities or config entities per field

Im not sure what these are supposed to mean then. The field should either support numeric or both. Doesnt matter whether its config or content?

jibran’s picture

That comment doesn't matter now we have new plan, turn target_id column to string by default and if all the target entity types id has integer schema then we'll change the column type of target_id form string to integer automatically. I hope this makes it clearer.

dpi’s picture

Turns out this is a big issue for Postgres. If I use a string in an EFQ condition for DER target entity id then Postgres will throw an exception. RNG/Courier issue: Postgres tests failing due to string usage in integer field.

dpi’s picture

Making a personal note here.

If exclude_entity_types is TRUE or entity_type_ids is omitted then it means target IDS are uncertain and should default to string.

chx’s picture

Edit: deleted.

jibran’s picture

jibran’s picture

I had a chat with @larowlan today and we both think that it'd be good to add a new field type for sting entity ID's so that we don't end up in the mixed situation. ER is doing the same thing as well.

Other then flag #2678756: Allow config entities to be flagged I don't see any use case of mixed entity IDs. flag can have two DER basefields one for int entity id and other is for sting entity id and flagging/unflagging action or preSave can populate the respective DER field base on flagged entity id.

Given that @berdir is already involved. I'll ping @yched and @fago about it as well.

chx’s picture

Edit: deleted.

chx’s picture

Edit: OK I will bite.

The solution is for Drupal to grow up and use databases for real and not like MySQL 4.1 or so.

On the application side, target_id should be string so that it can encompass everything.

The SQL database should maintain a cast-to-unsigned column and index it. MySQL 5.7 and Oracle, virtual column, PostgreSQL, SQLite and MySQL <5.7 triggers. The only purpose for this column is faster querying, it shouldn't be visible to (most of the app).

The app only needs to be aware of this column in the two abstraction APIs we put on top of SQL: Views and entity query. For Views, this is really simple, doesn't even need a core patch, just the views data needs to look at whether the referred entity type uses strings or ints as IDs, for entity query we would need to patch up the eldritch horror that is Tables::addField, it can't get much worse.

So

  1. Add a backend overrideable service to DER which adds this trigger/virtual column. Core patch is not needed, just use db_query. In the MySQL version split on database server version. Copy the necessary two LoC from Drupal\Core\Database\Driver\mysql\Install\Tasks::checkEngineVersion to figure out the MySQL server version. Figuring out the column name is a bit of work but I do not think it's too hard. I believe it's event based and EntityTypeEventSubscriberTrait is where we need to listen.
  2. dynamic_entity_reference_field_views_data should switch between these columns as necessary.
  3. Finally, this is the only core patch necessary: field type plugins need a new method that Tables::addField can call to rewrite the database column as necessary.

Altogether, hopefully all this is not a lot of code. The user does not need to be aware of anything funny going on. What's not to love?

chx’s picture

StatusFileSize
new7.92 KB

This is just the boilerplate. It doesn't do anything and indeed it'll break because it switches to _int in views unconditionally and it's not created yet. But it shows the way.

jibran’s picture

This apporoch is quiet interesting IMO. Let's see how this shape up. Nice start @chx. Here are couple of minor point.

  1. +++ b/src/Plugin/Field/FieldType/DynamicEntityReferenceItem.php
    @@ -73,7 +73,7 @@ class DynamicEntityReferenceItem extends EntityReferenceItem {
    -    $properties['target_id'] = DataReferenceTargetDefinition::create('integer')
    +    $properties['target_id'] = DataReferenceTargetDefinition::create('string')
    

    We need an upgrade path for and upgrade path test for this.

  2. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,23 @@
    +/**
    + * @file
    + * Contains \Drupal\dynamic_entity_reference\Storage\CreateColumn.
    + */
    

    Not needed.

chx’s picture

Status: Active » Needs review
StatusFileSize
new10.91 KB

With the attached patch, DynamicEntityReferenceItemTest runs...

Status: Needs review » Needs work

The last submitted patch, 18: 2555027_18.patch, failed testing.

The last submitted patch, 18: 2555027_18.patch, failed testing.

The last submitted patch, 18: 2555027_18.patch, failed testing.

jibran’s picture

You can copy EntityReferenceItemTest::testContentEntityReferenceItemWithStringId to DynamicEntityReferenceItemTest to check your changes as well.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new11.02 KB

Hrm, tests pass here :(

The last submitted patch, 18: 2555027_18.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 23: 2555027_23.patch, failed testing.

The last submitted patch, 23: 2555027_23.patch, failed testing.

The last submitted patch, 23: 2555027_23.patch, failed testing.

jibran’s picture

Here are some review points.

  1. +++ b/dynamic_entity_reference.services.yml
    @@ -2,3 +2,19 @@ services:
    +  dynamic_entity_reference.entity_type_subscriber:
    ...
    +  dynamic_entity_reference.storage.create_column:
    ...
    +  mysql.dynamic_entity_reference.storage.create_column:
    ...
    +  sqlite.dynamic_entity_reference.storage.create_column:
    

    Some doc love here please.

  2. +++ b/dynamic_entity_reference.views.inc
    @@ -50,7 +50,7 @@ function dynamic_entity_reference_field_views_data(FieldStorageConfigInterface $
    -        'relationship field' => $field_name . '_target_id',
    +        'relationship field' => $field_name . '_target_id_int',
    

    We'd need a follow up to update this for string target ids?

  3. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    + * Hands off field storage events to the integer column handler.
    

    Maybe explain it a little.

  4. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    +   * @var \Drupal\Core\Entity\EntityTypeManagerInterface
    ...
    +   * @var \Drupal\dynamic_entity_reference\Storage\IntColumnHandler
    ...
    +   * @var \Drupal\Core\Entity\EntityFieldManagerInterface
    

    Desc is missing.

  5. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    +   * EntityType constructor.
    

    Nope.

  6. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    +    if ($storage_definition->getType() == 'dynamic_entity_reference' && $storage instanceof SqlEntityStorageInterface) {
    

    I'm always confused, should this have to be instance or SESI or should we check the type of entity_id key?

  7. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    +      $field_name = $storage_definition->getName();
    +      $storage_definitions = $this->entityFieldManager->getFieldStorageDefinitions($entity_type_id) + [$field_name => $storage_definition];
    +      $mapping = $storage->getTableMapping($storage_definitions);
    +      $table = $mapping->getFieldTableName($field_name);
    +      $columns = $this->getColumns($storage_definitions, $mapping, $table);
    +      $column = $columns[$field_name];
    

    Some more doc love please.

  8. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    +   * @param \Drupal\Core\Field\FieldStorageDefinitionInterface[] $storage_definitions
    +   * @param \Drupal\Core\Entity\Sql\TableMappingInterface $mapping
    +   * @param string $table
    +   * @return array
    

    Incomplete docs. :)

  9. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,110 @@
    +    $filter = function (FieldStorageDefinitionInterface $definition) use ($mapping, $table) {
    +      return $definition->getType() == 'dynamic_entity_reference' && $mapping->getFieldTableName($definition->getName()) == $table;
    +    };
    +    $getter = function (FieldStorageDefinitionInterface $definition) use ($mapping) {
    +      return $mapping->getFieldColumnName($definition, 'target_id');
    +    };
    +    return array_map($getter, array_filter($storage_definitions, $filter));
    

    This is some real coding :D. More docs please.

  10. +++ b/src/Plugin/Field/FieldType/DynamicEntityReferenceItem.php
    @@ -73,7 +73,7 @@ class DynamicEntityReferenceItem extends EntityReferenceItem {
    -    $properties['target_id'] = DataReferenceTargetDefinition::create('integer')
    +    $properties['target_id'] = DataReferenceTargetDefinition::create('string')
    

    DERItem::schema also needs an update.

  11. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,29 @@
    +   * @var \Drupal\Core\Database\Connection
    

    Desc missing.

  12. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,29 @@
    +    ¶
    ...
    +    ¶
    

    Nope

  13. +++ b/src/Storage/IntColumnHandlerMySQL.php
    @@ -0,0 +1,36 @@
    +  public function onCreate($table, array $columns, $column) {
    

    Doc block missing.

  14. +++ b/src/Storage/IntColumnHandlerMySQL.php
    @@ -0,0 +1,36 @@
    +    $this->connection->query('LOCK TABLES {' . $table . '} WRITE');
    +    $schema = $this->connection->schema();
    +    $column_int = $column . '_int';
    

    Comments please.

  15. +++ b/src/Storage/IntColumnHandlerMySQL.php
    @@ -0,0 +1,36 @@
    +        'default' => 0,
    

    Do we need this?

  16. +++ b/src/Storage/IntColumnHandlerMySQL.php
    @@ -0,0 +1,36 @@
    +    $this->connection->query("DROP TRIGGER IF EXISTS {$prefixed_name}_update");
    +    $this->connection->query("DROP TRIGGER IF EXISTS {$prefixed_name}_insert");
    +    $update_trigger = "CREATE TRIGGER {$prefixed_name}_update BEFORE UPDATE ON $prefixed_name FOR EACH ROW SET";
    +    $insert_trigger = "CREATE TRIGGER {$prefixed_name}_insert BEFORE INSERT ON $prefixed_name FOR EACH ROW SET";
    +    $body = '';
    +    $delimiter = '';
    +    foreach ($columns as $current_column) {
    +      $body .= "$delimiter NEW.{$current_column}_int = CAST(NEW.$current_column AS UNSIGNED)";
    +      $delimiter = ',';
    +    }
    +
    +    $this->connection->query("$update_trigger$body");
    +    $this->connection->query("$insert_trigger$body");
    +    $this->connection->query('UNLOCK TABLES');
    

    This is gibberish to me so please add @see, @link, @code or docs to elaborate it. :)

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new40.93 KB

> We'd need a follow up to update this for string target ids?

Aye but since config entities are not stored in "regular" tables the follow up is only for "freak" content entities that have string IDs. I believe the entity system allows for that but AFAIK it is not tested anywhere in core.

> I'm always confused, should this have to be instance or SESI or should we check the type of entity_id key?

The type entity_id doesn't matter. SESI holds the getTableMapping method (and nothing else, in fact) and that's what we need.

This patch has significantly rewritten the subscriber, the code is much simpler and allows reaction to entity type create which makes the code work with hook_entity_base_field_info. When I get around to write an update hook it will also be helpful.

There is no interdiff as the interdiff is larger than the patch... it's a big change with only the trigger code remaining untouched although it has gotten a bit of defensive programming and comments.

larowlan’s picture

There's a test content entity in core that has string ids

chx’s picture

o_O well then! Hello entity_test_string_id! Since DER is already broken if you point to that, can we postpone handling that case to a followup still or should we roll it into this one? My plans for this issue were adding a query into a test or two to check the int column contains the right value; this is tested implicitly with views tests but I want to add an explicit test and also pgsql and sqlite drivers. That'll be enough for one go, what do you think?

Status: Needs review » Needs work

The last submitted patch, 29: 2555027_29.patch, failed testing.

chx’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new42.77 KB
new8.27 KB

Here we go for SQLite. I factored out the common parts into IntColumnHandler. Yes, some doxygen is missing.

Status: Needs review » Needs work

The last submitted patch, 33: 2555027_32.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
chx’s picture

Issue summary: View changes
jibran’s picture

Thanks @chx for updating the issue summary.
Let me have a look at theses so you can work on rest of them.

  1. Add a views field alias (if possible, I am not sure) to Views so target_id_int never shows up in the Views result rows. Probably won't allow reverting all tests since they directly test a mostly internal structure.
  2. Still in Views, support string ID'd entity types. With the current patch Views only supports integer IDs because it unconditionally switches to the integer column.

can we postpone handling that case to a followup still or should we roll it into this one?

I'd like to fix it here but this patch is already pretty big so let's wait for now.

Nice work on SQLite. Moar docs please.

chx’s picture

StatusFileSize
new9 KB
new46.83 KB

PostgreSQL. Yesssss.

Status: Needs review » Needs work

The last submitted patch, 38: 2555027_38.patch, failed testing.

The last submitted patch, 38: 2555027_38.patch, failed testing.

The last submitted patch, 38: 2555027_38.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new47.17 KB

Hrm, let's drop.

Status: Needs review » Needs work

The last submitted patch, 42: 2555027_40.patch, failed testing.

The last submitted patch, 42: 2555027_40.patch, failed testing.

The last submitted patch, 42: 2555027_40.patch, failed testing.

jibran’s picture

StatusFileSize
new10.5 KB

Do you think this will be enough for views fix?

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new1.88 KB
new47.39 KB

PostgreSQL is _annoying_ . I had DynamicEntityReferenceRelationshipTest passing before ; but that didn't drop things ; then I needed to DROP things and that broke PostgreSQL because even a DROP statement is elaborate and annoying. Oh well. I have shortened the trigger name while at it.

The views fix imo is a followup.

chx’s picture

Note that #2751363: Don't allow test entities to share base table fixes the fails as far as I can see. Not sure how to proceed.

jibran’s picture

StatusFileSize
new2.25 KB
new76.3 KB

Dump file.
Steps to reproduce.
Install testing profile.
Run drush scr test.php
Run php ./core/scripts/dump-database-d8-mysql.php --no-ansi > dump.php
Run gzip dump.php

chx’s picture

The PostgreSQL tests are failing with

DETAIL:  Key (proname, proargtypes, pronamespace)=(rand, , 2200) already exists.: CREATE OR REPLACE FUNCTION "rand"() RETURNS float AS

This is a core bug, when you try to install two Drupals in PostgreSQL in parallel then CREATE OR REPLACE runs in parallel and results in this error. A race condition gets triggered here.

chx’s picture

StatusFileSize
new148.73 KB

This will fail without #2751363: Don't allow test entities to share base table but here we are.

Status: Needs review » Needs work

The last submitted patch, 51: 2555027_51.patch, failed testing.

The last submitted patch, 51: 2555027_51.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
Related issues: +#275136: TinyMCE don't submit format, but only text
StatusFileSize
new149.54 KB

protected $installProfile = 'testing'; is necessary since the testing profile was used to dump.

Also, I coded around #275136: TinyMCE don't submit format, but only text. It's not pretty but it hopefully makes for a green test. Please do not add additional tests before mysql comes back green.

chx’s picture

Status: Needs review » Needs work

The last submitted patch, 54: 2555027_54.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new157.86 KB

Status: Needs review » Needs work

The last submitted patch, 57: 2555027_57.patch, failed testing.

jibran’s picture

These fails will go away after merging 8.x-1.x into the patch.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new150.19 KB
new1.02 KB
new2.19 KB

I dunno about those fails, I am rebased on origin/8.x-1.x. Here's a patch that passed locally and tests referring to a config entity because we can do that. I have also uploaded the script I use to test this patch, for it to work correctly commit the patch to branch 2550227 under modules/dynamic_entity_reference .

jibran’s picture

+++ b/src/Tests/Update/derUpdateTest.php
@@ -0,0 +1,42 @@
+    $this->assertEqual([1, 1, 1, 1, 0], db_query('SELECT field_test_mul_target_id_int FROM {entity_test_mul__field_test_mul} ORDER BY entity_id, delta')->fetchCol());
+    $this->assertEqual([1, 1, 1, 1, 'test'], db_query('SELECT field_test_mul_target_id FROM {entity_test_mul__field_test_mul} ORDER BY entity_id, delta')->fetchCol());

Oh! my.

chx’s picture

StatusFileSize
new151.69 KB
new12.64 KB

Let's see those fails.

chx’s picture

Issue summary: View changes
dawehner’s picture

+++ b/src/Storage/IntColumnHandler.php
@@ -0,0 +1,115 @@
+ */
+class IntColumnHandler implements IntColumnHandlerInterface {
...
+   */
+  protected function createBody($column_int, $column) {
+    throw new \LogicException('Not implemented');
+  }

What about making it abstract ?

In general it would be great to explain a bit of the reasoning behind the SQL trigger on the relevant classes, but of course, this is contrib, this could totally happen later.

jibran’s picture

Assigned: Unassigned » jibran

Thanks @chx for making it green. Should we create a new 8.x-2.x branch for this patch? We can release an alpha tag for this. There is an upgrade path and test for this so I think it is doable. Meanwhile we can make 8.x-1.x stable and forward port the patches as well.

+1 to #64. I think we should improve the overall docs.

I think we can move these things to followup:

I'll work on the trivial feedback.

  1. +++ b/dynamic_entity_reference.install
    @@ -0,0 +1,47 @@
    +  $kv = \Drupal::keyValue('entity.storage_schema.sql');
    

    Let's add a comment about it and let's change the name as of variable as well.

  2. +++ b/dynamic_entity_reference.install
    @@ -0,0 +1,47 @@
    +    if ($storage instanceof SqlEntityStorageInterface) {
    

    Let's add a comment here as well.

  3. +++ b/dynamic_entity_reference.install
    @@ -0,0 +1,47 @@
    +      foreach (array_intersect_key($storage_definitions, $map) as $storage_definition) {
    

    Let's add a comment here as well.

  4. +++ b/dynamic_entity_reference.install
    @@ -0,0 +1,47 @@
    +        $spec = [
    

    Let's move this out of the loop.

  5. +++ b/dynamic_entity_reference.install
    @@ -0,0 +1,47 @@
    +        $schema->changeField($table, $column, $column, $spec);
    ...
    +      $service->handleEntityType($entity_type_id);
    

    So we are converting the existing target_id column form int to string and service will add the trigger and target_id_int? Let;s add a comment about it.
    I think we can pass the field_name and field_storage_definition as well.

  6. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,137 @@
    +  public static function getSubscribedEvents() {
    

    We need KTB for all these events.

  7. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,137 @@
    +    $events[EntityTypeEvents::CREATE][] = ['onEntityType', 100];
    +    $events[EntityTypeEvents::CREATE][] = ['onEntityType', 100];
    

    Aren't these two the same lines?

  8. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,115 @@
    +   * @TODO not sure whether we want to bother with deleting.
    

    Do we need tests to be sure about it?

  9. +++ b/src/Tests/Update/derUpdateTest.php
    @@ -0,0 +1,43 @@
    +class derUpdateTest extends UpdatePathTestBase {
    

    DerUpdateTest

  10. +++ b/src/Tests/Update/derUpdateTest.php
    @@ -0,0 +1,43 @@
    +  protected $installProfile = 'testing';
    

    {@inheritdoc} is missing.

  11. +++ b/src/Tests/Update/derUpdateTest.php
    @@ -0,0 +1,43 @@
    +      __DIR__ . '/der_dump.php.gz',
    

    It should be good to document the original generating script here.

  12. +++ b/src/Tests/Update/derUpdateTest.php
    @@ -0,0 +1,43 @@
    +    $this->assertEqual([1, 1, 1, 1], db_query('SELECT field_test_mul_target_id_int FROM {entity_test_mul__field_test_mul}')->fetchCol());
    

    Let's document this.

  13. +++ b/src/Tests/Update/derUpdateTest.php
    @@ -0,0 +1,43 @@
    +    $etb = EntityTestBundle::create([
    +      'id' => 'test',
    +      'label' => 'Test label',
    +      'description' => 'My test description',
    +    ]);
    +    $etb->save();
    +    $entity = EntityTestMul::load(3);
    +    $entity->field_test_mul[] = $etb;
    +    $entity->save();
    +    $this->assertEqual([1, 1, 1, 1, 0], db_query('SELECT field_test_mul_target_id_int FROM {entity_test_mul__field_test_mul} ORDER BY entity_id, delta')->fetchCol());
    +    $this->assertEqual([1, 1, 1, 1, 'test'], db_query('SELECT field_test_mul_target_id FROM {entity_test_mul__field_test_mul} ORDER BY entity_id, delta')->fetchCol());
    

    Let's move this to follow up issue because we have to change the DER field storage form to allow config entities. Let's use entity_test_string_id instead.
    But I love this assert.

  14. +++ b/tests/modules/dynamic_entity_reference_275136/dynamic_entity_reference_275136.module
    @@ -0,0 +1,15 @@
    +function dynamic_entity_reference_275136_entity_type_alter(array &$entity_types) {
    

    so cool

jibran’s picture

StatusFileSize
new19.59 KB
new3.56 KB
new163.42 KB

RE: Trivial feedback.

  1. Fixed.
  2. Fixed.
  3. Fixed.
  4. This is well placed let's keep it that way.
  5. Moved inside the loop and added a comment.
  6. Added two passing tests for this. yay!!
  7. Converted the other line to UPDATE.
  8. Let's create an issue about it and discuss some more.
  9. Fixed.
  10. Fixed.
  11. Added a comment.
  12. Added docs and some more asserts.
  13. Used entity_test_string_id and it is causing some fails. PFA my updated script with field settings. Patch is still green on old db dump.

In this patch.

  • New DB dump using updated script.
  • Updated column lenght on @dawehner's feedback.
  • Fixed PHPCS errors.
  • Added more asserts to update test.
  • Added a test for referencing entities with string IDs.
  • Added a test for referencing entities with string and int IDs.
  • Attached interdiff only includes the code changes without DB dump.
  • I ran into this error.
    SQLSTATE[42000]: Syntax error or access violation: 1059 Identifier name 'simpletest381403entity_test__entity_reference_string_id_der_update' is too long: DROP TRIGGER IF EXISTS simpletest381403entity_test__entity_reference_string_id_der_update; Array
    (
    )
    
    core/lib/Drupal/Core/Database/Connection.php:671
    core/lib/Drupal/Core/Database/Connection.php:635
    core/lib/Drupal/Core/Database/Driver/mysql/Connection.php:81
    modules/dynamic_entity_reference/src/Storage/IntColumnHandler.php:63
    modules/dynamic_entity_reference/src/EventSubscriber/FieldStorageSubscriber.php:128
    modules/dynamic_entity_reference/src/EventSubscriber/FieldStorageSubscriber.php:85
    core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php:111
    core/lib/Drupal/Core/Field/FieldStorageDefinitionListener.php:77
    core/lib/Drupal/Core/Entity/EntityManager.php:411
    core/modules/field/src/Entity/FieldStorageConfig.php:332
    core/modules/field/src/Entity/FieldStorageConfig.php:286
    core/lib/Drupal/Core/Entity/EntityStorageBase.php:434
    core/lib/Drupal/Core/Entity/EntityStorageBase.php:389
    core/lib/Drupal/Core/Config/Entity/ConfigEntityStorage.php:259
    core/lib/Drupal/Core/Entity/Entity.php:361
    core/lib/Drupal/Core/Config/Entity/ConfigEntityBase.php:637
    modules/dynamic_entity_reference/tests/src/Kernel/DynamicEntityReferenceFieldTest.php:249
jibran’s picture

Assigned: jibran » Unassigned

Status: Needs review » Needs work

The last submitted patch, 66: support_non_numeric-2555027-66.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new1.28 KB
new160.68 KB

Thanks for the amazing amount of quick work! I have re-dumped and rerolled. Note that dumping this without https://www.drupal.org/files/issues/2555027.sh__0.txt is not feasible: as far as I can see the previous patch didn't include the fixup module. I reuploaded the script because now it points to test.php__17.txt instead of 15. There's no interdiff as there's no change (yet). The identifier issue is worrysome but easy to fix with a little hashing. Will do it next but first I want to see what the bot says.

Status: Needs review » Needs work

The last submitted patch, 69: 2555027_69.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new1.74 KB
new160.77 KB

Easy as pie. Instead of simpletest381403entity_test__entity_reference_string_id_der_update we have simpletest425511entity_test__entity_reference_string_id_208e2b58. It won't always be this nice but close. It uses the first 56 characters of the prefixed table and so in the non-test, most common case where the user has no prefix the hash won't be necessary: generateFieldTableName will produce a max 48 character name and _der_update only adds 11 which is below the max 64. It's only with simpletest (or other prefix) the hashing becomes necessary.

Did the abstract change dawenher asked for as well since I had the class open.

Note for reviewers: the patch is much smaller than 160kb, most of it is just the dump. Also, the only real change is

-    $properties['target_id'] = DataReferenceTargetDefinition::create('integer')
+    $properties['target_id'] = DataReferenceTargetDefinition::create('string')

everything else is just working around the strictly typed nature of SQL...

jibran’s picture

Status: Needs review » Reviewed & tested by the community

Thanks @chx for fixing my useless reroll. :D Great to see patch is green again. I think this is really ready. We can create a 8.x-2.x and continue work in follow ups.
As for the basefields upgrade path is concerned I think patch is already too big. Let's move basedfields upgrade path tests to follow up. Leaving this RTBC for two three days so that people can review this.
Meanwhile I'll provide the basefields script so that we can create another dump for testing.

As I said before this is ready. These are just doc improvements/questions and can be moved to follow up.

  1. +++ b/dynamic_entity_reference.install
    @@ -0,0 +1,56 @@
    +function dynamic_entity_reference_update_8001() {
    

    Would you like to add more technical description for this update? You told me to add references to SqlContentEntityStorageSchema::updateDedicatedTableSchema(), FieldStorageDefinitionUpdateForbiddenException and SqlContentEntityStorageSchema::saveFieldSchemaData()

  2. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,137 @@
    +class FieldStorageSubscriber implements EventSubscriberInterface {
    

    I think some description for this service and how der uses the whole trigger idea would be good. What do you think?

  3. +++ b/src/EventSubscriber/FieldStorageSubscriber.php
    @@ -0,0 +1,137 @@
    +    $events[FieldStorageDefinitionEvents::CREATE][] = ['onFieldStorage', 100];
    +    $events[EntityTypeEvents::CREATE][] = ['onEntityType', 100];
    +    $events[EntityTypeEvents::UPDATE][] = ['onEntityType', 100];
    

    You explained the reason to me in IRC about these events i.e. why we can't use FieldStorageDefinitionEvents::UPDATE and why we need EntityTypeEvents. Can we state that reason here?

  4. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,116 @@
    +abstract class IntColumnHandler implements IntColumnHandlerInterface {
    

    Let's add the trigger related docs here. I can understand the concept by looking at this patch but I don't now the complete idea. What triggers are how they work and etc.

  5. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,116 @@
    +        if (strlen($trigger) > 64) {
    

    Is there a constant for this we can use? I tried to find out this limit but I had no idea where to look.

  6. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,116 @@
    +   * @TODO not sure whether we want to bother with deleting.
    

    Let's create an issue for this and explore further.

  7. +++ b/src/Storage/IntColumnHandler.php
    @@ -0,0 +1,116 @@
    +  public function delete($table, $column) {
    

    This will be an abstract function too when we'll implement it?

dpi’s picture

Leaving this RTBC for two three days so that people can review this.

Thanks for this! I'm going to do some manual testing with my DER-dependent modules.

jibran’s picture

StatusFileSize
new9.02 KB

RE: #73 this will go into new branch so you don't have to upgrade immediately but feedback is welcome. :-)
@chx: Here is the interdiff for multiple basefields. For me this test is passing but you can add more specific asserts.

chx’s picture

Note: the fixup module could be nicer as EntityType has get/set methods. I can't even be bothered to reroll to simplify a very temporary module.

larowlan’s picture

+++ b/src/Storage/IntColumnHandler.php
@@ -0,0 +1,116 @@
+  public function delete($table, $column) {

Is this intentionally blank?

Got to say I was cautious about this patch, but now having read it, I think it is architected really nicely.

Great job folks.

jibran’s picture

Got to say I was cautious about this patch, but now having read it, I think it is architected really nicely.

Same here but now I'm really excited about it.

chx’s picture

Regarding delete, there's a TODO and I am well aware this might or even likely to break if you drop a DER field from a shared field table. That's extremely rare and I am confident to leave it to a followup.

chx’s picture

Since this is supposed to be a new branch could we go ahead with the commit so I can file followups?

jibran’s picture

I have already created 8.x-2.x branch. I just added you as a branch maintainer. Given that you have come up with the solution and build this from ground up it seems fair to us, Lee and me, that you should maintain this branch.

Thank you for your work and WELCOME TO THE JUNGLE!

chx’s picture

But... but... I don't want to commit without your most valued reviews and other help. What about I still post patches, you review and I commit?

I will commit this one soon. Thanks for the confidence.

jibran’s picture

But... but... I don't want to commit without your most valued reviews and other help. What about I still post patches, you review and I commit?

Who said you are getting away from my annoying reviews? :D

Well, we always followed core practices in this module and we intend to do the same for 8.x-2.x as well.

dpi’s picture

StatusFileSize
new42.42 KB

edit: removed sarcasm detector image

  • chx committed 1aec1bf on 8.x-2.x
    Issue #2555027 by chx, jibran: Support non-numeric entity ID's
    
chx’s picture

Status: Reviewed & tested by the community » Fixed

What sarcasm? I really do value jibran's review and help. This issue has been one of the most delightful experiences I had in Drupal land in years. Speedy and collaborative, as it should be.

Committed!

jibran’s picture

Version: 8.x-1.x-dev » 8.x-2.x-dev

What sarcasm?

Yup, there is no sarcasm here only gratitude. :)

I really do value jibran's review and help.

You are a legend of Drupal world and it's an honour to work with you.

This issue has been one of the most delightful experiences I had in Drupal land in years.

I'm so happy that I'm able to participate in this.

I have created a new dev release so that we can create 8.x-2.x backlog. I'm going to create some follow ups.

dpi’s picture

Version: 8.x-2.x-dev » 8.x-1.x-dev

> You are a legend of Drupal world and it's an honour to work with you.

I echo this. I only say its seemed like sarcasm because you are more than capable :)

Never mind.

Thankyou for your valuable contribution chx

jibran’s picture

Version: 8.x-1.x-dev » 8.x-2.x-dev

I have created 6 child issues

  1. #2766175: Fix the Views integration for entities with string ids.
  2. #2766181: Remove dynamic_entity_reference_275136
  3. #2766187: Allow config entity references
  4. #2766189: Fix IntColumnHandler::delete
  5. #2766193: Add tests for multiple base fields
  6. #2766195: Improve documentation of non-numeric entity ID's support

Write a core patch so that the field type can rewrite the field column as necessary when entity query runs.

Would you like to file a core issue for this? I think we should create a DER issue as well to track the progress we'll have to create an issue any way to make the code changes. Just a side note this going to be an api addition so I think we should speed up this process if we want it to be included in 8.2.x releases. First beta is due on 3rd August.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.