Problem/Motivation
Dynamic entity reference, unlike core entity reference currently only supports referring to entity types with integer entity IDs. There are two problems with this:
- Config entities are out
- Core actually supports content entities with string IDs (it's tested with the entity_test_string_id entity type).
Proposed resolution
This is tricky. If we were to store DER.target_id as a string then joining to any "normal" entity on PostgreSQL would throw an error. On MySQL it doesn't but it'd be a serious performance hit (I don't actually have a problem with since I work for a scalability agency and yay, job security! but I do not think the community would be happy about it). Also, sorting would break. Not sure why would anyone want to sort on an entity ID but I thought I will mention that.
So storing an integer is not enough and storing a string is broken in various sometimes subtle ways.
So the idea is to store both. For Entity API, define target_id as string. In the SQL database, since this is a SQL database problem, keep an integer column with the target_id casted to int and use a trigger to maintain it. Views and entity query needs to know about the integer column since they talk directly to SQL but nothing else should. In particular, the field item class should not know about it, it's quite complex already. Another possible solution offered by core is a custom storage which is probably more code than this and also doesn't allow a DER field in the base table. We don't want that.
Drupal never used triggers before and of course the syntax varies between databases but backend_overridable was made for this.
Remaining tasks
- Add a views field alias (if possible, I am not sure) to Views so target_id_int never shows up in the Views result rows. Probably won't allow reverting all tests since they directly test a mostly internal structure.
- Still in Views, support string ID'd entity types. With the current patch Views only supports integer IDs because it unconditionally switches to the integer column.
- Add even more explicit tests. derUpdateTest already checks after update but a test for referring string IDs in shared tables and field configs created (not updated) should be tested too.
- Write a core patch so that the field type can rewrite the field column as necessary when entity query runs.
User interface changes
None yet. In a followup we will need to figure out how to refer to config entities. That is always fun. Especially because not all has labels to boot.
API changes
Surprisingly none.
Data model changes
One new column per DER field is added into the database and a trigger per table containing a DER field is added.
Original report:
Cores Entity Reference will set its 'target_id' column schema to either integer or string. By determining the target entity type' primary ID data type.
Dynamic Entity Reference only supports entity types which have a integer for its ID data type.
DERItems 'target_id' schema data type should default to varchar, therefore supporting all entity types. With an opt-in setting for performance, setting the target_id schema to integer. When the field storage is being created in Field UI, show a list of entity types the field will/will not be able to reference depending on the choice.
| Comment | File | Size | Author |
|---|---|---|---|
| #83 | maxresdefault.jpg | 42.42 KB | dpi |
| #74 | interdiff.txt | 9.02 KB | jibran |
| #71 | 2555027_71.patch | 160.77 KB | chx |
| #71 | interdiff.txt | 1.74 KB | chx |
| #69 | 2555027.sh_.txt | 1.28 KB | chx |
Comments
Comment #2
dpiComment #3
jibranI'm giving it a lot of thought I need to collect some more info about it and http://larowlan.github.io/ds.core-til/#/6/3 so please bear with me. Thanks.
Comment #4
jibranThe idea is to introduce the new DER field type for config entity see #2365331-16: Update DynamicEntityReferenceItem according to EntityReferenceItem for more details. When I converted target_type to integer i thought all the fieldable entities have integer ids. After some research I realized I was wrong.
After discussing with @larowlan we decided to do following.
1) Show all the entities config/content on field storage form. Just like ER field in core.
2) If all traget_types have integer id. We create a schema with integer column.
3) If any one of the traget_type has a string id. We create a schema with varchar column.
If the field has data then it is not a problem because we can't change target types and we don't have to change schema.
If the field has no data and the schema was int before and after re-saving the field it changed to varchar or vise versa then Drupal will update the table automatically as it is done for ER field.
Comment #5
dpiWould I be wrong if I said content entities can have string ID's? Im not sure making a config/content distinction is correct.
Comment #6
jibranContent entities can have string ID's.
Comment #7
dpi9433787:
Im not sure what these are supposed to mean then. The field should either support numeric or both. Doesnt matter whether its config or content?
Comment #8
jibranThat comment doesn't matter now we have new plan, turn target_id column to string by default and if all the target entity types id has integer schema then we'll change the column type of target_id form string to integer automatically. I hope this makes it clearer.
Comment #9
dpiTurns out this is a big issue for Postgres. If I use a string in an EFQ condition for DER target entity id then Postgres will throw an exception. RNG/Courier issue: Postgres tests failing due to string usage in integer field.
Comment #10
dpiMaking a personal note here.
If exclude_entity_types is TRUE or entity_type_ids is omitted then it means target IDS are uncertain and should default to string.
Comment #11
chx commentedEdit: deleted.
Comment #12
jibranComment #13
jibranI had a chat with @larowlan today and we both think that it'd be good to add a new field type for sting entity ID's so that we don't end up in the mixed situation. ER is doing the same thing as well.
Other then flag #2678756: Allow config entities to be flagged I don't see any use case of mixed entity IDs. flag can have two DER basefields one for int entity id and other is for sting entity id and flagging/unflagging action or preSave can populate the respective DER field base on flagged entity id.
Given that @berdir is already involved. I'll ping @yched and @fago about it as well.
Comment #14
chx commentedEdit: deleted.
Comment #15
chx commentedEdit: OK I will bite.
The solution is for Drupal to grow up and use databases for real and not like MySQL 4.1 or so.
On the application side, target_id should be string so that it can encompass everything.
The SQL database should maintain a cast-to-unsigned column and index it. MySQL 5.7 and Oracle, virtual column, PostgreSQL, SQLite and MySQL <5.7 triggers. The only purpose for this column is faster querying, it shouldn't be visible to (most of the app).
The app only needs to be aware of this column in the two abstraction APIs we put on top of SQL: Views and entity query. For Views, this is really simple, doesn't even need a core patch, just the views data needs to look at whether the referred entity type uses strings or ints as IDs, for entity query we would need to patch up the eldritch horror that is
Tables::addField, it can't get much worse.So
db_query. In the MySQL version split on database server version. Copy the necessary two LoC fromDrupal\Core\Database\Driver\mysql\Install\Tasks::checkEngineVersionto figure out the MySQL server version. Figuring out the column name is a bit of work but I do not think it's too hard. I believe it's event based andEntityTypeEventSubscriberTraitis where we need to listen.Altogether, hopefully all this is not a lot of code. The user does not need to be aware of anything funny going on. What's not to love?
Comment #16
chx commentedThis is just the boilerplate. It doesn't do anything and indeed it'll break because it switches to _int in views unconditionally and it's not created yet. But it shows the way.
Comment #17
jibranThis apporoch is quiet interesting IMO. Let's see how this shape up. Nice start @chx. Here are couple of minor point.
We need an upgrade path for and upgrade path test for this.
Not needed.
Comment #18
chx commentedWith the attached patch, DynamicEntityReferenceItemTest runs...
Comment #22
jibranYou can copy
EntityReferenceItemTest::testContentEntityReferenceItemWithStringIdtoDynamicEntityReferenceItemTestto check your changes as well.Comment #23
chx commentedHrm, tests pass here :(
Comment #28
jibranHere are some review points.
Some doc love here please.
We'd need a follow up to update this for string target ids?
Maybe explain it a little.
Desc is missing.
Nope.
I'm always confused, should this have to be instance or SESI or should we check the type of entity_id key?
Some more doc love please.
Incomplete docs. :)
This is some real coding :D. More docs please.
DERItem::schema also needs an update.
Desc missing.
Nope
Doc block missing.
Comments please.
Do we need this?
This is gibberish to me so please add @see, @link, @code or docs to elaborate it. :)
Comment #29
chx commented> We'd need a follow up to update this for string target ids?
Aye but since config entities are not stored in "regular" tables the follow up is only for "freak" content entities that have string IDs. I believe the entity system allows for that but AFAIK it is not tested anywhere in core.
> I'm always confused, should this have to be instance or SESI or should we check the type of entity_id key?
The type entity_id doesn't matter. SESI holds the
getTableMappingmethod (and nothing else, in fact) and that's what we need.This patch has significantly rewritten the subscriber, the code is much simpler and allows reaction to entity type create which makes the code work with
hook_entity_base_field_info. When I get around to write an update hook it will also be helpful.There is no interdiff as the interdiff is larger than the patch... it's a big change with only the trigger code remaining untouched although it has gotten a bit of defensive programming and comments.
Comment #30
larowlanThere's a test content entity in core that has string ids
Comment #31
chx commentedo_O well then! Hello
entity_test_string_id! Since DER is already broken if you point to that, can we postpone handling that case to a followup still or should we roll it into this one? My plans for this issue were adding a query into a test or two to check the int column contains the right value; this is tested implicitly with views tests but I want to add an explicit test and also pgsql and sqlite drivers. That'll be enough for one go, what do you think?Comment #33
chx commentedHere we go for SQLite. I factored out the common parts into IntColumnHandler. Yes, some doxygen is missing.
Comment #35
chx commentedComment #36
chx commentedComment #37
jibranThanks @chx for updating the issue summary.
Let me have a look at theses so you can work on rest of them.
I'd like to fix it here but this patch is already pretty big so let's wait for now.
Nice work on SQLite. Moar docs please.
Comment #38
chx commentedPostgreSQL. Yesssss.
Comment #42
chx commentedHrm, let's drop.
Comment #46
jibranDo you think this will be enough for views fix?
Comment #47
chx commentedPostgreSQL is _annoying_ . I had DynamicEntityReferenceRelationshipTest passing before ; but that didn't drop things ; then I needed to DROP things and that broke PostgreSQL because even a DROP statement is elaborate and annoying. Oh well. I have shortened the trigger name while at it.
The views fix imo is a followup.
Comment #48
chx commentedNote that #2751363: Don't allow test entities to share base table fixes the fails as far as I can see. Not sure how to proceed.
Comment #49
jibranDump file.
Steps to reproduce.
Install testing profile.
Run
drush scr test.phpRun
php ./core/scripts/dump-database-d8-mysql.php --no-ansi > dump.phpRun
gzip dump.phpComment #50
chx commentedThe PostgreSQL tests are failing with
This is a core bug, when you try to install two Drupals in PostgreSQL in parallel then
CREATE OR REPLACEruns in parallel and results in this error. A race condition gets triggered here.Comment #51
chx commentedThis will fail without #2751363: Don't allow test entities to share base table but here we are.
Comment #54
chx commentedprotected $installProfile = 'testing'; is necessary since the testing profile was used to dump.
Also, I coded around #275136: TinyMCE don't submit format, but only text. It's not pretty but it hopefully makes for a green test. Please do not add additional tests before mysql comes back green.
Comment #55
chx commentedOh damn I typoed the module name. Oh well :D It's #2751363: Don't allow test entities to share base table .
Comment #57
chx commentedComment #59
jibranThese fails will go away after merging 8.x-1.x into the patch.
Comment #60
chx commentedI dunno about those fails, I am rebased on origin/8.x-1.x. Here's a patch that passed locally and tests referring to a config entity because we can do that. I have also uploaded the script I use to test this patch, for it to work correctly commit the patch to branch 2550227 under modules/dynamic_entity_reference .
Comment #61
jibranOh! my.
Comment #62
chx commentedLet's see those fails.
Comment #63
chx commentedComment #64
dawehnerWhat about making it abstract ?
In general it would be great to explain a bit of the reasoning behind the SQL trigger on the relevant classes, but of course, this is contrib, this could totally happen later.
Comment #65
jibranThanks @chx for making it green. Should we create a new 8.x-2.x branch for this patch? We can release an alpha tag for this. There is an upgrade path and test for this so I think it is doable. Meanwhile we can make 8.x-1.x stable and forward port the patches as well.
+1 to #64. I think we should improve the overall docs.
I think we can move these things to followup:
I'll work on the trivial feedback.
Let's add a comment about it and let's change the name as of variable as well.
Let's add a comment here as well.
Let's add a comment here as well.
Let's move this out of the loop.
So we are converting the existing target_id column form int to string and service will add the trigger and target_id_int? Let;s add a comment about it.
I think we can pass the field_name and field_storage_definition as well.
We need KTB for all these events.
Aren't these two the same lines?
Do we need tests to be sure about it?
DerUpdateTest
{@inheritdoc} is missing.
It should be good to document the original generating script here.
Let's document this.
Let's move this to follow up issue because we have to change the DER field storage form to allow config entities. Let's use
entity_test_string_idinstead.But I love this assert.
so cool
Comment #66
jibranRE: Trivial feedback.
entity_test_string_idand it is causing some fails. PFA my updated script with field settings. Patch is still green on old db dump.In this patch.
Comment #67
jibranComplete interdiff can be found at https://github.com/jibran/dynamic_entity_reference/commit/9ac74e6087dbe2...
Comment #69
chx commentedThanks for the amazing amount of quick work! I have re-dumped and rerolled. Note that dumping this without https://www.drupal.org/files/issues/2555027.sh__0.txt is not feasible: as far as I can see the previous patch didn't include the fixup module. I reuploaded the script because now it points to test.php__17.txt instead of 15. There's no interdiff as there's no change (yet). The identifier issue is worrysome but easy to fix with a little hashing. Will do it next but first I want to see what the bot says.
Comment #71
chx commentedEasy as pie. Instead of
simpletest381403entity_test__entity_reference_string_id_der_updatewe havesimpletest425511entity_test__entity_reference_string_id_208e2b58. It won't always be this nice but close. It uses the first 56 characters of the prefixed table and so in the non-test, most common case where the user has no prefix the hash won't be necessary:generateFieldTableNamewill produce a max 48 character name and _der_update only adds 11 which is below the max 64. It's only with simpletest (or other prefix) the hashing becomes necessary.Did the abstract change dawenher asked for as well since I had the class open.
Note for reviewers: the patch is much smaller than 160kb, most of it is just the dump. Also, the only real change is
everything else is just working around the strictly typed nature of SQL...
Comment #72
jibranThanks @chx for fixing my useless reroll. :D Great to see patch is green again. I think this is really ready. We can create a 8.x-2.x and continue work in follow ups.
As for the basefields upgrade path is concerned I think patch is already too big. Let's move basedfields upgrade path tests to follow up. Leaving this RTBC for two three days so that people can review this.
Meanwhile I'll provide the basefields script so that we can create another dump for testing.
As I said before this is ready. These are just doc improvements/questions and can be moved to follow up.
Would you like to add more technical description for this update? You told me to add references to
SqlContentEntityStorageSchema::updateDedicatedTableSchema(),FieldStorageDefinitionUpdateForbiddenExceptionandSqlContentEntityStorageSchema::saveFieldSchemaData()I think some description for this service and how der uses the whole trigger idea would be good. What do you think?
You explained the reason to me in IRC about these events i.e. why we can't use
FieldStorageDefinitionEvents::UPDATEand why we needEntityTypeEvents. Can we state that reason here?Let's add the trigger related docs here. I can understand the concept by looking at this patch but I don't now the complete idea. What triggers are how they work and etc.
Is there a constant for this we can use? I tried to find out this limit but I had no idea where to look.
Let's create an issue for this and explore further.
This will be an abstract function too when we'll implement it?
Comment #73
dpiThanks for this! I'm going to do some manual testing with my DER-dependent modules.
Comment #74
jibranRE: #73 this will go into new branch so you don't have to upgrade immediately but feedback is welcome. :-)
@chx: Here is the interdiff for multiple basefields. For me this test is passing but you can add more specific asserts.
Comment #75
chx commentedNote: the fixup module could be nicer as EntityType has get/set methods. I can't even be bothered to reroll to simplify a very temporary module.
Comment #76
larowlanIs this intentionally blank?
Got to say I was cautious about this patch, but now having read it, I think it is architected really nicely.
Great job folks.
Comment #77
jibranSame here but now I'm really excited about it.
Comment #78
chx commentedRegarding delete, there's a TODO and I am well aware this might or even likely to break if you drop a DER field from a shared field table. That's extremely rare and I am confident to leave it to a followup.
Comment #79
chx commentedSince this is supposed to be a new branch could we go ahead with the commit so I can file followups?
Comment #80
jibranI have already created 8.x-2.x branch. I just added you as a branch maintainer. Given that you have come up with the solution and build this from ground up it seems fair to us, Lee and me, that you should maintain this branch.
Thank you for your work and WELCOME TO THE JUNGLE!
Comment #81
chx commentedBut... but... I don't want to commit without your most valued reviews and other help. What about I still post patches, you review and I commit?
I will commit this one soon. Thanks for the confidence.
Comment #82
jibranWho said you are getting away from my annoying reviews? :D
Well, we always followed core practices in this module and we intend to do the same for 8.x-2.x as well.
Comment #83
dpiedit: removed sarcasm detector image
Comment #85
chx commentedWhat sarcasm? I really do value jibran's review and help. This issue has been one of the most delightful experiences I had in Drupal land in years. Speedy and collaborative, as it should be.
Committed!
Comment #86
jibranYup, there is no sarcasm here only gratitude. :)
You are a legend of Drupal world and it's an honour to work with you.
I'm so happy that I'm able to participate in this.
I have created a new dev release so that we can create 8.x-2.x backlog. I'm going to create some follow ups.
Comment #87
dpi> You are a legend of Drupal world and it's an honour to work with you.
I echo this. I only say its seemed like sarcasm because you are more than capable :)
Never mind.
Thankyou for your valuable contribution chx
Comment #88
jibranI have created 6 child issues
Would you like to file a core issue for this? I think we should create a DER issue as well to track the progress we'll have to create an issue any way to make the code changes. Just a side note this going to be an api addition so I think we should speed up this process if we want it to be included in 8.2.x releases. First beta is due on 3rd August.