Support revision tracking of entity references in core

Please use a separate field type for this.

@maximpodorov, why do you think we should use a separate field type?

There was the huge discussion in #1837650: Allow referencing a specific revision ID.
The main reason is entity reference and entity revision reference are two separate concepts:
- They require different column count for the field
- The same can be expressed mathematically: they point to different dimension spaces: entity reference is a scalar value, and entity revision reference is a vector which consists of two values, so ordering, equality are not quite clear concepts for them.

My first reaction was to tend to the idea to fix this issue as:

- Closed (Works as designed)

Reasoning:

What you want when publishing several changes together is 'workspaces' (like in workspace module) or 'site versions' (like in CPS).

However you are right that that would only work correctly with something like field collection, where updating a child entity updates the parent, too.

--

Could you explain how your wanted workflow would look like for an editor?

e.g. Even after publishing the changes to the child entity, nothing happens until the editor has selected the new upgraded version?

I am not yet clear how that would work.

#6: That does unfortunately not answer the question.

The question is (from a product perspective):

"How does such a workflow look like? How can we avoid to confuse people?"

e.g.

As a content manager when I want to publish content I need to do three steps:

1. [...]
2. [...]
3. [...]

So there are 2 ways to implement that:

If I change the child, would:

a) the parent be automatically updated with the new revision? (e.g. just by saving the child)
b) the parent be manually updated with the specific revision? (e.g. by an editor)

e.g. Do I understand correctly that you want field_collection like behavior (a), but without all the overhead and problems it brings?

Also:

How would this be displayed with content_moderation? Would only the parent be for moderation in that case with the included things?

#8: Also what happens in the child lives in several parents?

- Would then all parents be changed at once?
- Would only the parent be changed in which the child was updated? (e.g. track this via IEF somehow)

@Fabianx Field Collections is in discussion to be deprecated by Paragraphs:
#2784931: Proposal: Deprecate Field Collections for Drupal 8, focus on Entity Reference Revisions & Paragraphs

Paragraphs builds on top of ERR and relying on its feature "composite relationship". It means, Paragraphs declares that every paragraph is a composite of the host entity. Thus the paragraph entity points to only one single parent, being owned by that.

A paragraph is never edited on its own. Even if the UI could allow that, the paragraph revision will be entangled with the host entity revision. A child should not have own revisions that are forward or moderated separately. Instead, all updates happen in sync.
The child has no own workflow. I guess this can also be compared with the idea if fieldable fields.
#607396: Killer feature: Fieldable Fields in core

Combining the principle with revisions and workflow (forward revisions) and quickedit results in a crazy mess. Many people believe that in future Drupal should only support atomic revisions, but that will make it even worse. Doing 5 small (field) edits with a paragraph containing 5 paragraphs with each 3 fields in 5 languages results in... lots of uncountable updates and bad performance and that's not an extreme example. In reality there will be cases with 50 paragraphs, 20 languages and more.
To make this work acceptable, we need to think about ideas like:
- revision pinning, to create a single forward revision, but only update the same revision as long as it is not published during all quickedits.
- revision cleanup, allowing to drop intermediate revisions if they are excessive and add no value. Revisions can eat lots of unwanted space. We learned in Paragraphs that it's hard/complex to clean up.
- flagged unchangeable revisions, so in case a revision is published, we can consider it atomic.
- major / minor revisions, so users can promote a revision to keep.

Reusability: A regular Paragraph is by design not reusable. This will happen through a separate entity with its own workflow. Promoting one into reusability will allow to add a label and other metadata to maintain the library (extra fields, ...). This part is not yet implemented. A library item could have a tab listing its usages, we could inform the user that some of those point to past revisions and allow the user to update to the most recent one... Or we could allow a user to update all usages when a new revision is published. We could allow the site builder to decide about the workflow and the need of user awareness.
#2414865: [META] Reuse paragraphs / Library

Please ask specifically, if you need to know more about the precise design of Paragraphs or ERR and something is unclear.

So basically it is field_collection, but with a different technical implementation:

- A host + sub entities that are part of the host

But why would you want to change entity reference fields for that?

I would call that an "embedded entity field", which technically looks like an entity reference field, but internally ensures the embedded entity is always only:

- a) created as part of a host entity (or container)
- b) if edited, the host entity is updated with the new revision
- c) deleted as part of a host entity

Again this is the same as field collection, but without the restriction of only referencing field_collection entities.

It kinda is the best out of both worlds (ER + FC).

However compared to FC, where editing was a nightmare (especially performance wise), it would use the normal IEF or modals or .... Basically only updating the referenced revision in the host after editing. Would still need to ensure to remove the referenced entities from any kind of workflow though.

The main and difficult thing is to ensure that the referenced entities know their container and that the container is unique. (and there is different ways to ensure that obviously).

So for me this feels like a new field type and a different use case from normal ERs (which have re-usability and independence in mind) and I suggest to change the feature request accordingly.

Maybe it is possible to subclass ER to implement the missing parts?

#14: Agree and yes the field type should ship with the inlined UI - as per definition the referenced entities should not be edited outside of the context of the parent (as that does not make sense).

If this gets too large a feature request, it might be abetter idea to open an ideas issue with the roadmap and vision?

+1 For Entity Reference Revision in core.

This is important for many complex data modeling use cases, not just content moderation.

Fieldable-fields / paragraphs is imho a separate topic, who is interested in this might love #607396: Killer feature: Fieldable Fields in core.

I'd like to add that I think this is essential. I'm very much in favor of the "Extend the core Entity Reference module to (optionally) support revisions" option. I'm looking now to see if I can make a first-draft type patch in that direction.

I find the use cases outlined by DamienMcKenna to be very common among my customers.

As a content editor, I want to 1) create complex pages with different types of things dynamically on the page without having to use a hundred different predefined content types, and 2) be able to use an editorial workflow around it so that drafts of the sub elements behave as if they are parts of the parent entity.

If Drupal were to better support these use cases out of the box, this would be a game changer for me and my clients.

I've started on this and you can see some work here:

https://github.com/rgristroph/drupal-2812595/pull/1

It's not ready for a patch yet. I have an error when installing the comments module, I was trying to make my changes apply only if the "reference specific revisions" box was checked, so all existing entity reference fields should continue to work as designed, but I still have something to work on there. This is not yet read for a patch obviously, but hopefully I will post one here soon.

The error is
Drupal\Core\Entity\EntityStorageException: Exception thrown while performing a schema update. MySQL needs the 'entity_id' field specification in order to normalize the 'comment__num_new' index in Drupal\Core\Entity\Sql\SqlContentEntityStorage->wrapSchemaException() (line 1509 of /var/www/drupal/docroot/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

if that rings a bell for anyone ?

I fixed the issue I asked about in #21, and updated the repo linked to there.

I moved the checkbox to the field storage settings, and removed a lot of debugging statements that were in there previously.

Now, with that change I can install a fresh drupal (standard profile), create two content types, add an entity reference field on one pointing to the other -- when I save the nodes, and subsequently edit them, the revisions do show up in the node__field table and node_revision__field_ tables, in the "_target_revision_id" columns.

This doesn't do anything related to tracking revisions (like making parent and child go through workflow together) and has no tests, and no way to update or change previously existing fields. The next thing I will work on is the update process.

@damienmckenna Good points, let get into my thinking a bit and maybe I need to adjust the approach.

I don't want to write much or anything from scratch if I can help it, the changes above are almost all grabbing stuff from entity_reference_revisions and moving it to the entity_reference field in core.

I'm working on a project where it started building complex content with entity_reference a year ago, has accumulated a lot of it, and along the way added a moderation workflow, and then some some hack-ish code to sync workflow states from the top "page display" entity to it's children "page items" entities. It mostly works, the base case is OK but things like reverting a "page display" and getting back the same versions of "page items" don't work.

It would have made a difference to use entity_reference_revisions in all our content types from day one, of course, but it wasn't obvious at that point that we needed to. In the same way new projects starting now will use the core entity_reference also. The common pattern will be people build complex content and enter in a lot of it using entity_reference, as the project matures and gets more users they hit a wall with respect to revisioning their content -- they can't do very standard "drupally" things like revert to previous revisions, and the need to rebuild the content structure using entity_reference_revisions and then do a migration. That feels a bit disconcerting, I think if entity_reference is in core, revisions are in core, and now a content workflow is in core, entity_reference needs to be revision aware and allow these things to work together.

I think it comes down to that Drupal should be keeping all the revisions on the entity reference fields by default, so you don't get into that situation. 99% of the time the revision can be ignored when displaying content, and I think we can make a patch that doesn't change the site-building process for people using the current setup, but makes it so the components of core are all revision-aware if you need them to be.

All the philosophizing aside though, let's get practical --

It will take a while to get any change into core.

We can't set people up to have a data integrity mess, I see the issue with that. I lean towards thinking the current situation is a data integrity mess though, if we can at least reduce the potential for that to a few rare corner-cases that will be a big win.

I also think it will take a long time and a lot of testing to get confidence in any patch here.

Attaching a patch that encapsulates everything in that repo linked above.

I'll take a look at those failures and update.

This patch sets $strictConfigSchema = FALSE on two tests, and I'm not sure that's the right thing to do. Just uploading it to see what d.o does for test results.

Re #31: that won't work, you need to update core schema YML, and then fix the tests.

I've made some adjustments. Please look at the interdiff.

I've just realized that I have added composer.lock to the patch ... Sorry about that. Here is the clean version.

I think that at first we should not support revision reference fields through the widgets, because that raises a bunch of questions like - what happens with the field value if the referenced entity is saved in a new revision.

We could initially only support setting the field through the API. The field formatters however will be working.

Adding an update to populate the new setting for field storage configs.

Please, do this in the Entity Reference Revisions field type. Entity Reference field type is not the right place for such experiments.

I've worked on older state of 8.7.x, which is why the last patch couldn't apply. Here is a re-roll.

@maximpodorov, the new functionality is available only per setting. You don't get the columns in the DB nor the properties unless you activate the setting in the field storage config. Do you see any problems regarding such approach?

Yes, please see #4 and the whole separate thread ended with "Won't fix".

Okay, let me first get it green so that we have it as a reference and see what changes would be required for having one configurable field type. From there we can extract the important parts into a new field type.

I think we can close this after #2725523: Add a revision_parent field to revisionable entities

In the referenced issue we don't introduce a general revision reference field. With the RevisionReferenceItem from the referenced issue, we can reference only revisions from the entity type on which the field is placed and we don't have the computed property entity yet.
The issue here is about having a general field for referencing entity revisions across different entity types.

We could either extend from RevisionReferenceItem or add additional functionality to it.

Any possibility this will see any movement? Really vital feature, but seems to be abandoned.

Added reroll of patch #55 on Drupal 9.4.x.

This feature exists in the separate module. It's not necessary to get everything in the core (and break BC).

Well, it's a nightmare to convert base fields to contrib ...

Fixing failed Drupal CS issues of patch #67.

From @catch in the #yes-no-queue channel in Drupal Slack re: this functionality in Core:

I have a knee jerk reaction against it. I understand using it for commerce where you want to reference a product frozen in time on an order and similarish accounting stuff, it makes me uneasy for content workflow things for reasons similar to the ones fabianx gives on there. This doesn't mean it shouldn't be supported but I think the issue summary needs better pros and cons.

Given that entity_reference_revisions is currently the 6th most installed across the current codebase, I'd say the demand is there for the functionality to be in core. It's useful in a few places, such as referencing certain revisions of custom content entities as we do in custom blocks for Layout builder, but I admit ours is a corner case. Paragraphs uses this same model and without being able to reference a revision of a child element, reversion is complicated.