[META] Clean up references to deleted entities

Hi, I'm also seeing this behavior. We're using 8.1.8 over here. We'll be looking into this shortly.

I suggest this is major because it undermines data integrity.

This issue has a long history on D7: see #1368386: Delete references to deleted entities. There is no consensus there on how to fix it, because of the performance trade off - people want it:
- immediate (so the broken references don't persist for some unknown period of time)
- thorough (with hooks and consequences, so that other relationships in the data can be maintained)
- scalable (because someone might try to delete 100000 nodes in one go)

Because of the need to be thorough, it is impossible to be both scalable and immediate.

This is the identical dilemma found in #89181: Use queue API for node and comment, user, node multiple deletes.

We did provide a special solution for this in https://www.drupal.org/node/2390467 for user roles. The idea there was to use the same approach here by adding a generic method to the storage.

That might still be possible if we make it optional. However, it would be possible to go through the hooks.

There's no solution for now?

Is there any update on the issue ?

Because of the need to be thorough, it is impossible to be both scalable and immediate.

This is the identical dilemma found in #89181: Use queue API for node and comment, user, node multiple deletes.

Won't set the field record deleted column to 1 address both scalable (deletion will be silently done by cron) and immediate (field won't show up on UI)?

I was notified of this issue after I announced the availability of Entity Reference Guards. It aims to provide a rough framework for all kinds of entity reference handling, including reference clean-up and protection. Plans are made to support deferred guards, but those have not been implemented yet.

It looks like https://www.drupal.org/project/erg has the potential to solve this particular issue

any news about this?

Is https://www.drupal.org/project/erg the workaround for this or will this fixed in the core?

It's a really important function! It makes absolutely no sense if that is not the normal behavior.

Won't set the field record deleted column to 1 address both scalable (deletion will be silently done by cron) and immediate (field won't show up on UI)?

I think the issue is that we don't really expose anything like that on the entity storage API yet.

@dawehner and what does that mean in concrete terms?

hi all, same issue here... any idea how to solve this issue? is critical for me!

You could try out Entity Reference Guards and report back here what you're missing in detail, which could or should be a part of Core's entity reference functionality.

For me, the current behavior of Entity Reference makes sense. It's actually helpful, e.g. when you want to restore accidentally deleted entities and having their relations back. People regularly get confused about this behavior, especially when they come from a domain where referential integrity seems to be something "natural", which is not. Implementing this on application-level is a torture though. You won't be able to satisfy anyone, since this is usually a requirement on database-level.

https://www.drupal.org/project/erg Says it's abandoned and there are virtually no installs. Is there some other solution in the works? Is something coming in core? Where should efforts be directed to help with a solution?

Despite of the unsupported status, you can still verify whether this module might be suitable. If so, or at least if it already has implemented a lot to achieve the goal, then it can still be "revived" (thanks to GPLv2).

Referential integrity on database level will most likely not being supported by core Drupal 8, as well as Drupal 9.

This problem is quite painful, as legitimate actions performed by editors break facets. I think we could try to solve the issue by splitting it into smaller pieces, for example:

• #2978522: Add a batch job that removes orphaned references in a given field
• #2978521: Remove orphaned (dangling) entity references when an entity is deleted

The latter could probably be split further. When the deletion is triggered from the browser, eg. via the confirmation form, we could add a checkbox which would start the cleanup batch right away. For a non-interactive deletion (eg. PHP API, REST API) it might make sense to use cron queue workers.

Splitting up is great. But isn't #2978521: Remove orphaned (dangling) entity references when an entity is deleted just a duplicate of this issue? The title is basically identical...

That's true. This issue represents a broader problem, so turning it into meta.

+ Subscribing, this is an issue for us when using GraphQL entity reference fields, data for orphaned references are null value, which isnt helpful as its not usable Graph information.

I am having the same issue where when I delete the parent node reference, all the references are not deleted, is there any way we can trigger a entity delete when a parent reference is deleted?

is there any way we can trigger a entity delete when a parent reference is deleted?

Yes, but this isn't the best place for a general support question like that; I suggest Drupal Answers.

For reference, a similar functionality is implemented in Organic Groups to clean up orphaned group content. The implementation uses a plugin type for choosing the way to perform the purging of the orphans, with three plugins:

1. Batch: A batch process that can be triggered in the UI.
2. Cron: Clean up orphans in a cron job.
3. Simple: Cleans up orphans immediately, with an associated performance penalty.

On hook_entity_predelete() the entity being deleted is registered in the plugin manager for processing, ref og_entity_predelete(). There is a DeleteOrphan QueueWorker and the test coverage is in OgDeleteOrphansTest.

The main difference is that in this issue we would be cleaning up reference fields, while in the OG implementation it removes the referenced entities.

I like the idea of a plugin system, as in OG. More, I think the plugin should be configurable on field config level. Some fields might want to do a Simple cleanup, while others on Cron.

I also think that OG approach is terrific. But if we dont get there, I think its OK. We don't do any reference cleanup at all today. So even a slow and scalable implementation is an improvement (i.e. queue+cron), and would work on all sites.

This reference pruning will add a lot of new entity saves and thus a lot of new cache invalidation. Sites that are running hot will struggle even more. I think thats OK - referential integrity is more important. Just want to point out this consequence.

Moving this to a plan.

fwiw I think a queue would be fine, and we shouldn't make this configurable - if we want to add batch support we should add some kind of AJAX queue runner that can burn down queues.

I have a different perspective on this topic, and we implement the following approach on our customer projects: Never allow content entities from being deleted!

The reason behind this: with workflow and content moderation, we have a great tool to keep track on what happened with content, when, and by whom. However, when any content entity would get deleted, that trail would be lost as well. Which is not what people want. Content, which should "disappear" gets archived, i.e. unpublished and transitioned into an appropriate moderation state. With the correct configuration of permissions, such content appears as if it were deleted to most user roles.

Instead of trying to "improve" the data integrity when referenced entities get deleted, which will never be fully correct to everyone, I wish Drupal came with a mode, where the D in CRUD were not available.

#42 sounds like https://www.drupal.org/project/trash. @timmillwood did a lot of work on exploring the feasability of this kind of approach in core.

But that's a distraction from this issue: whether a referenced entity is archived or deleted, there's still the question of how to handle the outstanding reference.

Even though this issue is listed as the "Meta" issue, it's child #2978521: Remove orphaned (dangling) entity references when an entity is deleted has a more thorough summary, suggested alternatives, and also outlines scenarios where you _don't_ want automatic clean-up (so we need to allow both). So I wonder if we should close out this issue in favour of the 2978521???