[3.x] Add functionality to help merging / deleting duplicates

This module specifically does not address the cleanup. That's because it's a really hard problem to solve and dependent on your content structure and site setup.

The entity_usage module is pretty good, but doesn't cover all cases (like if you're embedding media through tokens). The workflow would be to find your references, clean them up, delete the old media items, and use the media duplicates module to track progress.

The purpose of this module is to help with detection and eventually to restrict the ability to create duplicates. We should also include some documentation about best practice for cleanup.

Perhaps we could implement this functionality as experimental submodule? So it could make progress while the key functionality is not affected? What do you think @nterbogt?

We'd be quite interested to help, as sooner or later we'll need this feature on client projects. @Christopher Riley it sounds like you could need the same? Would you also be interested to help or sponsor development?

What the submodule should be able to do is:
- Merge duplicates:

1. update entity references originally pointing on the duplicates to point on the selected representative media entity
2. delete the duplicates afterwards)

- Provide the UI to select duplicates and their representative media entity that should be kept and process the selected duplicates

The https://www.drupal.org/project/issues/auditfiles does similar things for Drupal files. We could have a look at the implementation and the UI / UX to not reinvent the wheel.

Things to consider:

• All entity reference fields in the project have to be identified, which point to media of the given type(s)
• If revisioning is enabled for the referencing entity, a new revision should be created (eventually as an option in the process)
• Identify & handle other kind of referenced media, e.g. Entity Embed - if enabled
• ...?

Perhaps we can find a way to be helpful in 80-90% of the cases and simply let the user know what the module won't cover.

Even marking media entities as duplicate in bulk, not to be used anymore or for manual resolution, could be a good starting point!

As written in #3356028: Offering to co-maintain Media duplicates this functionality should go into a submodule to separate this heavy part from the detection functionality.

We should first implement 3.x based on Custom Entity Type before starting that project, so we don't have to reengineer both.

Whao generally the merging seems to work, but I didn't yet test references etc.!

Left some comments, but this seems a good starting point. Still a lot of work though.

Should definitely have tests in the end.

And let's use "unpublish" as alternative to deletion for duplicates, to be a bit safer...

That's it!

IMPRESSIVE work @grevil. I left some minor comments on the MR, but I think this is good to go as experimental!

We should then try to get feedback from the community and test this on local copies of projects where we need the merging feature ourselves!

Whao, I didn't expect this to happen, really really helpful and a good basis for future extensions. Please create follow-ups. I'd say also add a follow-up to allow plugins instead of the hard-coded methods currently to handle references on duplicates. Architecturally that would make a lot of sense!

Thinking about this: Do we need if-clauses or dependencies on the references we currently check hard-coded? Entity Reference and Media Library Embeds? Could this break somehow if the modules are not installed?

Another good reason for using plugins in the future for any kind of references. :)
https://www.drupal.org/project/dynamic_entity_reference is another good example for a possible plugin. Yeah I like the plugin architecture idea a lot, just picking up the duplicate media entities and handling them!!

I'll create the follow up issue you mentioned. Plugins will definitly matter, once we support media reference sources. Currently the form contains too much "working" code anyway.

We won't have a problem with entity_reference, since it is baked in core directly and the media_embed filter is tied to the media module and we already have a dependency on media. But we need to conditionally check for the filter module, good call!

Ok, once the pipeline is green, I am going to merge this!