Plan for contributed modules with Media Entity API in core

Clarifying what the node ID is for those who don't have it memorized. Thanks @naveenvalecha!

Added two more modules to the list.

https://www.drupal.org/project/riddle_marketplace
https://www.drupal.org/project/media_entity_pinterest

Hi all, i boldly added #2879969: Make Media field mappings reusable between media types here as i feel this will be an issue and i might put some effort into it.
Please can somebody who knows the media issue space and agenda better tell what's a good place for it - thanks!

EDIT: Picked the wrong issue, moved to #2825215: Media initiative: Roadmap

SeanB just wrote this in the media meeting on this topic: We need to target media_entity / media_entity_document and media_entity_image as those will be the modules we actually will deprecate. Other modules will probably need to release new versions to adapt to the new API. I think that is mostly what is going on in this issue but more related issues would need to be opened for sure.

Just extending from #17 on possible scenarios, with the purpose of being explicit and having everyone on the same page :)
(I believe we are, but as they say... "better explicit than implicit")

I'm assuming all scenarios below are happening when Drupal 8.Y is released, where "Y" here is the version from which we will start saying "don't use media_entity contrib anymore because media is in core now". Based on https://www.drupal.org/node/2825215#followup-roadmap I "think" this will be 8.5, but TBH I'm not 100% sure about the consensus here. This would be also the point in time where we start officially considering media_entity (& co.) deprecated.

Scenario 1: A new site is built after 8.Y

- The site uses / wants to use:
-- media entities
-- files
-- images
-- "foo" media source (with stable 2.x branch)
-- "bar" media source (without stable 2.x branch)

This site is strongly encouraged to use the media-in-core path and contribute to port the source plugin "bar" during the site's development effort.

Scenario 2: An existing site uses media entities and needs to upgrade to 8.Y due to a security release

- The site uses:
-- media_entity
-- media_entity_document
-- media_entity_image
-- "foo" media entities (with stable 2.x branch)

This site will need to follow the upgrade paths:
#2880334: Add update path of media_entity config changes from 1.x to core media module
#2883449: Provide upgrade path for media_entity_document config changes to Drupal core
#2883452: Provide upgrade path for media_entity_image config changes to Drupal core
+ whatever the "foo" source plugin has defined in its upgrade path.

Scenario 3: An existing site uses media entities and needs to upgrade to 8.Y due to a security release

- The site uses:
-- media_entity
-- media_entity_document
-- media_entity_image
-- "foo" media entities (with stable 2.x branch)
-- "bar" media entities (without stable 2.x branch)

This site has then 2 options:

3.a) Stick with media_entity contrib
In this case the site will upgrade core to 8.Y but will continue using the whole set of solutions from contrib (i.e. media_entity 1.x). This example shows why it is important to ensure that whateaver is added to 8.Y, it shouldn't break existing sites that want/need to keep using contrib media_entity modules for some time.

3.b) Upgrade "on their own"
Many sites may be able to:
- have their teams to develop their own upgrade path to "bar" entities
- use an untested / non-stable patch from the "bar" issue queue
- even maybe drop entirely the "bar" entities from their site
so they can still fall into scenario 2 above and upgrade everything. IMHO this shouldn't be considered the rule though.

Does this make sense? Have I missed any other possible scenario?

Oh there is indeed another one:

Scenario 4: An existing site uses the D8 "media" module itself and needs to upgrade to 8.Y due to a security release

- The site uses:
-- The D8 version of media
-- media_entity
-- media_entity_document
-- media_entity_image
-- "foo" media entities (with stable 2.x branch)
-- "bar" media entities (without stable 2.x branch)

This site unfortunately won't be able to opt to something like 3.a, because in 8.Y core media will have taken the namespace of the media module. The only option this site has will be to uninstall the module and delete it from the codebase before upgrading to 8.Y.
This sounds a big deal, but should be mitigated by the following:
- There was never a stable release of media for D8 (only -dev), and its usage doesn't seem to indicate there was time enough for it to be adopted by many real sites
- Media in D8 was mainly a glue module that provided some default configuration, some CSS, JS and other tweaks. Once it relied always on the media_entity(_*) suite, all storage layer and configuration will still be compatible to the new media-in-core.
- Because of the point above, all upgrade paths indicated in Scenario 2 would be directly applicable here, and the additional work would be to:
-- effectively uninstall / remove the media module itself
-- follow all upgrade paths from Scenario 2 for the config renaming and API code changes
-- potentially apply back (in custom code or by some additional contrib module) the tweaks that the "glue code" implemented in the media module.

Scenario 3 is the most problematic obviously. There could be a conflict when we want people to use media instead of file and image (and we change things in core with this in mind) but people not being able to remove media_entity yet because of contrib modules not being ready.

This is something we should discuss some more. If we need to wait with the deprecation of media_entity until all contrib modules have an upgrade path, this significantly increases the amount of work. On the other hand we can't force people to upgrade when we know their site will break if they use certain modules.

Adding the media_entity issue as well. That should be on the top of this list :)

We should also decide which are the must have and the nice to haves on this list, since when you update media_entity to core, you will need to update all contrib modules as well!

Added new modules to the IS:

- brightcove
- media_entity_dailymotion
- media_entity_vimeo
- media_entity_carto
- media_entity_imgur
- media_entity_kaltura
- media_entity_d500px
- iptc_media
- media_entity_libsyn

Adding a link to the CR at the top of the summary.

Added dropzonejs to the IS once it provides an entity browser widget that relies on the 1.x API.

I think entity browser itself also needs to be added here. AFAIK, so far it was media_entity that provided entity_browser integration, that will need to be moved to entity_browser now.

Some small IS improvements, linking to some issues on contrib module queues.

In case it is helpful for prioritizing efforts, I (non-comprehensively) cross-referenced the list of modules in the issue summary with top Drupal 8 modules by usage and here's a "top 10" list based on that:

1. Devel - 25K sites
2. Video Embed Field - 20K sites
3. Inline Entity Form - 17K sites
4. Entity Browser - 15K sites
5. Media Entity - 14K sites
6. Embed - 13K sites
7. Entity Embed - 12K sites
8. Media Entity Image - 11K sites
9. Dropzonejs - 8K sites
10. Media Entity Document - 7K sites

After that, it starts to fall off more significantly; Media Entity Twitter is the next highest @ 4K sites. Given there are currently 161K Drupal 8 sites, we're talking about the remainder being effort expended for <= 2% of sites out there.

Fixing link to Slick Media.

Added Media Entity Browser to the IS

Audio Embed Field is no longer a sandbox.

Lightning Media Flickr is there twice.

IPTC media port to the proposed Media core module API

Adding a reference to the IEF issue (#2912010: Create a branch of IEF compatible with Media in core (>= 8.4.x)), even if there's nothing needed there right now.

After talking to @seanB and @phenaproxima, we agreed that the best is to have the media_entity upgrade (#2880334: Add update path of media_entity config changes from 1.x to core media module) deal with the config updates for images and files, so we can also cross off those two issues from the list.

Instagram landed! #2880122: Create a port of media_entity_instagram compatible with media in core API

Adding a link to #2912102: Generate core Media entities to the IS.

Updated issue - added #2870174: SoundCloud Port to the proposed Media core module API to list

Add related issue for Media entity Spotify.

Added #2912613: Tumblr Port to the proposed Media core module API to the list.

Twitter landed!

Added #2912728: Create a port of riddle_marketplace compatible with media in core API

Audio and Slideshow landed!

Added missing issue for media_entity_flickr.

Added #2908544: Support media module in the Drupal core (introduced in 8.4.x) for the Crop module

I found the list hard to read, but if this doesn't look good I'll put it back. Apologies if anyone hates it.

- Alphabetized.
- Updated some modules.
- Turned into a table.
- ✓ means I found a release available, - means branch is available but no release.

Added back the link to the bright cove issue.

The ordering was reflecting number of sites that are using each plugin AKA priority for porting. With the alphabetical ordering we lost this info. I'd return to the by priority sort or add number of usages of each plugin into the table.

Other than that I like the new look.

Crop landed! #2908544: Support media module in the Drupal core (introduced in 8.4.x)

Riddle #2912728: Create a port of riddle_marketplace compatible with media in core API and Pinterest #2905333: Pinterest Port to the proposed Media core module API landed

Added status numbers back in. Sorry.

Formatting.

Formatting again. Last one. Sorry.

@mattlt Thanks for helping out with that! The table looks definitely better than the list we had before.

Re: sorting, I believe the original sort order was lost at some point along the way when people (myself included) started adding new items to the end of the list. To make it easier, I have changed the "Priority" column to be "Reported installs" (or one approximation at least), so we don't need to bother with the list order, and still have a way of identifying the most likely used modules.

Entity Browser #2902324: Create a new branch compatible with the new Media in core API and DropzoneJs #2897255: Support Media core module (Drupal core 8.4.x-dev) landed.

#2912613: Tumblr Port to the proposed Media core module API landed!

Is anything planned for the File Entity module? Some users are running into issues with File Entity's entity/field definitions when upgrading to 8.4.0, so the question has come up as to whether these users could eventually migrate to core media. (Unfortunately I don't think it solves the immediate issue for those sites, but still worth knowing.)

Added issue for media_entity_video.

Released 8.x-2.0-alpha1 for media_entity_video.

I'll second the ask. Is anything planned for the File Entity module? I'm running into issues with File Entities entity/field definitions when upgrading to 8.4.0 and would like to set up an migration path to core Media. Or at least have a fix for File Entity so that we can upgrade to 8.4.0.

Re: #62 and #67:

In the contrib space sites would normally choose between the "File Entity" approach OR the "Media Entity" approach. So as a starting point, IMHO, I would say that the upgrade from a site using file entities to core Media shouldn't be included as part of the effort described in this issue (Media Entity upgrade).

Having that said, I don't see why a migration shouldn't be possible, and why potential issues that needed to be ironed out on that process couldn't be solved as part of #2835825: Add d8 media migration (or follow-ups).

As much as I'd like to be able to provide an upgrade path from file_entity, that will be so much more complicated than a media_entity => media migration, where the main concepts stay basically the same, we just renamed a bunch of things. File entity is an entirely different architecture.

From the top of my head, those are all things that would need to be done:

* Convert file types to media types
* Create replacement fields on media types that existed on different file types
* Create a media entity for all those files, migrate the data of the fields
* Convert file/image fields referencing those fields to an entity reference and update them to point to the media entity instead
* Also update widget/formatter configurations and everything related to that, e.g. someone might have used entity browers with upload widgets, file views and so on
* And then also lots of manual/code things, like templates

That said, quite a few of those things will also be needed (minus the file types and fields basically) if we ever want to offer a migration/update from old/plain core file/image, and as it wouldn't make sense to do those things twice, I believe a file_entity migration will only become something to look into once we figured out how to deal with the plain core file/image stuff.

Updated summary to reflect media entity browser now has a compatible 2.x branch

Media Entity Image might still need a 2.x version to get the EXIF support which core left out. Did any other things get left out? Discuss here: #2925712: Media Entity Image 8.x-2.x - core Media API roadmap & integration

RFC: I just submitted #2931878: Provide default configuration for a working media setup for 8.4.x core as a child issue of this. I can't believe I'm the only person trying to do that. ;) Am I missing something obvious? Is that the right place to put energy into trying to solve this interim need for a working default config that works with core 8.4.x? Thoughts? Thanks!

Mark Media Entity GoogleDocs as ready for Media core.

Mark Media Entity D500px as compatible with Media from core.

Updated to show Slick Media's 2.x branch supporting media in core.

Tumblr has a branch for media in core.

This is the correct tumblr, sorry.

media_entity_soundcloud opened a branch supporting media in core.

Crop should also support core media

Updating issue - video_embed_field has a 2.x release.

Linkit module also has a Patch to support Media core. Patch link: https://www.drupal.org/node/2930171

media_entity_soundcloud 2.x release has landed

The remaining modules have a relatively small amount of installs. I think there is no real need to keep this issue open. Thanks for the great work everyone!