Media initiative: Roadmap

Moving to active initiative as per #2786785-68: Media in Drupal 8 Initiative.

@Gábor Hojtsy and @slashrsm raised that many parts of this plan depend on the Media entity being stable. However, the summary says "even as an experimental modules if needed to begin with" so I could use some help on the specifics.

When someone has a chance, can we outline which parts depend on Media entity and subsequent APIs and UIs being stable, versus beta? The specifics of what that means are here: https://www.drupal.org/core/experimental#beta Additionally, stable core code cannot depend on beta core code (so for example, if we make additions to the Standard profile, then those additions are blocked on the APIs and UIs being stable).

What I imagine happening is introducing the entity type as beta, to indicate we won't make BC breaks unless they are critical. Then, followups can be filed for any requirements, and as the initiative progresses, other experimental code can also be built on top of it. When the first change to stable core is needed, that's when we need to make a decision about whether the entity type is fully stable or not.

By the time we get to 8.3.0-rc1 in March, we would have lots of time to complete followups for it to be stable. If changes to stable core are ready for 8.3.0-beta1, then we would make the decision earlier based on the needs of those stable changes. If they aren't ready for 8.3.0-beta1, then they will be targeted for 8.4.x anyway, but the Media entity itself could still be stable or rc or whatever was appropriate.

When all this pieces are ready change standard profile to ship with Media entity based solution by default. Don’t force existing sites to migrate. They can keep using files as we don’t remove any of that from core. We can provide optional migration path later, but don’t require it until D9.

Agreed. That part definitely depends on Media Entity and every other bit of code and UI being stable. I wouldn't imagine this happening before 8.4 at least.

Re-create features that core offers at the moment on top of it (local images and files).

This depends on how we re-create them. We still need to provide BC for those entity types for the duration of 8.x. Will they be updated to extend Media entity internally while providing full BC? Or will they simply be deprecated in favor of replacement Media\File and Media\Image entity types, which could be experimental initially until we make them the default for new installs?

Add one implementation of remote media (preferably YouTube - through oEmbed?) and support simple editorial workflows related to it and display configuration.

This could all be experimental until it's ready to be stable.

The Media Library also should be experimental at first, and does not require Media Entity to be stable. The embedding workflows will be more complicated, but it sounds like consensus is needed there still before we start prototyping that.

What is the dev/user experience process when/if the media module is experimental.

1. You install Drupal stable.
2. Create a few content types, create some content.
3. Figure out you need to embed youtube videos or want to reuse your files.
4. Install media module.
5. Now you have your content types with your file/image fields and another way to have media on the site.
6. Win? (Nope).

One way for media to be experimental and not be a usability nightmare to try any site other than a totally empty one is to remove the creation of default file/image fields from all entities in 8.3.0 so users would need to create either those or use media. I don't think that is what we want? Still then, @berdir's concerns from #2831274-139: Bring Media entity module to core as Media module would apply (in short it does not make any contrib's life easier to have an experimental media module in 8.3.0, its better not to have anything for media then):

I already mentioned this in IRC and it has been discussed a bit, so here is the official comment to propose my idea:

First, note that media_entity/media.module is very different for pretty much anything (the closest is moderation/workflow) else we put in core as experimental. bigpipe is a perfect use case for an experimental module. As it now official declares it has no API, it has no data.
You can enable it and disable it again, any time you want.

Media is different. It contains data but just as important, it *is* an API for a large ecosystem of contrib and custom modules that integrate with it. I think I can speak for all the media related/depending projects and distributions (like Thunder and Lightning) maintainers
that having experimental media module in core would bring us nothing and it would be impossible (mostly due to the amount of time it would require) to maintain modules that work with both media in core and media_entity in contrib, and as a result it would also be of extremely limited usefulness for users.

And while we can argue that moderation/workflow data can be removed without losing too much (your content is still there and still correctly published/not published), that is not the case for Media. You can't just uninstall it because we have no upgrade path in
8.4.x, all your media and metadata for it would be gone.

This proposal based on @catch's comment in https://www.drupal.org/node/2789315#comment-11762228, where he confirmed that anything new that has been added to 8.3.x is "experimential" until we release 8.3.0-beta1.

That means if we commit this issue now, then we still have time until then to improve and stabilize it to reach a MVP. What we need is a set of goals that we need to achieve until then. Both processes (like code/UX/testing quality) but also an actual feature set.

If we do not achieve that in time (say, a certain amount of days before beta1), we remove it again and try again for for 8.4.x and stick to media_entity in contrib. Nobody of us wants that, but IMHO it is better than having something experimental in core that nobody can use.

Basically, this is exactly what Dries proposed in his keynote in Barcelona with Feature branches, we just don't have them yet. But since (almost?) everything we do will be contained in one module, it should be relatively easy to remove again if we have to.

@Gábor Hojtsy, I agree that it the complete user experience should involve things being provided on installation, which requires everything to be stable. However, I feel that Berdir's comments undervalue that experimental modules are expected to become stable within one or two minor releases. Adding something to the Standard profile once it's complete is not mutually exclusive as introducing it experimentally at first. We definitely intend to use experimental modules to develop features that will eventually ship as the default for new installations.

If we add the API now, in November, we have until March to make sure that it is indeed ready to be stable for contrib and mark it as such. I do understand that the contrib ecosystem's dependency on the Media Entity is really important, so we should have releasing it as stable in 8.3.0 as our goal. The decision to do so will be based on consensus between core maintainers and media maintainers.

We would not add it to Standard profile until all the other features in the feature set were also ready to be stable, including the new file and image entity types, the library, a really polished UI, embedding, etc. For that change to be in 8.3.x, all of those things would have to be ready in February. I'd be really, truly amazed at such progress. But that would mean all of those things would need to be done and fully polished. I think the 8.4.x beta in August is a better target for really polishing all the different parts of the implementation, and therefore we would not be changing Standard profile in 8.3.x.

For the future work the initiative intends and for the contrib ecosystem, I really don't think there is a big difference in calling the module stable now, versus marking it beta now and aiming to have it stable by release. However, it does make a big difference for release management, for managing followup issues, and for what we expect of the initial patch.

Tagging for framework manager review given that this "represents a significant change in architecture or public APIs". There is little point (or according to @berdir several negatives) if only parts of this happens and we are to keep using file/image fields as-is for example. So the resulting system will need to make significant changes. Implementation issues for this include but not limited to #2831936: Add "File" MediaSource plugin, #2831937: Add "Image" MediaSource plugin to reproduce file/image wrapping functionality, #2831940: Create file field widget on top of media entity and #2831941: [PP-1] Re-create Image field widget on top of media entity to support widgets for them and #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode to support formatters on them. (These issues are sub-issues under each section above). Once those are in (if all components are stable), standard profile would come with media fields instead of file and image fields, and file and image would be deprecated. Which is a pretty significant change to say the least.

I believe the media team understood this review is not needed because the ideas review meeting approved the plan but according to @xjm that is not the case, so we need that review. Hopefully that does not invalidate the steps and directions identified. #fingerscrossed

Let's do this, to make it clear it is an agreed plan that just needs committer feedback and signoff if appropriate.

The media entity being in beta doesn't mean that we'd break people's data, in fact it means we specifically wouldn't. However it does mean some other things:

- we might make such a change in 8.3.x itself before beta/release candidate, I don't think we'd even do that if it goes straight in as stable except reverting or something.
- something like theme hooks/formatters or similar API but not data changes could be done with bc breaks if necessary
- it gives us leeway in the case of unknown unknowns that would add to the critical bugs queue

For comparison, the initial patch for content_moderation module wasn't possible to uninstall. The contrib workbench_moderation roadmap had it going to 1.0 with that architectural issue. Fortunately we spotted that before it went in, but if it had gone in as stable without being spotted we'd be pulling our hair out now.

fwiw I'm 100% on board with deprecating the image + file fields, having those created in the standard profile instead, changing the migration from 6.x and 7.x so it migrates into media fields, and also providing an 8-8 migration from image/file to media so that existing Drupal 8 sites can make the change too.

It'll be incredibly confusing having both in core, so we should aim to do a full replacement.

Should also add we are already trying to make data structure changes to core entity types that are stable, by making them revisionable, however that's extremely difficult to do.

fwiw I'm 100% on board with deprecating the image + file fields, having those created in the standard profile instead, changing the migration from 6.x and 7.x so it migrates into media fields, and also providing an 8-8 migration from image/file to media so that existing Drupal 8 sites can make the change too.

It'll be incredibly confusing having both in core, so we should aim to do a full replacement.

Should also add we are already trying to make data structure changes to core entity types that are stable, by making them revisionable, however that's extremely difficult to do.

So the plan was/is to provide new media fields on new sites and leave existing sites alone until a well tested/proven migration path is implemented. The migration path was/is not targeted for inclusion in 8.3.0 as we need to focus on building the thing that people would migrate over to. In some ways this plan is similar to how Drupal 8 was released by a migration path itself, targeted at new site builds. That said, we need to implement the migration path to deprecate for at least de-emphasize file/image fields because otherwise the adoption of media may be hard.

So the plan was/is to provide new media fields on new sites and leave existing sites alone until a well tested/proven migration path is implemented. The migration path was/is not targeted for inclusion in 8.3.0

I don't mean for 8.3.0, I just mean in general that should be the aim.

Moving to an active initiative to reflect reality. Not moving any of the signoff tags, especially given other active initatives like outside-in have the signoff tags on them too.

@effulgentsia pointed out that depending on how much we're able to get stable for 8.3.x, just the media entity in core could be confusing for new users if they discover and enable it, without any of the contrib modules.

I might have a compromise to keep the module hidden = true even if it's stable - then it can be shown when the contrib modules are installed that need it. Once there's enough in core to make it viable without any contrib, we then just remove the hidden = true.

Janez (slashrsm), would you like to be the initiative lead for this? I think you'd be a great initiative lead but I wanted to formally ask instead of assume. :-)

That sounds interesting and technically feasible I guess. I wouldn't be in the marketing people's place in that case for sure:

NEW! Drupal 8.3 core now has media capabilities!! Erm, how? Well, download this contrib module, its cool. Huh?

@Gabor it'd be similar to adding it as an entity type in \Drupal\Core, I think it's OK to say we added an API (+ UI scaffolding), use contrib until those bits have migrated to core too. Or just don't say anything but it's better than it getting stuck beta in core for six months which no-one wants, or staying in contrib for another 6 months (and having to negotiate the patch landing early in 8.4.x vs. the contrib module).

@catch: I agree its better and I agree its similar to adding an API.

Even though this does not have the product manager review tag, wanted to share that we discussed this issue on this week's Ideas Queue Review.

From a product/UX POV, we are comfortable with the list of issues outlined in the summary, with the exception that we can't call this MVP "done" until/unless the user-facing components are in there as well. So while the framework/release manager call to split it between 8.3 and 8.4 makes sense, we would not want to expose the media entity to end users until/unless the user-facing components are done as well.

What stops us from removing "Needs release manager review" and "Needs framework manager review"?

Should this be moved to fixed or it being an active initiative as RTBC is how we signify an active initiative?

Opened #2843399: Add media initiative to MAINTAINERS.txt for adding the maintainers.txt entry.

@Gábor Hojtsy, you are the one who moved it to active initiative (twice actually); that's how it ended up marked as an active initiative before we signed off and how the review process got a bit mixed up. In the future I think committers should always do that so there's a clear indication on the issue that we are on the same page. This is all a minor process aside though.

@effulgentsia is going to provide framework manager review later today hopefully. There are outstanding release management issues that need to be resolved and so that tag will need to remain on as we address them. I will add a proposal once this has framework signoff and as the initial media entity issue continues to receive framework manager review. My top priority is providing a smooth transition for contrib without compromising the core product. In the meanwhile I'd encourage reviewers and contributors to continue to work on the first three steps of #2801277: [META] Support remote media assets as first-class citizens by adding Media Entity module in core and try to get #2831274: Bring Media entity module to core as Media module to RTBC.

I agree with everything in #21. Thanks @slashrsm.

Oh, and thanks @webchick for #22; I agree also.

Updated the issue summary with:

• Fleshed out the Media Entity section.
• Removed "Open core issues" links that were already linked above, and moved the rest to the embed/wysiwyg section.
• Tweaked formatting: <h5> to <h4>.

This functionality will be the MVP for Media in Drupal 8 core. The objective of this MVP is to provide a simple media solution to make Drupal 8 easy to use out of the box for basic use cases.

From a framework management perspective, I'm very happy with the plan in this issue's summary (my changes in #27 were just me detailing out the plan as I understand it, incorporating information from other issues, and not making any intentional changes to what the Media team has been planning), so removing the "Needs framework manager review" tag. I checked with @alexpott and @catch and they had no objections to me doing that. I think all of the steps and issues outlined in this plan are architecturally sound, and I agree that each one is "Essential" (per the issue title), and that the combination of all of them makes for a nice MVP of core media handling. I think some of the "Extras" in #2786785: Media in Drupal 8 Initiative might also be appropriate to include into core and some might not be, so this issue is just the "first round of improvements", and I'd be happy for the "Media initiative" to continue beyond this first round so long as people are available and motivated to work on it.

This MVP can ideally be achieved in a short time -- to be included in Drupal 8.3 RC1 (even as an experimental modules if needed to begin with).

I'm not a release manager, so leaving the "Needs release manager review" tag for someone who is. However, I don't see a path for all of the issues here to land into 8.3, and especially for all of the "Media entity type in core for 8.3.0" ones to land in a non-experimental state. I'd love to be proven wrong on that, but that's my current feeling.

So while the framework/release manager call to split it between 8.3 and 8.4 makes sense, we would not want to expose the media entity to end users until/unless the user-facing components are done as well.

Because I feel that there will be such a split as to what makes it into 8.3 and what doesn't, we need to be very careful to make sure that each step along the way leaves Drupal 8.3 in a shippable state. We can discuss the details of that as we get to each step, and I'll start with a more detailed comment soon on #2831274: Bring Media entity module to core as Media module with thoughts as to how we can make that one shippable on its own.

Opened #2843399: Add media initiative to MAINTAINERS.txt for adding the maintainers.txt entry.

Yay! Thank you, @slashrsm, and everyone who's been doing really fabulous work on all of the issues.

@xjm:

@Gábor Hojtsy, you are the one who moved it to active initiative (twice actually); that's how it ended up marked as an active initiative before we signed off and how the review process got a bit mixed up. In the future I think committers should always do that so there's a clear indication on the issue that we are on the same page. This is all a minor process aside though.

Right, sorry for that, I realize it was a mistake. I thought that a lot of other conditions practically made this an active initiative (eg. Dries blogging about it with all the steps outlined that are here, paying three of his employees to attend a sprint to implement this plan, etc). Once I learned that those are not good indications of a plan being approved, I immediately stopped recruiting people to work on something that may be futile to work on. Now that we have a plan approved I think we may be able to have something nice in Drupal 8.4.

@slashrsm: how does your plan (a) deal with field widgets? Would they be in a separate contributed module then that is created just for the sake of complementing Drupal 8.3? Also would that be the same contributed module that would hold a media library or the media library would be a separate contributed module?

@slashrsm: sure, I don't think you answered the question though :) If we have media entity + file + image handlers in 8.3.x then we still need a way for sites to have the widgets. I believe in 8.x contrib so far widgets/formatters are bundled with he handlers, no? For 8.3.x we'd need a separated approach somehow for them to live in a dedicated contrib module, right? (I may be missing something sorry).

@slashrsm: ok that makes sense. Then if there is a contrib version of the media library, then that will need to house those. But to avoid the mess with migrating a contrib module into core, it is probably simpler to just do the development in core direct (in a fork/sandbox for testing).

Thanks @slashrsm and @Gábor Hojtsy for the feedback. I agree with @slashrsm that the media entity, file, and image patches are what we should be targeting for the core API. (@effulgentsia said he did not think image needed to be, but in either case it's a small addition once the file media patch is complete. The main entity patch is certainly the biggest investment.)

I definitely agree that the timeline is now most likely too short to present a complete out-of-the-box user experience for media in core in 8.3.0. Even if we provide a minimally viable/"not super broken" user experience for the core modules, they're going to be very disappointing without further UI work or a feature like the media browser. There are so many integration points we would have to get right that are out of scope for the initial patch. It actually could hurt perception of the product; it would distract from the fact that there is a rich suite of media solutions in contrib and make it look like D8 as a whole is not usable yet.

All this is why I strongly encourage marking the module as hidden, even if it is in the release of 8.3.0. If we do that, we can treat it as an API addition, and give ourselves more time to get the UX MVP, documentation, etc. right. We still should make the UX minimally viable when the module is hidden and/or experimental, because it will still be exposed to and used by everyone who installs the contrib modules. But if the module is not hidden and/or experimental, then we must make it polished.

@slashrsm's two approaches in #29 are actually fairly similar at this point. The goal in both cases is to get an architecturally complete, finalized API into core as soon as possible, so that both core and contrib development have a shared, stable foundation. Once the code freeze for alpha starts, the target branch becomes 8.4.x instead of 8.3.x. However, it's worth noting we recently clarified our policy about what can be committed during these phases:

Core developers should plan to complete changes that are only allowed in minor releases prior to the alpha release for inclusion in that minor. Following the alpha release, such changes will be committed primarily to the next development branch instead.

There are no specific restrictions on what changes are allowed in alpha releases beyond the allowed changes for minors, but additions and disruptive changes are backported to the alpha only at committer discretion, with criticals, strategic initiatives, and product management priorities as the first focus.

(From https://www.drupal.org/core/d8-allowed-changes#alpha)

If review concerns, architecture, etc. are addressed and all three patches is committed to 8.4.x between alpha and beta, and if they are marked hidden, then at that time we can evaluate whether to backport them to 8.3.x together. The question is what will be best for core, contrib, and site owners. There are pros and cons for backporting. For example:

1. Pro: If the APIs are in 8.3.0, contrib modules can also have stable branches depending on them after April 5.
2. Pro: Contrib will not have to maintain two versions from February until 8.4.0 in October, only for the two months between February and April.
3. Pro: Once the File/Image types are added, site owners who also install contrib media modules can begin creating their data with the new data model, so there will be less data overall that needs to be migrated from the core type.
4. Pro: The investment we have made in Media so far would be proven, and the core product would be committed (pun!) to it, as @slashrsm expressed earlier, because we shipped it in a core release. This could also help incentivize further resources for the initiative (which @slashrsm also raised above as a need).
5. Pro: No more rerolls of a 200K patch or managing sandbox merges. :P
6. Con: As Gábor pointed out, "we added something you can't see to core" does not have much marketing value on its own.
7. Con: Gábor also raised a concern that it might mean there would not be enough advantage to justify the work contrib would need to do to port to the new API.
8. Con: If we do get something wrong in the API, we have to provide BC for it anyway.
9. Con: Even if the API is in 8.3.x before beta, other API additions on top of it can still only go into 8.4.x, and so 8.3.x and 8.4.x could diverge, which means potential added work for backporting bugfixes, etc.
10. Con: There is more risk for core's minor release, because we'll be using the absolute minimum allowed window for flushing out bugs.
11. Etc.

That list is not exhaustive, and it will be hard to guess what will be best until the first patches are actually committed to the dev branch, whether it's 8.3.x or 8.4.x at the time those commits happen. And it also becomes irrelevant if the patch is not really, really ready. So my recommendation is that we continue to really focus on the API side, and if the API is ready before beta, then have a meeting with initiative members, contrib maintainers, and core committers to decide together whether to backport it to 8.3.x as a hidden module. I'm going to keep this tagged for release manager review until we have a chance to evaluate the status of the initial commits and have that discussion. Until then, though, let's focus on getting the initial API right, while planning to iterate on the user-facing core featureset in the longer term.

Edit: grammar.

Given the severe architectural concerns raised by @WimLeers 3 days ago and the resulting rewrites he is heavily working on still resulted in mostly an entire reorganization and rewrite of all corners of the media entity patch, I don't think the argument of the proven architecture relied on by a dozen contributed modulesholds anymore. I think it would be dangerous to try to get a stable module in core which just got rearchitected from the ground up a couple weeks before a deadline, because there is no way we can prove with the other dozen modules in this timeframe that the rewrite is stable enough.

I think the only way forward then is to target this to 8.4 exclusively. I still think it would be nice to settle the base entity module sooner than later because then work can commence on the plugins and widgets and formatter and library, but then there is no deadline attached anytime soon.

I also think then we need to mark this issue Needs work, but since I made several mistakes in how this issue was marked, I am not going to change status again.

I was just informed that before I came in it was mostly cosmetic changes, but also fixing incorrect usage of the plugin system etc, so not only cosmetic.

I'd almost suggest adding the media_entity module as-is to Drupal core, with only adding test coverage and improving documentation. Without renaming anything. Without any other changes. Because then we would be actually putting a stable module into core, without changing any API.

But apparently it's going to need fixes anyway, so the API is going to be changing anyway.

Yeah I think contrib needs updating for the version of the proposed media module anyway, so doing the update twice is not a good plan to work with (to update to proposed changes API again). So its better to update once.

The architecture is NOT bad. I have serious concerns about a SUBSET.

See https://www.drupal.org/node/2831274#comment-11878883 for more nuance.

On #2835825: Add d8 media migration @xjm posted this release management feedback which IMHO concerns the overall approach to requirements and how/when things land, so posting a response here.

@xjm's comment there

If Media does not have a migration destination and becomes stable, then that moves Migrate Drupal further from being stable. We should be requiring migration destinations for all added storages at this point. They will also help for migrating core image and file data into media entities once they are stable. So, I think we need to have the migration destinations available for Media to be stable (a "must" have on the initiative roadmap).

However, migrations are not a blocker for adding the API to core so long as the module is still in beta or earlier. For media, we agreed that the module itself could considered a stable API addition but experimental otherwise, so long as the module remains marked hidden. The same should apply here. So we can postpone this issue on finalizing the initial API and on finalizing the file and image types (although we may consider incling the migation destinations for files and images in those patches). Marking postponed on the initial entity issue, at least.

My response

Trying to process your comment as to what options are there for Media module vs. this media migration implementation (and possibly other things). Maybe there are these two options if I understand it right.

1. Get media entity committed as stable (in 8.4.x) without migration integration BUT media entity marked as hidden. Implementing migration integration would then be part of the must haves to mark it non-hidden.

2. Get media entity committed as experimental (in 8.4.x) without migration integration. Implementing migration integration would then be part of the must haves to mark it stable (still possible before 8.4.0(?)).

For option 2, I looked at https://www.drupal.org/core/experimental and it says

Experimental modules may only become stable modules in minor or major releases.

which in my reading does not rule out adding an experimental module and making it stable within the same dev cycle before a new minor is released. It only rules out making an experimental stable in a patch release, which is not an option the media team wants anyway.

The option 1 case is a brand new way to add an API module, at least I don't remember we did this before, so we may need to document why and if any policies need to be adjusted for this process, so people understand what we are doing. The option 2 case at least maps to an existing process, but in the option 2 case though, we need to keep the "early" rollback scenario in mind if the module does not reach stable by the minor release.

IMHO the hidden module idea was useful when we were talking about getting this into 8.3 as stable without useful user features. Not sure it is useful as part of the development of 8.4.0-dev since we DO want people to try it out and use it while 8.4.0 is being developed as opposed to hiding it from developers.

Thanks @Gábor Hojtsy.

Personally, I would still prefer to call it beta experimental when we commit it, and just clarify in our communications, the experimental handbook page, etc. that the API and data model are stable, but other aspects are not. But @slashrsm and @Berdir had concerns that that would not be a strong enough stance for contributed modules that need to depend on it and have stable releases. That's why the hidden module workaround is the next best thing.

I guess we could have a "Must" have issue to mark the module as hidden if it's not stable in all aspects by 8.4.0-beta1. The risk there is that 8.4.x is not shippable if we need to cut an off-schedule minor for some reason, so in general I don't want to add required issues like that.

so we may need to document why and if any policies need to be adjusted for this process, so people understand what we are doing

This plan is a special case because of the unique needs of media contrib modules, so I'd prefer not to make any overall policy changes. After all, we decided the policies to meet our needs and it's documented that we can make exceptions to them also when it also meets our needs. :)

which in my reading does not rule out adding an experimental module and making it stable within the same dev cycle before a new minor is released. It only rules out making an experimental stable in a patch release, which is not an option the media team wants anyway.

Definitely. This is actually what I wanted to do initially in 8.3.x anyway: Add it as beta experimental in November, and mark it stable by release if at all possible. We can still do that for 8.4.x.

So what I suggest is:

1. Commit the initial API to 8.4.x-dev as a beta experimental module (in the core experimental package), but not hidden, and communicate to all contributed modules that we will provide full BC and upgrade paths for the API and data starting sometime before 8.4.0.
2. If the module is ready to be stable by 8.4.0-alpha1 (per the stabilization requirements we have defined already, must-haves completed etc.), then we just make it stable the normal way.
3. If the module is not ready to be stable by 8.4.0-alpha, instead switch it to hidden: true, but still move it out of the experimental package at that time.
4. If the module gets to stable between 8.4.0-alpha1 and 8.4.0, simply remove the hidden: true at that time.

BTW the migrations are just one of several must-haves we will have as followups to the media entity patch. See #2795833: [plan] Add layouts to entity displays (both form and view) for an example of a followup roadmap. I don't want to take up initiative contributors' time sorting through these things while we are still finalizing the architecture, so I am gathering notes as I do my own review of the initial patch and will try to help update the roadmap here and in #2801277: [META] Support remote media assets as first-class citizens by adding Media Entity module in core once I'm done. (It will have the categorization of those followups into Must--things like UX gate issues, Angie's product manager requirements in #22, my recommendation about migrations, etc.--Should, Could, and the steps I've outlined here if that plan works for initiative contributors and contrib.) Tagging for that summary update once I get to it.

The bottom line is that even dev branches need to be in a state where we could release them today if we needed to, but we can still make things smooth and reliable for contrib.

Edits: clarifications and not contradicting myself (I hope).

That makes sense to me IMHO, it gives us a way out if we do need to release a minor out of schedule (which I don't see in which case would be necessary but that does not matter here and does not change anything :).

Updated version numbers to 8.4.0 from 8.3.0.

And two more places I missed.

So despite the plan I posted in #42 and slashrsm agreeing to it, there's still feedback on the main issue pushing for it to be non-experimental and non-hidden. The module must either be experimental or be marked hidden to be committed so that we don't add release blockers. Let's please just stick with that as it is simple enough to expose the module when enabling it on its own (rather than indirectly through a dependent module) provides a stable user experience.

I think that shown in the beta experimental package would be more correct (and give more users the opportunity to test, as has been raised). It will then be up to the media team to either submit a patch to move the module out of experimental once the followups we agree on are complete, or to submit a patch that switches it from stable to hidden if the followups are not complete and they prefer a stable API for contrib over an experimental one. That way 8.4.x is never in an unshippable state, but the followups for the worst-case scenario are still small.

I took a stab at determining what the roadmap for Essentials should look like. At @xjm's request, I have divided the goals into three tiers of importance. This is a draft, essentially, so take words like "must" and "should" with a grain of salt. All the listed goals are in descending order of priority.

Must-have

All of these must be in core for 8.4.0, or Media cannot be considered stable.

1. #2831274: Bring Media entity module to core as Media module. Nothing else has any point without this. Getting it committed is top priority. It should be visible, and beta experimental, meaning the API will be stable.
2. #2831936: Add "File" MediaSource plugin. This blocks support for images. Must have parity with core File fields.
3. #2831937: Add "Image" MediaSource plugin. Images are the most common and critical use case. Must have parity with core Image fields.
4. #2831941: [PP-1] Re-create Image field widget on top of media entity. Image widgets are critical and must come before File widgets. Must have parity with core Image widgets.
5. #2831940: Create file field widget on top of media entity. Must have parity with core File widgets.
6. #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode. Should probably be split into two issues, one for an Image formatter and one for a File formatter. Both must have parity with core File and Image formatters. The Image formatters are the higher priority.

Should-have

Once the must-haves are complete, these should be attainable by 8.5.0. They add enormous value to Media, but do not block stability in 8.4.0. However, they must all be done before the Essentials phase of the initiative can be considered complete, and should be targeted for 8.5.0.

1. Expose Media as a stable, non-experimental module in core. Enable it out of the box in the Standard profile, which should now include File and Image media types as default configuration.
2. Provide a stable migration path from core File and Image fields to media reference fields.
3. Deprecate core File and Image modules in favor of Media and the types included in Standard. Discourage the use of File and Image in the UI, but otherwise leave them untouched.
4. #2834729: [META] Roadmap to stabilize Media Library: Media Library as an experimental, non-hidden module.
5. #2831944: Implement media source plugin for remote video via oEmbed. A new media source, included with core Media, that supports remote assets via oEmbed, plus a media type in Standard that uses this source.

Could-have

Once the should-haves are complete, these should ideally be attainable by 8.5.0, at least experimental stability. If they don't land in 8.5.0, that's no problem. Should be targeted (tentatively) for 8.6.0.

1. An input filter, or some other strategy, for embedding media items into CKEditor. Possibly adapting some code from Entity Embed. Discussion is at #2801307: [META] Support WYSIWYG embedding of media entities.
2. Media Library as a stable core module, included with Standard.

Thanks @phenaproxima! That looks like a great start to me. I think I agree in all those cases and will iterate on that.

One correction here:

Nothing else has any point without this. Getting it committed is top priority. It should be hidden, but NOT experimental.

What we agreed on is to start off with it visible and beta experimental (which still means a stable API and upgrade path) so that people can actually find it to test with it between now and 8.4.0. (It already confused people at DDD who even knew what was going on trying to update contrib.) Then if the rest of the "must" haves are not completed in time for 8.4.0-beta1, we switch it to non-experimental, but hidden, so that stable contrib modules can also depend on it for 8.4.0.

Thanks, @xjm. I have updated my comment. Sorry for the confusion!

Adding a stub to the summary to discuss with @phenaproxima etc.

@phenaproxima and I sorted all the followup issues from the summary of the initial issue to this issue.

Removing #2862453: Make "update any media" a restricted permission, or remove the flag from "delete any media" from the plan as this is worked as designed and not to be the must have. Refer #2831274-389: Bring Media entity module to core as Media module

//Naveen

Added #2775131: Media entities should support contextual links to the list of "could-have" (lower priority) issues.

Re #49: Some of this seems to conflict with the work going on in #2113931: File Field design update: Upload field.. Is there some way these efforts could be combined? Could that work be incorporated over here? It would be a shame to throw it away.

@colan, Thanks for bringing up #2113931: File Field design update: Upload field.! We definitely should take into account the design efforts there when we expose Media in the UI eventually.

Since #2113931: File Field design update: Upload field. is an improvement to existing stable core features, it can take precedence. Media might not be shown through the UI in 8.4.x, so appropriately redesigning its entity reference widgets could potentially be something added in 8.5.x. I've added ensuring design parity with any results of that to the roadmap.

Oh, and on #58, agreed on contextual links support as a could-have. Node, Block, Block Content, and Taxonomy have them, but File and Image do not, and contextual links are also fraught (we know from past issues) because they are often more destructive than the user expects or for a different element than the user understands. The contrib issue also mentions Quick Edit support; does it implement both?

The contrib issue also mentions Quick Edit support; does it implement both?

From the testing performed during the sprint day at DrupalCon, I think so, yes.

A quick review during our product management review. This has been rtbc for a while now. Does this need more fleshing out or can we consider this ready for what we want to achieve in 8.4?

Added #2877378: Add token replacements for Media to the list of must-haves, as per #390 in #2831274: Bring Media entity module to core as Media module.

Added #2877400: Find a way to make title handling generic in entity templates and #2877397: Determine when and if media items should validate themselves to the IS.

Adding the rest of the follow-ups filed in #2877346: Media Entity finalish review (disregard if you're not part of the Media Initiative; working around long issue node) to the IS.

Refreshing linked issue statii in the IS.

Refreshing linked issue statii in the IS.

If only d.o ran on D8 so it had cache tags… :) Instantaneous updates without these hacks!

Hi all, i boldly added #2879969: Make Media field mappings reusable between media types here as i feel this will be an issue and i might put some effort into it.
Please can somebody who knows the media issue space and agenda better tell what's a good place for it - thanks!
EDIT: Also close if it's already in a related issue - id did not see it.

Adding #2883815: Move Media Entity configuration from media module to standard profile to the IS, since this is part of our stated goal to move Media configuration into Standard.

Swapped in #2883813: Move File/Image media type into Standard once Media is stable to replace #2883815: Move Media Entity configuration from media module to standard profile, since it's more complete.

Thanks @axel.rutz, I put that issue in as a "Could have" for now.

I think we have a few more followups from the meeting yesterday to add here; I lost track of which though. :(

Also, fixing the disconnect between the status and component. The issue was RTBC as a plan for release manager review, but @Gábor Hojtsy sort of bypassed that. :P

I had not signed off yet at the time, but after discussing the plan in detail with @phenaproxima at Baltimore and following the ongoing progress, I think we're definitely on the right track, so I am signing off on the plan now. :) We will of course continuously update the plan as we have new information, and we'll definitely need to revisit things when we start looking at unhiding the module and what the implications are for core, but so far so good! Well done.

I have opened #2885002: Provide an optional migration path for sites using files/images to media entities but wasn't sure where to place it in the roadmap (must/should/could), or if it should even be part of it.

Adding #2885486: Media entity is revisionable but doesn't have a revision link template as a should-have follow-up, since #2856363: Path alias changes for draft revisions immediately leak into live site may break Media's path aliasing if there are forward revisions.

Adding #2885729: Smooth out timezone handling when retrieving date/time info from EXIF to the IS as a should-have, spun off from #2831937: Add "Image" MediaSource plugin.

Added #2888138: Content Moderation should be tested with Media entity as a should-have.

Whoops, had added the wrong issue.

Added #2889855: Unpublished media entity can not be accessed by owner and update any media/delete any media access possibly cached by user and #2889438: Delete media delivered icons on uninstall to the roadmap in the IS.

Trimmed a couple of duplicate issues from the IS.

Added #2894270: Users unable to add an extension to the file upload field and #2894271: Users unable to change a media source file/image from public to private. Removed #2862454: Improve label of 'full' viewmode.

With the new approach/scope in #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode, @webchick reviewed it and recommended that it be a must-have before we show the module in core, but that it's okay to have the issue outstanding while the module is hidden. Updating the roadmap accordingly (and removing an outdated issue for JS tests).

Reordering so that the image cart is not before the file horse. For some reason the IS had it the other way around? But image widget is postponed and file is not.

Per @Berdir, #2877378: Add token replacements for Media is not actually a hard requirement for pathauto support, so we can move that way down the roadmap since it's not going to be disruptive for contrib at least. I've put it under "Should" for now, but it might be closed or moved to "Could".

I've also moved #2894271: Users unable to change a media source file/image from public to private to "Should" following discussion with @seanB. It might get fixed anyway as a side effect of fixing #2894270: Users unable to add an extension to the file upload field which is still a must-have as it has no workaround and not being able to configure the allowed file extensions is kind of a serious problem. :)

We've discussed also whether #2831940: Create file field widget on top of media entity is actually a hard requirement before 8.4.0, since the Media Library will provide widgets to select Media from the library. I'm hesitant to move that requirement, though, because I could not figure out how to upload a file without the widget. :P Furthermore, as it currently stands, you have to have to be able to access admin/content to upload a file or image, because the only place it's linked is on admin/content -> Media tab -> "Add media" button. (That might be an issue on its own...)

#2895001: Use the bundle label (e.g. "Media type") instead of "Bundles" in the Entity Reference field configuration came up in a usability review of a different issue, so adding under "Should".

@catch and I agreed earlier tonight that we can (if necessary) ship without the widgets for 8.4.0. I'd prefer to have them since Media file/image fields are basically unusable without them, and we want to give experienced site builders (who can figure out how to turn a hidden module on and aren't daunted by the all the sitebuilder UX bugs) the chance to start storing everything as media so that there's less to migrate later. There's also @Gábor Hojtsy's outstanding question of whether we should not add any widgets other than the theoretical eventual widgets that Media Library will theoretically provide, but I'll bring that up on #2831940: Create file field widget on top of media entity.

#2894270: Users unable to add an extension to the file upload field is still a must-have because the solution is probably to stop locking the fields which is a disruptive change (and the bug itself is pretty bad).

Reodering the list again based on that.

Let's define locked fields and solve more then one bagofeature #2289551: Clarify what 'locked' means for a config entity and whether it's okay for code to rely on a locked config entity existing

Added #2895059: Move media module out of Core (Experimental) package and into the Core package, but mark it hidden since there was no issue for it yet.

@xjm++
@xjm++
@xjm++

Thank you so much for bringing much-needed clarity what are must-haves for Media to become stable!

Since #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode is now out of the critical path (and is about a general Entity/Field UI usability problem), I created #2895382: Hide label, thumbnail, etc. for default display of media file and image references for the "bad defaults" aspect of it. I think we should consider marking that a "must-have".

Can we please add #2835767: Media + REST: comprehensive test coverage for Media + MediaType entity types to the list of issues that must happen before this module can achieve stability, i.e. as a must-have? It's important to have that test coverage issue committed, to ensure the API-First support of Media does not change in the future, which would cause BC breaks.

Just discussed this with Wim Leers. I totally agree that we should not expose a media API in core yet. We are probably going to want to fix a whole bunch of things. Exposing the API will prevent us from changing certain things because of BC.

Media entity in contrib could contain an upgrade path and add support for REST while we are working on this, and could theoretically keep providing support for the "broken" version of the API for as long as people need it. Upgrading to the new and improved core API can be done in their own pace.

Moving the issue to "Required before the module is shown through the UI"

Why is #2274433: Do not allow to alter Locked field via UI a "could have" in the IS, yet #2894270: Users unable to add an extension to the file upload field is a must have? I think we want to add #2895879: Discuss (and agree upon) method(s) for preventing certain Media Source field settings from being changed as a must-have for Required before the module is shown through the UI (i.e. the second phase), and remove #2274433?

Re: #93, as above, I made that change because we resolved #2894270: Users unable to add an extension to the file upload field by removing the field locking instead of trying to change the entity API (because the locking did not work perfectly and had numerous bad side effects). @phenaproxima and @seanB have been discussing different solutions.

Replaced #2862457: Add action.configuration.* with #2877383: Add action support to Media module in the list of "Could have"

#2878113: Find possible random errors in Media's JavaScript tests from the must-haves landed!

Added #2896427: Prevent enabling the "media" module if "media_entity" contrib 1.x is already enabled to the list of "must-haves before visible" group.

Thanks @marcoscano; agreed that #2896427: Prevent enabling the "media" module if "media_entity" contrib 1.x is already enabled is a good idea. I discussed it with the other framework and release managers and we agreed that we'd ideally complete that before 8.4.x.

On Wednesday we discussed the roadmap in the Media Initiative meeting, and on Thursday we went over the status of the module for 8.4.x with the core committers. Based on those discussions:

• The widgets are strongly recommended for 8.4.x, but not a blocker to marking the module stable.
• #2835767: Media + REST: comprehensive test coverage for Media + MediaType entity types will not block 8.4.x; see #2835767-38: Media + REST: comprehensive test coverage for Media + MediaType entity types for more detail. It will remain in its current place on the roadmap for now.
• #2860796: Plan for contributed modules with Media Entity API in core and #2880334: Add update path of media_entity config changes from 1.x to core media module are important to do soon and are the next key steps for the contrib-to-core transition.
• #2896427: Prevent enabling the "media" module if "media_entity" contrib 1.x is already enabled would be good to do before 8.4.x as well.

I'm updating the roadmap based on those discussions, with a new section for "Preferred for 8.4.x & Required before the module is shown in the UI". Based on all this, #2895059: Move media module out of Core (Experimental) package and into the Core package, but mark it hidden can now be unpostponed. This has consensus from the product, framework, and release managers; and will just need Dries' signoff. Amazing work, everyone! This is a big milestone.

Clarifying that contrib ports do not block core, just that they are a next priority for the initiative overall.

A few updates for recent issues:

• The REST discussion is now split into two parts, #2835767: Media + REST: comprehensive test coverage for Media + MediaType entity types (which helps with the testing gate and with providing BC for the current API) and #2895573: Decide on normalization of Media module entities (the followup to discuss the functionality we eventually want). Updating accordingly to put the first in the "Preferred before 8.4.0" section and the second down where the first was.
• #2897009: MediaInterface is missing setName() and getName() is also an API addition that is preferred before 8.4.0.
• #2897026: \Drupal\media\Entity\Media::preSaveRevision() handles the revision timestamp compared to other entity types is now a "Should".

Sorry for the noise; just adding some IDs to link to from https://www.drupal.org/core/roadmap.

Was fleshing out the roadmap with @marcoscano at DrupalCon, and noticed this entire swath of issues that I didn't even realize were blockers to turning the module on in the UI:

https://www.drupal.org/node/2825215#8.5.0-module-ui

It might be worth re-evaluating that list and see if those things are truly must-haves, or if they could be moved down to the "Eventually" heading (or some more urgent-sounding equivalent of Eventually :P).

Adding #2915738: Increase reliability of upgrade path to Media in core as part of the upgade path must-haves.

We have a need to improve the UX of the media functionality in Drupal. We'll be conducting a series of UX workshops and user testing. We'd like to get involved in this project to start work on an improved UX. Is that something the media initiative is interested in or should we just work on a contrib module?

@kurtfoster, thanks for your offer!

In the issue summary there is a general roadmap for this part of the plan, and many issues are directly UX-related. In particular, you'll see that:

• File / Image widget: #2831940: Create file field widget on top of media entity + #2831941: [PP-1] Re-create Image field widget on top of media entity
• Improving the "Formatter"'s UX: #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode
• Media library: #2834729: [META] Roadmap to stabilize Media Library
• Field API UX improvement: #2862458: [META] Once media is enabled, having the File, Image and Media reference fields all listed is confusing
• Field API UX improvement: #2895001: Use the bundle label (e.g. "Media type") instead of "Bundles" in the Entity Reference field configuration
• Improve UX for metadata handling: #2836153: Improve metadata mapping UI on Media type form

are all identified issues that need to be solved as part of this plan. All help in any of them will be very helpful.

Did you have something in particular in mind when you mention "improve the UX of the media functionality in Drupal" ?

In any case, thanks again, and welcome on board!

I'd be glad to offer free testing through our usability testing platform userfeel.com.

Updating the IS to move completed issues to a different section, so it's easier to see the work remaining.

And separating the upgrade path issues from the UI issues, because conflating them does not make sense.

Another completed bullet.

Couple other things that weren't in the right place.

And a couple more.

@marcoscano created this awesome dashboard of what we're currently working on:
https://docs.google.com/spreadsheets/d/1luk0p-ZsLVJdtPdKKIrk--DJyXzoSPOK...

Also removing some outdated/redundant text from the initial planning before things were fleshed out into (now completed) issues.

Better header links.

@xjm,

I see that #2835825: Add d8 media migration is included in the section "Required for the upgrade path", and just wanted to make sure we are all on the same page about that concept.

There is currently an effort to allow an easy transition from existing sites relying on Media Entity contrib that want to start using Media in core. This is what we generally refer to as "upgrade path".

In the (hopefully not too far) future, we'll get to the "eventually" phase, so we will be able then to:

1. Deprecate core File and Image modules in favor of Media and the types included in Standard. Discourage the use of File and Image in the UI, but otherwise leave them untouched.
2. Enable Media of the box in the Standard profile, which should now include File and Image media types as default configuration.
3. #2834729: [META] Roadmap to stabilize Media Library: Media Library as an experimental, non-hidden module.
4. Media Library as a stable core module, included with Standard.

At that point, Media in core will be the go-to solution, and existing D8 sites that weren't using media yet (or D6/D7 sites) may want to go through an "upgrade path" to start using the core Media in D8. (That wouldn't imply they'd stop using the file/image fields & widgets & formatters, as explained here, but they'd had another layer in between that would allow them to take advantage of other media functionalities.)

In short, which of these "upgrade path"s did you have in mind for #2835825: Add d8 media migration ?

I personally think it should be the second one, but the other issues in the "Required for the upgrade path" group refer to the first one, so I'm not sure we're expecting the same from them.

Thanks!

This issue:

#2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode

is marked as "Required before shown through the UI", but it has evolved since its creation (originally thought for a "media's formatter"), and is now completely a Field UI issue.

We believe the combination of: #2928699: Add an alter hook for the pre-configured field UI options and implement it in the Media module and #2895382: Hide label, thumbnail, etc. for default display of media file and image references should meet the original need that was behind putting this issue there, so these two should be the "blockers", instead of #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode.

Opinions?

Re: #118, @Gábor Hojtsy mentioned during today's meedia meeting that it would probably make things clearer for people not familiar to these issues to have a demo of what we are talking about, so here it is:

https://www.drupal.org/files/issues/referencing_media_entities.mp4

This is the process of creating a "media field" on a node, and what it outputs out of the box, with the patch from #2928699: Add an alter hook for the pre-configured field UI options and implement it in the Media module applied on 8.5.x HEAD. Basically, the entity reference field uses now the "Rendered entity" formatter (instead of the "Label" default), and the "Default" viewmode for the image media type ships with more sane defaults (showing only the title + image).

I like what we would get with #2928699: Add an alter hook for the pre-configured field UI options and implement it in the Media module and IMHO is absolutely sufficient to show media in the UI.

Added #2932226: Media Type entities don't validate machine name properly to the list of blockers for showing the module on the UI.

Wherever to sort that.

So even if #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode makes changes to Field UI instead of the Media module itself, we still need there to be no UX regression from core File and Image fields before we expose Media in the UI.

It's definitely better than it was before #2928699: Add an alter hook for the pre-configured field UI options and implement it in the Media module and #2895382: Hide label, thumbnail, etc. for default display of media file and image references were fixed. Media files are fine now. For images, though, you get an itsy-bitsy thumbnail instead of any reasonably sized image style. Like I upload a screenshot and I get this teeny tiny little rectangle instead of anything that even looks remotely like what I uploaded. (Especially since a new field is displayed after the comment field by default, it's easy to miss that the thumbnail is even there at all.) Whereas with core image fields, I can upload an image to an image field and it looks pretty OK without me ever having to even discover that the "Manage display" tab exists.

It's definitely less important than the widget and CKeditor issues, but I'm still not convinced we can take it off the must-have list. What we should probably do is do a before-and-after UX review, comparing adding a core image field with adding a media image field. (Said comparison would be done with the widget patch applied since it's not an equal comparison otherwise.)

For images, though, you get an itsy-bitsy thumbnail instead of any reasonably sized image style. Like I upload a screenshot and I get this teeny tiny little rectangle instead of anything that even looks remotely like what I uploaded.

That does sound pretty broken, and should definitely be fixed, and should definitely be a blocker to showing Media in the UI. But it's also quite a different scope from #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode, since it sounds like it affects image media items specifically, and the aforementioned issue has transformed into a general improvement to Field UI. So I think we should probably open a new issue (ideally with a screenshot of the behavior you're seeing) to deal specifically with the problem you found, and make that a must-have in this issue. Then we can remove #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode from the must-have list here.

Created followup to comment #123 here: #2934850: Media Images should be rendered at a reasonable size by default

Looks like #2934162: Provide better defaults to media form displays exists as well already, which was the other thing I ran into. Adding these both to the summary.

Alright, given:

• #2934850: Media Images should be rendered at a reasonable size by default
• #2928699: Add an alter hook for the pre-configured field UI options and implement it in the Media module
• #2895382: Hide label, thumbnail, etc. for default display of media file and image references
• #2934162: Provide better defaults to media form displays

I'm comfortable moving #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode down to "should-have".

One less blocker. Yay!!

Adding a stub section to the roadmap specifically about things that will block enabling Media in Standard, and cleaning up fixed issues.

And shuffling the previously established must-haves for Standard into that stub section.

Another blocker complete.

Actually #2934162: Provide better defaults to media form displays is in the "should" box as well ATM; the blocker was just that it should be filed.

And #2889855: Unpublished media entity can not be accessed by owner and update any media/delete any media access possibly cached by user is just a major bug.

Alright, so:

I think #2801307: [META] Support WYSIWYG embedding of media entities needs to be a blocker for adding Media to the Standard profile. However, based on the other progress that's been made with Media features and UX, I don't think it needs to block us showing the module in the UI. Obviously, WYSIWYG support is an important feature; however, Media provides a rich featureset for structured content, including audio and video support. Reusable WYSIWYG images will be a good feature and will allow us to get the most out of Media, but it's different from the issues related to dedicated fields (which need feature and UX parity with the existing file and image types).

I still need to run this idea by catch and the product managers, but Wim Leers (the core WYSIWYG maintainer) has signed off on the idea.

That means the remaining two blockers would be:

• #2831940: Create file field widget on top of media entity (which is an absolute must-have for UX).
• Whatever parts of #2862458: [META] Once media is enabled, having the File, Image and Media reference fields all listed is confusing we decide to require. To reveal the module in the core UI, we'll need at least some help text on the "Add field" screen and details on the help page. Other UX changes for the various steps in the Field UI will definitely be blockers for adding it to Standard, but aren't necessarily for the immediate needs to show the module in the UI (per the recent UX review).

Adding one should-have that was missing (for ER field UX), and cleaning up closed issues.

Another.

@xjm gave a demo of the current state of Media on today's UX call, and asked for product management/UX blessing on the idea of moving Media/WYSIWYG integration to a "Standard profile" blocker vs. an "unhide Media from the Modules UI" blocker.

There was alignment among @yoroy, @Gábor Hojtsy, and myself that this is fine, as long as the file upload widget issue is addressed in advance. (Which is what's proposed.) Giving people only an E-R field to work with on the node add/edit form represents a few too many dead-ends for such a major piece of user-facing functionality.

There's still the workaround for images that you can right-click the URL of an uploaded image and paste it in the WYSIWYG image box. Not so for videos, unfortuantely, but there are contrib solutions for this (e.g. Entity Embed).

Ideally, then, 8.5 ships with Media module turn-onnable, and 8.6 ships with WYSIWYG integration + Media Library, and Media enabled by default in Standard profile (and hopefully Umami profile, too! :D).

@xjm So if someone wants to contribute to the media initiative, is the google doc the source of truth?
Or is there any other place to look for the whole list of issue queue?

Thanks @gargsuchi! I missed your comment earlier. This issue is the canonical high-level overview of the most critical requirements, but it's hard to keep up to date for all the initiative activity. I'd say the best ways to contribute are to look at the spreadsheet and to come to meetings at 1400 UTC Wednesdays in #Media slack.

@xjm Ouch - that is 12:30 AM Thursdays for me.
But will try to keep an eye on the slack channel. Thank you.

You can join there any time, there's someone around to help you help us most of the time, just ask and someone will respond eventually.

A question that came up from Dries in a meeting last week about initiative statuses and whatnot... do we need a hard dependency on a migration/upgrade path to make the Media functionality stable + installed by default in Standard (for new sites)? Rationale being is existing sites are already set up with whatever media solution they're using (File/Image or Media from contrib or whatever). They benefit comparatively little from the stuff in core becoming stable, because they already have something that works for them. Versus new sites could be set up on a good path from the get-go, sooner, if we were to remove migration+upgrade path from the list of "must-haves" for "replace File/Image with Media as the default solution." (Maybe move these to more of a D9 target/blocker?)

(Just the messenger, please don't shoot me. :D)

My 2c - I don't think there should be a hard dependency on the migration for media from D7 to D8. Certainly work could continue on that piece for another core release (8.8?) for those who have used the media module in D7. Specifically for us, we are waiting on the media module UI to be added in core so we can be sure that the code we use in our custom modules referencing media entities works without being on a dev version of core.

In my opinion a decision does need to be made since there are way too many things lacking with the core file system functionality and there are still too many things left up in the air as far as the core media setup. How can I justify spending the time setting up one system or another for a client knowing that somewhere soon around the next corner things could be changing and I am going to have to either donate my time or bill them extra for something that other content management systems handle the way that I believe the core media system will eventually turn out. In my opinion if you install a new version of D8 you should be getting Media and if you are on 8.6 or before you should get converted to the core media as soon as you enable the core media module. Right now if you install things and then do a upgrade from another site you still have to jump through way too many hoops to get the files imported and then converted to use media if that is the way you want the site to eventually end up and for the majority of customers they are going to want to stay with D7 which is working or wait until the "Powers that be" decide which way media in D8 is going to be handled.

Just my two-cents.

Re #143:

Unless I'm missing something, the upgrade / migration from a D8 site using file/image fields to media in core is very unlikely an automated process where we take care of everything in code. Since features are uneven, manual action would be often needed, and probably some dev action too (at a minimum, the theming would need to be adjusted, since I wouldn't expect the markup of referenced media entities match that of file/image fields).

Another example, for sites where private files is a thing, it would be nearly impossible to provide an upgrade path with feature-parity in the short term, at least not until core has a (currently non-existent) method of restricting entity access the same way it does for files in file fields.

Because of all that, I'm +1 for considering the eventual upgrade/migration of D8 file/image fields a parallel effort that shouldn't block marking media modules as stable and enabling it by default in standard.

Note however that IMHO enabling Media by default in standard doesn't necessarily mean marking file/image fields as deprecated (or hiding them in most places). Using "media fields" everywhere by default does involve being able to meet users' current expectations of what's possible with those fields. Before doing that, I believe it would be ideal to solve all the UX important issues we have open, as well as making sure there is no room for confusion that could potentially lead into sites being inadvertently misconfigured (re: private access, etc).

#145 make sense

This is a first update of the IS. Remove a lot of outdated stuff. I'm planning on adding more issues as I triage the issue queue.

I'm assuming #2113931: File Field design update: Upload field. is at least related, so adding as such.

#2998091: Remote videos overflow their containing element is in!

#2889855: Unpublished media entity can not be accessed by owner and update any media/delete any media access possibly cached by user was committed, so I'm moving that to the "done" list. :)

I had a call today with @chr.fritsch, @marcoscano, and @seanB (so, all the subsystem maintainers). We have unanimous agreement on what we would like to do in order to get Media added to Standard. Our plan of attack has four parts:

1. #2801307: [META] Support WYSIWYG embedding of media entities and #2983179: [META] Implement stricter access checking for the media library must be fixed, since they are hard-blocking the stability of Media Library. It must be feature-complete before we can add Media to Standard.
2. For #2930514: [meta] Migration to media, we'd like to create a contrib module which provides a migration UI that site builders can use, at their own pace and according to their specific needs, to migrate to the Media system. This would cover the migration of file and image fields to media fields, along with file and image entities to media entities. We'd like to do this in contrib so that we are not constrained by the core development process, but it should be considered the "official", core-endorsed way for existing sites to migrate from File/Image to the Media system. So although we'd do this in contrib, we'd be open to declaring it a Standard-blocker for Media in core.
3. #2904842: Make private file access handling respect the full entity reference chain is a hard problem. Files and images are inherently attached at the hip to their parent entity, but media entities were never designed or intended to work that way. Media entities have their own existence, by design, and that is a fundamental aspect of the media system's architecture that we don't think should be changed. However, there are valid use cases for keeping a media entity "private", especially in the media library. Our preferred solution is to create a contrib module that implements the concept of "privacy" for media entities. It would probably add a boolean base field indicating if a media entity is private, and then alter the media library so that it doesn't show private media items to anyone (except perhaps the person who created them). Even though we have disabled canonical URLs for media items out of the box (which is a huge victory on this front), heavily customizing access to media entities (not just in the media library) is an even harder problem that we don't think can be solved in a one-size-fits-all manner. The ideal thing would be to make the existing node access system, which does allow the proper levels of granularity, generic for all entity types. There is an issue for that, but it's outside the scope of the Media Initiative and should not block Media's inclusion in Standard.
4. Finally, for #2862458: [META] Once media is enabled, having the File, Image and Media reference fields all listed is confusing, we think that, at the same time that we add Media to Standard (i.e., in the very same patch), we should simply remove File and Image from the list of common reference targets. This means that, when adding a new field, under the "Reference" group, you'd just see "Media". You would still be able to add a standard File or Image field, but you'd need to choose "Other...", which would take you to another screen where you can choose "File" or "Image" from the list of referenceable entity types. Other than that change, File and Image would continue to work exactly the way they always have. Additionally, we might tweak the labels a bit to make it clearer that "Media" refers to things like files/images/video/etc., and that "File" and "Image" refer to one-off, non-reusable files.

I think we'd like to get buy-in from the product, framework, and release managers at DrupalCon -- but we on the Media team are in agreement that this, in our opinion, is the smoothest and clearest way to get Media in Standard before Drupal 9.

> we should simply remove File and Image from the list of common reference targets. This means that, when adding a new field, under the "Reference" group, you'd just see "Media". You would still be able to add a standard File or Image field, but you'd need to choose "Other...", which would take you to another screen where you can choose "File" or "Image" from the list of referenceable entity types. Other than that change, File and Image would continue to work exactly the way they always have.

As discussed in the referenced issue, that's not really possible with the way things work now. Those two sentences (work as before vs. select after other) contradict each other. file and image are actual field types and must remain field types, as they store their own data, while the others are just shortcuts for a pre-configured entity reference field. We can't move them to Others, unless we offer a new intermediate step or some magic on the that form to then afterwards change the field type, that seems strange.

IIRC, what we discussed is to move those two out of the Reference group (to generic for example, or maybe their own group), they probably should never have been in Reference, nobody would think of file and image fields as "references".

> [images and files] probably should never have been in Reference, nobody would think of file and image fields as "references".

+1 for that.

We met at DrupalCon to discuss the proposal in #157; notes are here: https://docs.google.com/document/d/1l4OTDRu1XYY-eahI4cEseVaf1dsSHIHtXt2t...

There was agreement on the general scope of Media Library blockers, handling the migration problem in contrib.

The specific pieces of pushback were:

- The "drop-zone" file field widget is a significant part of the design of the Media Library (both UX + product agreed on this), so the outcome was to keep it in the must-have list for now, but drop it down to should have should we end up in a situation where everything else is done and the remaining timeline puts Media Library making it to stable for 8.8 at risk.
- There was strong desire from both product + release management to handle feature parity between file/image fields and media in core, as a way of addressing both #1 and #3. For example, two toggles: "Private access" and "Listed in Media Library" or, one-off media that follows parent's access restrictions (user profiles viewable by anon = user pictures viewable by anon) Subsequent discussions talked about handling this in a similar way to Layout Builder's "Inline block" concept.
- We need some solution for access restrictions in an HTTP API (e.g. JSON:API) context. This requires a follow-up media security discussion.

Next steps are to start with some user stories of the desired behaviour with file / image vs media, and then figure out how to fill the feature gap and solve the migration/security issues when that is done.

Making a couple of small changes to the IS's point about a migration path.

Adding #2973124: Provide better context in accessible names of EntityOperation views plugin drop-button operations to the should-have list, as per discussion in #2834729-170: [META] Roadmap to stabilize Media Library.

After discussion with @xjm and @seanB, I am updating the roadmap to include issues that have been created since late March, 2019, which is the last time our issue queue was properly triaged.

I'm adding:

• #3042383: Document how to attachment JavaScript inside the oEmbed iframe (could-have)

Adding #2987911: Make media compatible with content moderation to the list as a must-have.

https://wimleers.com/blog/media-embedding-drupal-8.8#comment-20422 made me look at this issue summary again. Is it really true that media migration support must be completed before we can enable Media in the Standard install profile?

Something else that needs to be done before enabling media by default is #3002770: Provide authors with tools to manage transcripts and captions/subtitles for local video and audio. It was mentioned in #2786785: Media in Drupal 8 Initiative last year, but I've just noticed that plan was closed and it hasn't been added here.

WCAG Guideline 1.2 Time-based media wasn't in scope for Drupal core for 8.0.0 because there was no time-based media. The local audio/video and remote video bundles added to standard profile have brought this large section of WCAG into scope, along with the related ATAG parts (chiefly B.2.1: Ensure that accessible content production is possible and B.2.3: Assist authors with managing alternative content for non-text content).

The plan for captions and transcripts is already in progress, and a working demo was shown at one of the UX meetings. (TODO: dig out the link.)

My apologies if this is the wrong place to post this question...would it be possible for #2998713: Add a media source for remotely hosted files to be listed as a 'should' or 'could' have for the Media Initiative? Specifically, remote audio: https://www.drupal.org/project/drupal/issues/2998713#comment-13247616

Moving #3073901: Determine an upgrade path from CKEditor image button to media library from #2834729: [META] Roadmap to stabilize Media Library.

#167 - I don't see why not. I think it's probably a could-have. Remote audio would be a common need for a podcast, say.

Discussed #3084011: The source of alt text in embedded image media is not clear with @phenaproxima, and agreed to treat it as a must-have before enabling media library in standard profile. There is a feasible proposal for a configurable policy and/or an upgrade path.

Updating the IS with some specifics on the Media Library requirements.

Hi! I searched around but came up empty so far -- in other words, if this is the wrong place, just let me know ;-)

Is there (or was there previously) discussion about allowing media entities to be indexed by the search module included in Drupal core? I don't mean like, the contents of file attachments (like search_api_attachments), I mean just indexing "media entities" themselves, and any non-file fields they have (like nodes).

I feel like the fact that I'm not seeing any mentions of "search" on this Roadmap issue (or #2831274: Bring Media entity module to core as Media module or #2801277: [META] Support remote media assets as first-class citizens by adding Media Entity module in core) means that it was probably discussed at some point and folks came to a decision not to pursue this functionality -- but (a) I wanted to check and make sure; and, (b) I'd be interested in reading about that discussion, if it happens to live on d.o :)

Thanks so much!

@alisonjo315 - can you file a separate issue about that search index issue please? Because - who knows? - maybe it wasn't discussed at all.

I tried looking for "search" in all issues assigned to the media system.

https://www.drupal.org/project/issues/drupal?text=search&status=All&comp...

I skimmed through the issue titles there - it doesn't turn up anything that looks relevant, either.

Parenting to easy out of the box.

Would it be possible for issue #3163228: Add Remote audio media source/type to Core Media to be listed as a 'Should have' or 'Could have', which would fall in line with the "Easy out of the box" Drupal core initiative?

I think it's time to completely refresh the issue summary here. The Media Initiative is in a very different place than it was the last time this was thoroughly updated.

So, I went through the following meta issues:

• #2825215: Media initiative: Roadmap (this issue)
• #2930514: [meta] Migration to media
• #2801307: [META] Support WYSIWYG embedding of media entities
• #2834729: [META] Roadmap to stabilize Media Library
• #2988431: [Meta] Accessibility plan for Media Library Widget
• #3087227: [META] Split up and refactor MediaLibraryTest
• #2862458: [META] Once media is enabled, having the File, Image and Media reference fields all listed is confusing

In each one of these, I took note of the open issues linked in the issue summary and sorted them into lists, divided by topic area (accessibility, UX, DX, site builder experience, integrations, WYSIWYG, and so forth...the areas were arbitrarily chosen by me) and further segmented by the MoSCoW priority they had in the meta they came from.

The resulting list of issues should be an accurate aggregation of the state of all those metas, consolidated and organized into this one so that we have a central place to refer to the stuff we still need to do to make Media easy out of the box. Therefore, I'm going to close most of those metas in favor of this one. I'm going to leave open the ones which are tagged as metas, but have so far had very little activity or direction and therefore aren't clearly ready to be closed.

I'm also removing the completed issues from the issue summary.

Adding #3102011: Media Library's embed modal is almost unusable with less than 600px browser viewport width as a should-have in the WYSIWYG area, which was linked by @huzooka in #2801307: [META] Support WYSIWYG embedding of media entities.

Adding Easy Out of the Box tag.

Adding NorthAmerica2021 tag for visbility.

DrupalCon NA is April 12-16 with a focus on EOOTB on Wednesday, April 14.
Thanks

Removed #3067116: text_summary() returns malformed (not normalized) HTML for basic_html and other formats that use filter_html instead of filter_htmlcorrector, since it is now fixed.

For easier scanning, I moved the Must-have issues up from the topic sections and into its own top-level section.

I just moved the existing issues; I did not change their designations. However, I'm not convinced that all of the must-have issues need to necessarily block putting Media into the Standard profile, but that's a topic for a future triage discussion.

Found a fun one today: #3326447: Deleting a Media view mode should not result in the deletion of an entire text format

I moved #3326447: Deleting a Media view mode should not result in the deletion of an entire text format up to the must-haves; it's been promoted to "critical data loss bug" following discussion with @catch and myself.