Background and Problem

Drupal core is an amazing content structuring tool with options to give rich structure to basically everything in the system. While Drupal core includes basic file and image support, it is a far cry from what a modern web system should support out of the box for media handling. External media cannot be embedded easily in core and media cannot be reused.

Due to very limited functionality provided by core it is also very hard for contrib to build on top of it. Core should provide base APIs, design patterns and paradigms to guide contrib work.

Proposed resolution

Improve media handling in core based on the work that has already been done in contrib. Parallel to that continue improving contrib components that won't be added to core yet and consider adding them in the future.

This is a 8.4 subset of the broader plan outlined in #2786785: Media in Drupal 8 Initiative.

Process and where to find it

The initiative has weekly meetings in #drupal-media on Freenode IRC at 2pm GMT on Wednesdays.

Proposal roadmap

This functionality will be the MVP for Media in Drupal 8 core. The objective of this MVP is to provide a simple media solution to make Drupal 8 easy to use out of the box for basic use cases. This MVP can ideally be achieved in a short time -- to be included in Drupal 8.4 RC1. This goal can also ideally be achieved with the help of some of the new contributors/development efforts, assisted by the existing media team members.

The plan is based on the DrupalCon Drublin core conversation and discussions that followed (slides, recording).

Media entity type in core for 8.4.0 - #2801277: [META] Support remote media assets as first-class citizens by adding Media Entity module in core

  • Create a media module that is released in 8.4.0 as non-experimental. See #2831274-139: Bring Media entity module to core as Media module for why it's critical for contrib to be able to rely on a non-experimental media.module. At a minimum, this module must contain the Media entity type and all supporting code and APIs needed by it. The core patch for this is in #2831274: Bring Media entity module to core as Media module, which is mostly a straight copy from the Media Entity contrib module.
  • Allow site builders to start creating entity reference (ER) fields to Media entities instead of File fields, without compromising features or UX for content authors and site visitors. This requires adding:

    The patches in the above issues currently add this code to media module, which means non-experimental before release. Alternatively, it might be worth considering splitting some or all of that code to a separate, experimental module.

  • Allow site builders to start creating entity reference (ER) fields to Media entities instead of Image fields, without compromising features or UX for content authors and site visitors. This requires adding:

    The patches in the above issues currently add this code to media module, which means non-experimental before release. Alternatively, it might be worth considering splitting some or all of that code to a separate, experimental module.

  • When all the above is done, change Standard profile to ship with ER fields to Media entities instead of File/Image fields. Don’t force existing sites to migrate. They can keep using files as we don’t remove any of that from core. We can provide optional migration path later, but don’t require it until D9.
  • Add one implementation of remote media (YouTube and/or other oEmbed providers): #2831944: Implement media type plugin for remote video. That patch is currently adding this to media module, which means non-experimental before release. Alternatively, it might be worth considering splitting it to a separate, experimental module.

Build consensus for how to embed media entities (or whether to not it) - #2801307: Support WYSIWYG embedding of entities other than files

To retain current core’s functionality related to WYSIWYG image embedding we need to figure out how to embed image media entities. There are two possible solutions to that problem:

  1. Move parts of Entity embed in core (simplified text filter) and handle embedding through it. Entity embed display plugins and embed button config entities stay in contrib.
  2. Keep embedding files as we do now, but implement editorial workflow through media entities. This will allow content creators to select from media library or create one by uploading an image. When media entity to be embedded is selected we extract image information behind the scenes and inject it in the body field.

Second solution seems to be slightly more preferred than the first one since doesn’t require us to move another module to core right away, which helps us from getting into too much complexity at the same time. Even if we go with the second option we still can move entity embed into core as part of some later minor release.

Additional core issues that affect media embedding experience:

Media library in core for 8.4.0 - #2834729: Create an MVP for adding and re-using Media for implementation, #2796001: [prototype] Create design for a Media Library and #2828538: Produce high fidelity screens based on Media prototype for design

Build a media library on top of media entities. Library will be implemented in a form of a view, which will ship as part of a core experimental module. We will use a special media entity view mode to render each item in it, which will give us enough flexibility and allow contrib modules to easily include additional media types into the library.

When library is implemented use it to support basic re-usability of media. For more complex use cases in this area rely on the Entity browser module in contrib.

Team and resources

Existing members of the Media initiative:

+ new contributors (critical for success of the initiative).

Our goal is to organize few focused week long sprints to push the initiative forward. It is absolutely crucial to have all of the core members attend those sprints. Details about dates, location, funding are still to be defined.

Signoffs

TBD

Signoffs needed

TBD

Signoffs given

TBD

Comments

slashrsm created an issue. See original summary.

Gábor Hojtsy’s picture

Component: Idea » Active Initiative
Status: Needs review » Active

Moving to active initiative as per #2786785-68: Media in Drupal 8 Initiative.

xjm’s picture

@Gábor Hojtsy and @slashrsm raised that many parts of this plan depend on the Media entity being stable. However, the summary says "even as an experimental modules if needed to begin with" so I could use some help on the specifics.

When someone has a chance, can we outline which parts depend on Media entity and subsequent APIs and UIs being stable, versus beta? The specifics of what that means are here: https://www.drupal.org/core/experimental#beta Additionally, stable core code cannot depend on beta core code (so for example, if we make additions to the Standard profile, then those additions are blocked on the APIs and UIs being stable).

What I imagine happening is introducing the entity type as beta, to indicate we won't make BC breaks unless they are critical. Then, followups can be filed for any requirements, and as the initiative progresses, other experimental code can also be built on top of it. When the first change to stable core is needed, that's when we need to make a decision about whether the entity type is fully stable or not.

By the time we get to 8.3.0-rc1 in March, we would have lots of time to complete followups for it to be stable. If changes to stable core are ready for 8.3.0-beta1, then we would make the decision earlier based on the needs of those stable changes. If they aren't ready for 8.3.0-beta1, then they will be targeted for 8.4.x anyway, but the Media entity itself could still be stable or rc or whatever was appropriate.

xjm’s picture

When all this pieces are ready change standard profile to ship with Media entity based solution by default. Don’t force existing sites to migrate. They can keep using files as we don’t remove any of that from core. We can provide optional migration path later, but don’t require it until D9.

Agreed. That part definitely depends on Media Entity and every other bit of code and UI being stable. I wouldn't imagine this happening before 8.4 at least.

Re-create features that core offers at the moment on top of it (local images and files).

This depends on how we re-create them. We still need to provide BC for those entity types for the duration of 8.x. Will they be updated to extend Media entity internally while providing full BC? Or will they simply be deprecated in favor of replacement Media\File and Media\Image entity types, which could be experimental initially until we make them the default for new installs?

Add one implementation of remote media (preferably YouTube - through oEmbed?) and support simple editorial workflows related to it and display configuration.

This could all be experimental until it's ready to be stable.

xjm’s picture

The Media Library also should be experimental at first, and does not require Media Entity to be stable. The embedding workflows will be more complicated, but it sounds like consensus is needed there still before we start prototyping that.

Gábor Hojtsy’s picture

What is the dev/user experience process when/if the media module is experimental.

1. You install Drupal stable.
2. Create a few content types, create some content.
3. Figure out you need to embed youtube videos or want to reuse your files.
4. Install media module.
5. Now you have your content types with your file/image fields and another way to have media on the site.
6. Win? (Nope).

One way for media to be experimental and not be a usability nightmare to try any site other than a totally empty one is to remove the creation of default file/image fields from all entities in 8.3.0 so users would need to create either those or use media. I don't think that is what we want? Still then, @berdir's concerns from #2831274-139: Bring Media entity module to core as Media module would apply (in short it does not make any contrib's life easier to have an experimental media module in 8.3.0, its better not to have anything for media then):

I already mentioned this in IRC and it has been discussed a bit, so here is the official comment to propose my idea:

First, note that media_entity/media.module is very different for pretty much anything (the closest is moderation/workflow) else we put in core as experimental. bigpipe is a perfect use case for an experimental module. As it now official declares it has no API, it has no data.
You can enable it and disable it again, any time you want.

Media is different. It contains data but just as important, it *is* an API for a large ecosystem of contrib and custom modules that integrate with it. I think I can speak for all the media related/depending projects and distributions (like Thunder and Lightning) maintainers
that having experimental media module in core would bring us nothing and it would be impossible (mostly due to the amount of time it would require) to maintain modules that work with both media in core and media_entity in contrib, and as a result it would also be of extremely limited usefulness for users.

And while we can argue that moderation/workflow data can be removed without losing too much (your content is still there and still correctly published/not published), that is not the case for Media. You can't just uninstall it because we have no upgrade path in
8.4.x, all your media and metadata for it would be gone.

This proposal based on @catch's comment in https://www.drupal.org/node/2789315#comment-11762228, where he confirmed that anything new that has been added to 8.3.x is "experimential" until we release 8.3.0-beta1.

That means if we commit this issue now, then we still have time until then to improve and stabilize it to reach a MVP. What we need is a set of goals that we need to achieve until then. Both processes (like code/UX/testing quality) but also an actual feature set.

If we do not achieve that in time (say, a certain amount of days before beta1), we remove it again and try again for for 8.4.x and stick to media_entity in contrib. Nobody of us wants that, but IMHO it is better than having something experimental in core that nobody can use.

Basically, this is exactly what Dries proposed in his keynote in Barcelona with Feature branches, we just don't have them yet. But since (almost?) everything we do will be contained in one module, it should be relatively easy to remove again if we have to.

xjm’s picture

@Gábor Hojtsy, I agree that it the complete user experience should involve things being provided on installation, which requires everything to be stable. However, I feel that Berdir's comments undervalue that experimental modules are expected to become stable within one or two minor releases. Adding something to the Standard profile once it's complete is not mutually exclusive as introducing it experimentally at first. We definitely intend to use experimental modules to develop features that will eventually ship as the default for new installations.

If we add the API now, in November, we have until March to make sure that it is indeed ready to be stable for contrib and mark it as such. I do understand that the contrib ecosystem's dependency on the Media Entity is really important, so we should have releasing it as stable in 8.3.0 as our goal. The decision to do so will be based on consensus between core maintainers and media maintainers.

We would not add it to Standard profile until all the other features in the feature set were also ready to be stable, including the new file and image entity types, the library, a really polished UI, embedding, etc. For that change to be in 8.3.x, all of those things would have to be ready in February. I'd be really, truly amazed at such progress. But that would mean all of those things would need to be done and fully polished. I think the 8.4.x beta in August is a better target for really polishing all the different parts of the implementation, and therefore we would not be changing Standard profile in 8.3.x.

For the future work the initiative intends and for the contrib ecosystem, I really don't think there is a big difference in calling the module stable now, versus marking it beta now and aiming to have it stable by release. However, it does make a big difference for release management, for managing followup issues, and for what we expect of the initial patch.

Gábor Hojtsy’s picture

Issue summary: View changes

Tagging for framework manager review given that this "represents a significant change in architecture or public APIs". There is little point (or according to @berdir several negatives) if only parts of this happens and we are to keep using file/image fields as-is for example. So the resulting system will need to make significant changes. Implementation issues for this include but not limited to #2831936: [PP-1] Add "File" media handler plugin, #2831937: [PP-1] Add "Image" media handler plugin to reproduce file/image wrapping functionality, #2831940: Create file field widget on top of media entity and #2831941: Re-create Image field widget on top of media entity to support widgets for them and #2831943: Re-create image and file field-like formatters on top of media entity to support formatters on them. (These issues are sub-issues under each section above). Once those are in (if all components are stable), standard profile would come with media fields instead of file and image fields, and file and image would be deprecated. Which is a pretty significant change to say the least.

I believe the media team understood this review is not needed because the ideas review meeting approved the plan but according to @xjm that is not the case, so we need that review. Hopefully that does not invalidate the steps and directions identified. #fingerscrossed

Gábor Hojtsy’s picture

xjm’s picture

Component: Active Initiative » Plan
Status: Active » Reviewed & tested by the community

Let's do this, to make it clear it is an agreed plan that just needs committer feedback and signoff if appropriate.

slashrsm’s picture

I entirely support @berdir's opinion. It also seems that his proposal is almost identical to what @xjm is proposing. With one little detail. I feel (and please correct me if I am wrong), that one plan assumes we commit for the storage component to become stable when the minor version it was committed to is released while the other makes this a target which we don't commit 100% to.

This might sound like a non important difference, but it is. By leaving main storage component experimental in a stable version of core we leave the doors open for data structure changes. If we make it stable we commit to not do that. This comes down to responsibility. Are we willing to deal with our user's data with extreme caution and respect or not.

Almost everything else that we do as part of this plan (media library, formatters, field widgets, ...) can be experimental initially. Data model, however, can not. I stated this when I initially proposed first parts of this plan in Dublin and I still firmly stand behind it.

It is also worth mentioning that there is nothing really special about the data model of the media entity. It is very simple and standard and not really different from other core's entity types such as nodes or terms.

Deciding against this would have significant implications for this plan and I would need to re-consider my involvement and commitment to it in that case. This is my personal opinion and I obviously can't speak for the rest of the media team.

catch’s picture

The media entity being in beta doesn't mean that we'd break people's data, in fact it means we specifically wouldn't. However it does mean some other things:

- we might make such a change in 8.3.x itself before beta/release candidate, I don't think we'd even do that if it goes straight in as stable except reverting or something.
- something like theme hooks/formatters or similar API but not data changes could be done with bc breaks if necessary
- it gives us leeway in the case of unknown unknowns that would add to the critical bugs queue

For comparison, the initial patch for content_moderation module wasn't possible to uninstall. The contrib workbench_moderation roadmap had it going to 1.0 with that architectural issue. Fortunately we spotted that before it went in, but if it had gone in as stable without being spotted we'd be pulling our hair out now.

fwiw I'm 100% on board with deprecating the image + file fields, having those created in the standard profile instead, changing the migration from 6.x and 7.x so it migrates into media fields, and also providing an 8-8 migration from image/file to media so that existing Drupal 8 sites can make the change too.

It'll be incredibly confusing having both in core, so we should aim to do a full replacement.

Should also add we are already trying to make data structure changes to core entity types that are stable, by making them revisionable, however that's extremely difficult to do.

Gábor Hojtsy’s picture

fwiw I'm 100% on board with deprecating the image + file fields, having those created in the standard profile instead, changing the migration from 6.x and 7.x so it migrates into media fields, and also providing an 8-8 migration from image/file to media so that existing Drupal 8 sites can make the change too.

It'll be incredibly confusing having both in core, so we should aim to do a full replacement.

Should also add we are already trying to make data structure changes to core entity types that are stable, by making them revisionable, however that's extremely difficult to do.

So the plan was/is to provide new media fields on new sites and leave existing sites alone until a well tested/proven migration path is implemented. The migration path was/is not targeted for inclusion in 8.3.0 as we need to focus on building the thing that people would migrate over to. In some ways this plan is similar to how Drupal 8 was released by a migration path itself, targeted at new site builds. That said, we need to implement the migration path to deprecate for at least de-emphasize file/image fields because otherwise the adoption of media may be hard.

catch’s picture

So the plan was/is to provide new media fields on new sites and leave existing sites alone until a well tested/proven migration path is implemented. The migration path was/is not targeted for inclusion in 8.3.0

I don't mean for 8.3.0, I just mean in general that should be the aim.

Gábor Hojtsy’s picture

Component: Plan » Active Initiative

Moving to an active initiative to reflect reality. Not moving any of the signoff tags, especially given other active initatives like outside-in have the signoff tags on them too.

catch’s picture

@effulgentsia pointed out that depending on how much we're able to get stable for 8.3.x, just the media entity in core could be confusing for new users if they discover and enable it, without any of the contrib modules.

I might have a compromise to keep the module hidden = true even if it's stable - then it can be shown when the contrib modules are installed that need it. Once there's enough in core to make it viable without any contrib, we then just remove the hidden = true.

Dries’s picture

Janez (slashrsm), would you like to be the initiative lead for this? I think you'd be a great initiative lead but I wanted to formally ask instead of assume. :-)

Gábor Hojtsy’s picture

That sounds interesting and technically feasible I guess. I wouldn't be in the marketing people's place in that case for sure:

NEW! Drupal 8.3 core now has media capabilities!! Erm, how? Well, download this contrib module, its cool. Huh?

catch’s picture

@Gabor it'd be similar to adding it as an entity type in \Drupal\Core, I think it's OK to say we added an API (+ UI scaffolding), use contrib until those bits have migrated to core too. Or just don't say anything but it's better than it getting stuck beta in core for six months which no-one wants, or staying in contrib for another 6 months (and having to negotiate the patch landing early in 8.4.x vs. the contrib module).

Gábor Hojtsy’s picture

@catch: I agree its better and I agree its similar to adding an API.

slashrsm’s picture

I agree with the hidden: true idea that @catch proposed. We will at least need to communicate this change to people that are using media_entity or media in contrib at the moment as they will need to upgrade. We already started preparing update path for media_entity users and as soon as media (hopefully) gets into core we'll start communicating that as encouraging users to test it on their sites.

Janez (slashrsm), would you like to be the initiative lead for this? I think you'd be a great initiative lead but I wanted to formally ask instead of assume. :-)

Thanks for asking :). I feel that I've been doing a lot of the things that are required as part of this role already (with a lot of help from others). I guess that making it official is the logical next step.

While working on #2831274: Bring Media entity module to core as Media module in Berlin the question about the subsystem maintainers for the new Media subsystem was brought up. After some discussion @phenaproxima and myself agreed to step into that role (both under the condition that there is more than one person wearing this hat).

I'd like to use this opportunity to express importance of the large-enough core team and sustainable funding. This is a big initiative which will succeed only if there are multiple parties actively taking part. Both financially and in other ways. I collected more thoughts about this topic in http://www.md-systems.ch/en/blog/md-systems/2017/01/09/current-status-of....

webchick’s picture

Even though this does not have the product manager review tag, wanted to share that we discussed this issue on this week's Ideas Queue Review.

From a product/UX POV, we are comfortable with the list of issues outlined in the summary, with the exception that we can't call this MVP "done" until/unless the user-facing components are in there as well. So while the framework/release manager call to split it between 8.3 and 8.4 makes sense, we would not want to expose the media entity to end users until/unless the user-facing components are done as well.

Gábor Hojtsy’s picture

What stops us from removing "Needs release manager review" and "Needs framework manager review"?

Should this be moved to fixed or it being an active initiative as RTBC is how we signify an active initiative?

Gábor Hojtsy’s picture

Opened #2843399: Add media initiative to MAINTAINERS.txt for adding the maintainers.txt entry.

xjm’s picture

@Gábor Hojtsy, you are the one who moved it to active initiative (twice actually); that's how it ended up marked as an active initiative before we signed off and how the review process got a bit mixed up. In the future I think committers should always do that so there's a clear indication on the issue that we are on the same page. This is all a minor process aside though.

@effulgentsia is going to provide framework manager review later today hopefully. There are outstanding release management issues that need to be resolved and so that tag will need to remain on as we address them. I will add a proposal once this has framework signoff and as the initial media entity issue continues to receive framework manager review. My top priority is providing a smooth transition for contrib without compromising the core product. In the meanwhile I'd encourage reviewers and contributors to continue to work on the first three steps of #2801277: [META] Support remote media assets as first-class citizens by adding Media Entity module in core and try to get #2831274: Bring Media entity module to core as Media module to RTBC.

I agree with everything in #21. Thanks @slashrsm.

xjm’s picture

Oh, and thanks @webchick for #22; I agree also.

effulgentsia’s picture

Issue summary: View changes

Updated the issue summary with:

  • Fleshed out the Media Entity section.
  • Removed "Open core issues" links that were already linked above, and moved the rest to the embed/wysiwyg section.
  • Tweaked formatting: <h5> to <h4>.
effulgentsia’s picture

This functionality will be the MVP for Media in Drupal 8 core. The objective of this MVP is to provide a simple media solution to make Drupal 8 easy to use out of the box for basic use cases.

From a framework management perspective, I'm very happy with the plan in this issue's summary (my changes in #27 were just me detailing out the plan as I understand it, incorporating information from other issues, and not making any intentional changes to what the Media team has been planning), so removing the "Needs framework manager review" tag. I checked with @alexpott and @catch and they had no objections to me doing that. I think all of the steps and issues outlined in this plan are architecturally sound, and I agree that each one is "Essential" (per the issue title), and that the combination of all of them makes for a nice MVP of core media handling. I think some of the "Extras" in #2786785: Media in Drupal 8 Initiative might also be appropriate to include into core and some might not be, so this issue is just the "first round of improvements", and I'd be happy for the "Media initiative" to continue beyond this first round so long as people are available and motivated to work on it.

This MVP can ideally be achieved in a short time -- to be included in Drupal 8.3 RC1 (even as an experimental modules if needed to begin with).

I'm not a release manager, so leaving the "Needs release manager review" tag for someone who is. However, I don't see a path for all of the issues here to land into 8.3, and especially for all of the "Media entity type in core for 8.3.0" ones to land in a non-experimental state. I'd love to be proven wrong on that, but that's my current feeling.

So while the framework/release manager call to split it between 8.3 and 8.4 makes sense, we would not want to expose the media entity to end users until/unless the user-facing components are done as well.

Because I feel that there will be such a split as to what makes it into 8.3 and what doesn't, we need to be very careful to make sure that each step along the way leaves Drupal 8.3 in a shippable state. We can discuss the details of that as we get to each step, and I'll start with a more detailed comment soon on #2831274: Bring Media entity module to core as Media module with thoughts as to how we can make that one shippable on its own.

Opened #2843399: Add media initiative to MAINTAINERS.txt for adding the maintainers.txt entry.

Yay! Thank you, @slashrsm, and everyone who's been doing really fabulous work on all of the issues.

slashrsm’s picture

I think some of the "Extras" in #2786785: Media in Drupal 8 Initiative might also be appropriate to include into core and some might not be, so this issue is just the "first round of improvements", and I'd be happy for the "Media initiative" to continue beyond this first round so long as people are available and motivated to work on it.

Totally agree.

Because I feel that there will be such a split as to what makes it into 8.3 and what doesn't, we need to be very careful to make sure that each step along the way leaves Drupal 8.3 in a shippable state. We can discuss the details of that as we get to each step, and I'll start with a more detailed comment soon on #2831274: Bring Media entity module to core as Media module with thoughts as to how we can make that one shippable on its own.

8.3 was a goal around the Dublin time. Now we're obviously in a different situation. I see two possible paths forward at this point:
a) We try to get #2831274: Bring Media entity module to core as Media module, #2831936: [PP-1] Add "File" media handler plugin and #2831937: [PP-1] Add "Image" media handler plugin in for 8.3 and plan the rest for 8.4. This improvements would not be really visible and only sites that already use Media entity would benefit from them. In this case it would make sense to hide "Media" module for 8.3.
b) We prepare #2831274: Bring Media entity module to core as Media module, #2831936: [PP-1] Add "File" media handler plugin, #2831937: [PP-1] Add "Image" media handler plugin and try to get them committed as soon as 8.4 is opened. Then we have a lot of time to add more visible features in the same minor release and make sure that media entity is tested and stable. This also gives existing sites more time to test the media entity upgrade path that will be provided in contrib (8.x-2.x branch of the Media entity module - @seanB and @phenaproxima started working on that in Berlin).

I am fine with both approaches, but would actually prefer b) at this point. Mostly due to the fact that we give people that run existing sites much more time to test the upgrades.

Gábor Hojtsy’s picture

@xjm:

@Gábor Hojtsy, you are the one who moved it to active initiative (twice actually); that's how it ended up marked as an active initiative before we signed off and how the review process got a bit mixed up. In the future I think committers should always do that so there's a clear indication on the issue that we are on the same page. This is all a minor process aside though.

Right, sorry for that, I realize it was a mistake. I thought that a lot of other conditions practically made this an active initiative (eg. Dries blogging about it with all the steps outlined that are here, paying three of his employees to attend a sprint to implement this plan, etc). Once I learned that those are not good indications of a plan being approved, I immediately stopped recruiting people to work on something that may be futile to work on. Now that we have a plan approved I think we may be able to have something nice in Drupal 8.4.

@slashrsm: how does your plan (a) deal with field widgets? Would they be in a separate contributed module then that is created just for the sake of complementing Drupal 8.3? Also would that be the same contributed module that would hold a media library or the media library would be a separate contributed module?

slashrsm’s picture

@slashrsm: how does your plan (a) deal with field widgets? Would they be in a separate contributed module then that is created just for the sake of complementing Drupal 8.3? Also would that be the same contributed module that would hold a media library or the media library would be a separate contributed module?

Idea is to add field widgets to core (probably as experimental in the beginning?). Core MVP would not be complete without them. The reason why I didn't include them in a) is the fact that they depend on media patch + handler plugins. Considering the fact that we're 2 weeks from freeze, complexity of core development/process and the fact that I am on vacation in the last week of January I assumed it is not very realistic to expect all that to land in such a short time.

Gábor Hojtsy’s picture

@slashrsm: sure, I don't think you answered the question though :) If we have media entity + file + image handlers in 8.3.x then we still need a way for sites to have the widgets. I believe in 8.x contrib so far widgets/formatters are bundled with he handlers, no? For 8.3.x we'd need a separated approach somehow for them to live in a dedicated contrib module, right? (I may be missing something sorry).

slashrsm’s picture

@Gábor Hojtsy: Ah! That. No, contrib mostly uses Entity browser and/or Inline entity form at the moment. There are no field widgets that would be bundled with the plugins. There are some entity browser plugins, but we'll move them to the main entity browser module when plugins are moved to core.

Gábor Hojtsy’s picture

@slashrsm: ok that makes sense. Then if there is a contrib version of the media library, then that will need to house those. But to avoid the mess with migrating a contrib module into core, it is probably simpler to just do the development in core direct (in a fork/sandbox for testing).

xjm’s picture

Thanks @slashrsm and @Gábor Hojtsy for the feedback. I agree with @slashrsm that the media entity, file, and image patches are what we should be targeting for the core API. (@effulgentsia said he did not think image needed to be, but in either case it's a small addition once the file media patch is complete. The main entity patch is certainly the biggest investment.)

I definitely agree that the timeline is now most likely too short to present a complete out-of-the-box user experience for media in core in 8.3.0. Even if we provide a minimally viable/"not super broken" user experience for the core modules, they're going to be very disappointing without further UI work or a feature like the media browser. There are so many integration points we would have to get right that are out of scope for the initial patch. It actually could hurt perception of the product; it would distract from the fact that there is a rich suite of media solutions in contrib and make it look like D8 as a whole is not usable yet.

All this is why I strongly encourage marking the module as hidden, even if it is in the release of 8.3.0. If we do that, we can treat it as an API addition, and give ourselves more time to get the UX MVP, documentation, etc. right. We still should make the UX minimally viable when the module is hidden and/or experimental, because it will still be exposed to and used by everyone who installs the contrib modules. But if the module is not hidden and/or experimental, then we must make it polished.

@slashrsm's two approaches in #29 are actually fairly similar at this point. The goal in both cases is to get an architecturally complete, finalized API into core as soon as possible, so that both core and contrib development have a shared, stable foundation. Once the code freeze for alpha starts, the target branch becomes 8.4.x instead of 8.3.x. However, it's worth noting we recently clarified our policy about what can be committed during these phases:

Core developers should plan to complete changes that are only allowed in minor releases prior to the alpha release for inclusion in that minor. Following the alpha release, such changes will be committed primarily to the next development branch instead.

There are no specific restrictions on what changes are allowed in alpha releases beyond the allowed changes for minors, but additions and disruptive changes are backported to the alpha only at committer discretion, with criticals, strategic initiatives, and product management priorities as the first focus.

(From https://www.drupal.org/core/d8-allowed-changes#alpha)

If review concerns, architecture, etc. are addressed and all three patches is committed to 8.4.x between alpha and beta, and if they are marked hidden, then at that time we can evaluate whether to backport them to 8.3.x together. The question is what will be best for core, contrib, and site owners. There are pros and cons for backporting. For example:

  1. Pro: If the APIs are in 8.3.0, contrib modules can also have stable branches depending on them after April 5.
  2. Pro: Contrib will not have to maintain two versions from February until 8.4.0 in October, only for the two months between February and April.
  3. Pro: Once the File/Image types are added, site owners who also install contrib media modules can begin creating their data with the new data model, so there will be less data overall that needs to be migrated from the core type.
  4. Pro: The investment we have made in Media so far would be proven, and the core product would be committed (pun!) to it, as @slashrsm expressed earlier, because we shipped it in a core release. This could also help incentivize further resources for the initiative (which @slashrsm also raised above as a need).
  5. Pro: No more rerolls of a 200K patch or managing sandbox merges. :P
  6. Con: As Gábor pointed out, "we added something you can't see to core" does not have much marketing value on its own.
  7. Con: Gábor also raised a concern that it might mean there would not be enough advantage to justify the work contrib would need to do to port to the new API.
  8. Con: If we do get something wrong in the API, we have to provide BC for it anyway.
  9. Con: Even if the API is in 8.3.x before beta, other API additions on top of it can still only go into 8.4.x, and so 8.3.x and 8.4.x could diverge, which means potential added work for backporting bugfixes, etc.
  10. Con: There is more risk for core's minor release, because we'll be using the absolute minimum allowed window for flushing out bugs.
  11. Etc.

That list is not exhaustive, and it will be hard to guess what will be best until the first patches are actually committed to the dev branch, whether it's 8.3.x or 8.4.x at the time those commits happen. And it also becomes irrelevant if the patch is not really, really ready. So my recommendation is that we continue to really focus on the API side, and if the API is ready before beta, then have a meeting with initiative members, contrib maintainers, and core committers to decide together whether to backport it to 8.3.x as a hidden module. I'm going to keep this tagged for release manager review until we have a chance to evaluate the status of the initial commits and have that discussion. Until then, though, let's focus on getting the initial API right, while planning to iterate on the user-facing core featureset in the longer term.

Edit: grammar.

Gábor Hojtsy’s picture

Given the severe architectural concerns raised by @WimLeers 3 days ago and the resulting rewrites he is heavily working on still resulted in mostly an entire reorganization and rewrite of all corners of the media entity patch, I don't think the argument of the proven architecture relied on by a dozen contributed modulesholds anymore. I think it would be dangerous to try to get a stable module in core which just got rearchitected from the ground up a couple weeks before a deadline, because there is no way we can prove with the other dozen modules in this timeframe that the rewrite is stable enough.

I think the only way forward then is to target this to 8.4 exclusively. I still think it would be nice to settle the base entity module sooner than later because then work can commence on the plugins and widgets and formatter and library, but then there is no deadline attached anytime soon.

I also think then we need to mark this issue Needs work, but since I made several mistakes in how this issue was marked, I am not going to change status again.

Wim Leers’s picture

I was just informed that before I came in it was mostly cosmetic changes, but also fixing incorrect usage of the plugin system etc, so not only cosmetic.

I'd almost suggest adding the media_entity module as-is to Drupal core, with only adding test coverage and improving documentation. Without renaming anything. Without any other changes. Because then we would be actually putting a stable module into core, without changing any API.

But apparently it's going to need fixes anyway, so the API is going to be changing anyway.

Gábor Hojtsy’s picture

Yeah I think contrib needs updating for the version of the proposed media module anyway, so doing the update twice is not a good plan to work with (to update to proposed changes API again). So its better to update once.

slashrsm’s picture

But apparently it's going to need fixes anyway, so the API is going to be changing anyway.

If there are any fixes required then they need to be done initially. There are almost 8000 sites using media entity as of today. We can not afford to mess with those sites in an irresponsible manner. Then there are at least 20 modules in contrib that build on top of it (+ who knows how many custom). We can't force them into a chasing-core hell. If we do that they simply won't follow and media entity will effectively stay in contrib no matter what we do. And we'll end up in a D7 like situation. Not what we want.

I can't agree that the current architecture is as bad as @Wim Leers is concerned. There are two main reasons for that: a) it is proven in contrib; there are 3 years of work invested into it and some of the questions that we're dealing with now have been discussed in the early stages of development already. There are many live websites using it, including Thunder and Lightning. b) We tried to improve the architecture in #2831274: Bring Media entity module to core as Media module, came up with two alternative solutions and none of them ended up being better than what we currently have.

Contrib has gone its way in D8. As I said many times already; it works and one could argue that there is not much benefit from adding moving it to core. We all know that this is not true as core needs to lead. But if it will to far contrib simply won't follow. I think that we want to avoid that.

Wim Leers’s picture

The architecture is NOT bad. I have serious concerns about a SUBSET.

See https://www.drupal.org/node/2831274#comment-11878883 for more nuance.

Gábor Hojtsy’s picture

On #2835825: Media: Migration integration @xjm posted this release management feedback which IMHO concerns the overall approach to requirements and how/when things land, so posting a response here.

@xjm's comment there

If Media does not have a migration destination and becomes stable, then that moves Migrate Drupal further from being stable. We should be requiring migration destinations for all added storages at this point. They will also help for migrating core image and file data into media entities once they are stable. So, I think we need to have the migration destinations available for Media to be stable (a "must" have on the initiative roadmap).

However, migrations are not a blocker for adding the API to core so long as the module is still in beta or earlier. For media, we agreed that the module itself could considered a stable API addition but experimental otherwise, so long as the module remains marked hidden. The same should apply here. So we can postpone this issue on finalizing the initial API and on finalizing the file and image types (although we may consider incling the migation destinations for files and images in those patches). Marking postponed on the initial entity issue, at least.

My response

Trying to process your comment as to what options are there for Media module vs. this media migration implementation (and possibly other things). Maybe there are these two options if I understand it right.

1. Get media entity committed as stable (in 8.4.x) without migration integration BUT media entity marked as hidden. Implementing migration integration would then be part of the must haves to mark it non-hidden.

2. Get media entity committed as experimental (in 8.4.x) without migration integration. Implementing migration integration would then be part of the must haves to mark it stable (still possible before 8.4.0(?)).

For option 2, I looked at https://www.drupal.org/core/experimental and it says

Experimental modules may only become stable modules in minor or major releases.

which in my reading does not rule out adding an experimental module and making it stable within the same dev cycle before a new minor is released. It only rules out making an experimental stable in a patch release, which is not an option the media team wants anyway.

The option 1 case is a brand new way to add an API module, at least I don't remember we did this before, so we may need to document why and if any policies need to be adjusted for this process, so people understand what we are doing. The option 2 case at least maps to an existing process, but in the option 2 case though, we need to keep the "early" rollback scenario in mind if the module does not reach stable by the minor release.

IMHO the hidden module idea was useful when we were talking about getting this into 8.3 as stable without useful user features. Not sure it is useful as part of the development of 8.4.0-dev since we DO want people to try it out and use it while 8.4.0 is being developed as opposed to hiding it from developers.

xjm’s picture

Thanks @Gábor Hojtsy.

Personally, I would still prefer to call it beta experimental when we commit it, and just clarify in our communications, the experimental handbook page, etc. that the API and data model are stable, but other aspects are not. But @slashrsm and @Berdir had concerns that that would not be a strong enough stance for contributed modules that need to depend on it and have stable releases. That's why the hidden module workaround is the next best thing.

I guess we could have a "Must" have issue to mark the module as hidden if it's not stable in all aspects by 8.4.0-beta1. The risk there is that 8.4.x is not shippable if we need to cut an off-schedule minor for some reason, so in general I don't want to add required issues like that.

so we may need to document why and if any policies need to be adjusted for this process, so people understand what we are doing

This plan is a special case because of the unique needs of media contrib modules, so I'd prefer not to make any overall policy changes. After all, we decided the policies to meet our needs and it's documented that we can make exceptions to them also when it also meets our needs. :)

which in my reading does not rule out adding an experimental module and making it stable within the same dev cycle before a new minor is released. It only rules out making an experimental stable in a patch release, which is not an option the media team wants anyway.

Definitely. This is actually what I wanted to do initially in 8.3.x anyway: Add it as beta experimental in November, and mark it stable by release if at all possible. We can still do that for 8.4.x.

So what I suggest is:

  1. Commit the initial API to 8.4.x-dev as a beta experimental module (in the core experimental package), but not hidden, and communicate to all contributed modules that we will provide full BC and upgrade paths for the API and data starting sometime before 8.4.0.
  2. If the module is ready to be stable by 8.4.0-alpha1 (per the stabilization requirements we have defined already, must-haves completed etc.), then we just make it stable the normal way.
  3. If the module is not ready to be stable by 8.4.0-alpha, instead switch it to hidden: true, but still move it out of the experimental package at that time.
  4. If the module gets to stable between 8.4.0-alpha1 and 8.4.0, simply remove the hidden: true at that time.

BTW the migrations are just one of several must-haves we will have as followups to the media entity patch. See #2795833: [plan] Add layouts to entity displays (both form and view) for an example of a followup roadmap. I don't want to take up initiative contributors' time sorting through these things while we are still finalizing the architecture, so I am gathering notes as I do my own review of the initial patch and will try to help update the roadmap here and in #2801277: [META] Support remote media assets as first-class citizens by adding Media Entity module in core once I'm done. (It will have the categorization of those followups into Must--things like UX gate issues, Angie's product manager requirements in #22, my recommendation about migrations, etc.--Should, Could, and the steps I've outlined here if that plan works for initiative contributors and contrib.) Tagging for that summary update once I get to it.

The bottom line is that even dev branches need to be in a state where we could release them today if we needed to, but we can still make things smooth and reliable for contrib.

Edits: clarifications and not contradicting myself (I hope).

Gábor Hojtsy’s picture

That makes sense to me IMHO, it gives us a way out if we do need to release a minor out of schedule (which I don't see in which case would be necessary but that does not matter here and does not change anything :).

slashrsm’s picture

I agree with the plan that @xjm outlined in #42.

Gábor Hojtsy’s picture

Issue summary: View changes

Updated version numbers to 8.4.0 from 8.3.0.

Gábor Hojtsy’s picture

Issue summary: View changes

And two more places I missed.