Problem/Motivation

Core comes with some basic support for local files. With "local files" we usually refer to files that are stored on the same server as a given website is hosted from. That usually works fine for images. A lot of other media asset types are often remote nowadays. "Remote media" usually means media assets that are hosted on 3rd party services and also act as standalone resources on the internet, but they can incorporated in other resources too (usually by using embed codes). Examples of such remote resources are YouTube videos, Instagram photos, SoundCloud audio assets, Tweets, Facebook posts, ... Drupal core doesn't provide any support for those.

Contrib space has been solving this problem on its own for ages. In Drupal 7 and earlier we had many different solutions that approached it in a slightly different way. In Drupal 8 we standardised on a single solution. Media entity provides storage solution for local and remote media, handles metadata, thumbnails, business logic and validation in a standardised way. It was adopted by distributions like Lightning and Thunder and it is on a good way to become one of the most popular Drupal 8 modules.

Main problem in the current Drupal 8 situation it the lack of guidance from core. Since there is absolutely nothing in it contrib needs to make all decisions on its own. If, on the other hand, core would provide some basic solutions we could build on that in contrib with much more confidence.

Proposed resolution

Design core solution for handling of the remote media. Ideally rely on solutions that already exist in contrib space and build on top of that.

Remaining tasks

  1. Resolve #2669802: Add a content entity form which allows to set revisions , which is currently blocking this (media_entity depends on Entity API because of it)
  2. Move media_entity to core (more or less as it is), see #2831274: Bring Media entity module to core as Media module
  3. #2831936: Add "File" MediaSource plugin
  4. #2831937: Add "Image" MediaSource plugin
  5. Define default bundles (similar to what @Sam152 proposed in #2786785-48: Media in Drupal 8 Initiative) - part of the two "plugin" issues above for now
  6. #2831940: Create file field widget on top of media entity
  7. #2831941: [PP-1] Re-create Image field widget on top of media entity
  8. Ideally re-think field widgets UX and improve it along the way. There are some issues about that already. See: #2113931: File Field design update: Upload field.
  9. #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode

At this point we achieved core feature parity. In order to show what is supported by the media entity and to have an example implementation in core also:

  1. #2831944: Implement media source plugin for remote video via oEmbed
  2. Create field widget for it.
  3. Create formatters for it.

Finally, the upgrade path from contrib to core. @phenaproxima explained this in #18:

As per @Wim Leers' comment over at #2831936-148: Add "File" MediaSource plugin, let me explain a little about the plan for creating an upgrade path from Media Entity to core Media.

At the December 2016 media sprint in Berlin, we made a conscious decision to keep the data model of Media entities consistent between Media Entity and core Media. The purpose of this was twofold -- first, the data model was pretty tight as it was, and there wasn't really anything we felt needed to change about it. Secondly, we wanted to keep the upgrade path as smooth as possible, and messing with the data model is always a squirrelly proposition.

So really, the upgrade path from Media Entity to Media is mostly about API changes. You have to change any code that integrates with Media Entity to integrate with Media instead -- which have very similar APIs, though with some relatively minor differences. The biggest data-layer change involves the renaming of media_bundle config entities to media_type, but Media Entity 2.0 will provide an update hook to make that change.

Speaking of which -- Media Entity 2.0 is intended to be a simple upgrade path to core Media. That's all it is; just a bridge to get people (and contrib modules) onto core Media. It will provide no new functionality.

User interface changes

TBD

API changes

TBD

Data model changes

TBD

Comments

slashrsm created an issue. See original summary.

effulgentsia’s picture

Thank you for this issue!

The parent issue's summary says:

Media entity in core for 8.3.0 - #2801277: Support remote media assets as first-class citizens
Move Media entity module in core (not as experimental module). Re-create features that core offers at the moment on top of it (local images and files)...When all this pieces are ready change standard profile to ship with Media entity based solution by default. Don’t force existing sites to migrate. They can keep using files...

So, I'm wondering what the scope of this issue should be. Is it just to move Media entity to core, with more or less its existing feature set? Or should it also include changing Standard profile's field.field.node.article.field_image.yml and field.field.user.user.user_picture.yml from

field_type: image

to

field_type: entity_reference
settings:
  handler: 'default:media'
  handler_settings:
    target_bundles:
      image: image

?

And, if we do that, then a few questions:

  1. What changes do we need to make in the "Select a field type" drop down? Currently, within the "Reference" group, the options are "Content, File, Image, Taxonomy term, User, Other". Do we want to keep "File" and "Image" as primary option labels, but change them to actually create an ER field for a Media entity type with "File"/"Image" bundle? But then, we'd still need a way to add an actual file/image field type, for example, as fields within the Media bundles themselves. Which could maybe be somehow buried within "Other", but currently "Other" takes you to an ER field configuration, so where can we branch to instead choose a file/image field type if that's what you want? Or do we want to totally deprecate file/image field types, and then let the field within the Media bundle be an ER field for a File entity type?
  2. Are we concerned about the performance overhead of making all referenced local files require 2 entity loads (the media entity and the file entity) instead of 1? E.g., currently when displaying a comment with a user picture, we load 3 entities: the comment entity, the user entity, and the file entity for the user picture. With the above change, we'd now be loading 4 entities: the same 3 plus also the media entity for the user picture as well. If someone decides they don't have any desire for user pictures to show up in a media library, should it be a legitimate/supported feature for them to be able to make the field.field.user.user.user_picture.yml field be a direct image field or ER field to a File entity rather than having to make it an ER field to a Media entity? In other words, should we leave such a capability around not only for existing sites prior to some future forced migration, but also for new sites as well?
Sam152’s picture

I wrote about how some of the issues raised in #2 could be addressed.

slashrsm’s picture

My idea was to use parent issue to present general plan and to discuss technical details in sub-issues like this one. In order to achieve what this issue is proposing (and have feature parity with what core currently offers) I think we need to do the following steps:

1. Move media_entity to core (more or less as it is)
2. Define default bundles (similar to what @Sam152 proposed in #2786785-48: Media in Drupal 8 Initiative)
3. Create field widgets that work with "Image" and "Document" bundles and offer similar UX as current core's file and image formatters
3a. Ideally re-think UX and improve it along the way. There are some issues about that already. #2113931: File Field design update: Upload field. is one example.
4. Create formatters that offer feature partiy with current core's formatters.

At this point we achieved core feature parity. In order to show what is supported by the media entity and to have an example implementation in core also:

5. Create one remote video bundle. Probably YouTube. Could be implemented through oEmbed, which would technically support many more providers.
6. Create field widget for it.
7. Create formatters for it.

Or do we want to totally deprecate file/image field types, and then let the field within the Media bundle be an ER field for a File entity type?

Either that or maybe limiting file/image fields to media entity bundles through isApplicable().

If someone decides they don't have any desire for user pictures to show up in a media library, should it be a legitimate/supported feature for them to be able to make the field.field.user.user.user_picture.yml field be a direct image field or ER field to a File entity rather than having to make it an ER field to a Media entity?

To keep things consistent I'd rather still go with media entity. Media can be excluded from the library in other ways. Media module adds a "Show in library" boolean field on the media entity for example which is on by default.

Are we concerned about the performance overhead of making all referenced local files require 2 entity loads (the media entity and the file entity) instead of 1?

There is definitely some additional overhead, but I think that benefits outweigh it. This problem should also be mitigated by the fact that we now have entity (and super-duper render) cache in core.

slashrsm’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Title: Support remote media assets as first-class citizens » Support remote media assets as first-class citizens by adding Media Entity module in core
seanB’s picture

This would be great! I can definitly see the benefit of a more flexible base for media in core. As far as I'm concerned this doesn't have to replace the file/image fields, as long as they all store the data as media entities. Just having a media field with different widgets for different use cases could work just as well though.

There could be some debate over the bundles and modules it ships with out of the box. Should remote media be optional or not? Currently all media entity types are submodules. Having everything as submodules is the best solution imho, but this makes the whole media suite really complex. This is something to think about when implementing this. It might be good to trade some flexibility to make everything simple to install.

Besides that, I think the entity type should just be named 'Media', since all other entities don't have the _entity suffix.

slashrsm’s picture

There could be some debate over the bundles and modules it ships with out of the box. Should remote media be optional or not? Currently all media entity types are submodules. Having everything as submodules is the best solution imho, but this makes the whole media suite really complex. This is something to think about when implementing this. It might be good to trade some flexibility to make everything simple to install.

Types that core ships with don't need to be in submodules I think. Obviously there will be much more plugins available as contrib modules.

Besides that, I think the entity type should just be named 'Media', since all other entities don't have the _entity suffix.

I agree with this. However, we shouldn't forget about sites that are already using media_entity today. We want to allow smooth transition for those.

slashrsm’s picture

Title: Support remote media assets as first-class citizens by adding Media Entity module in core » [META] Support remote media assets as first-class citizens by adding Media Entity module in core
Category: Feature request » Plan
Issue summary: View changes
Parent issue: #2786785: Media in Drupal 8 Initiative » #2825215: Media initiative: Roadmap
slybud’s picture

I am very happy to see this happening for better media management is clearly the next step to make Drupal core greater than ever.
As a project director with many big media projects developped and maintained (starting in D6, and now continuing on D8), I can only +1 this approach, which is scalable and more DAM-oriented, and on top of which we can build great UX.

Thanks @slashrsm and all the Media Entity/Initiative people involved for all the great work done so far and to be done !

Gábor Hojtsy’s picture

@slashrsm started a child issue for this at #2831274: Bring Media entity module to core as Media module to work on completing the first step and make media entity available in core. The conversion of other things related to it will still need more issues, therefore this being a META now.

Gábor Hojtsy’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes
slashrsm’s picture

Issue summary: View changes
Issue tags: +Berlin media sprint

Link to sub-issues

slashrsm’s picture

Issue summary: View changes
Boobaa’s picture

Issue summary: View changes

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

phenaproxima’s picture

As per @Wim Leers' comment over at #2831936-148: Add "File" MediaSource plugin, let me explain a little about the plan for creating an upgrade path from Media Entity to core Media.

At the December 2016 media sprint in Berlin, we made a conscious decision to keep the data model of Media entities consistent between Media Entity and core Media. The purpose of this was twofold -- first, the data model was pretty tight as it was, and there wasn't really anything we felt needed to change about it. Secondly, we wanted to keep the upgrade path as smooth as possible, and messing with the data model is always a squirrelly proposition.

So really, the upgrade path from Media Entity to Media is mostly about API changes. You have to change any code that integrates with Media Entity to integrate with Media instead -- which have very similar APIs, though with some relatively minor differences. The biggest data-layer change involves the renaming of media_bundle config entities to media_type, but Media Entity 2.0 will provide an update hook to make that change.

Speaking of which -- Media Entity 2.0 is intended to be a simple upgrade path to core Media. That's all it is; just a bridge to get people (and contrib modules) onto core Media. It will provide no new functionality.

Wim Leers’s picture

Issue summary: View changes

Lifted #18 into the IS for better discoverability.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

moshe weitzman’s picture

Any changes worth noting in the IS? I know we did Oembed since the last update.

phenaproxima’s picture

Issue tags: -D8Media +Media Initiative

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

seanB’s picture

Status: Active » Fixed

Since #2831944: Implement media source plugin for remote video via oEmbed and #2996029: Add oEmbed support to the media library have landed, I think we can say remote media is now supported in core. Yay!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.