Problem/Motivation

We have some limitations in the current approach for field mapping in the media module it at the moment. In comment #289 there was a proposal to move this forward. These are some examples of limitations from that comment:

  • It doesn't handle complex field types: currently the code takes metadata value and assigns it to the main property of the field. It doesn't handle fields that might rely on more than one property. An example of that would be the Link field with "uri" and "title" properties. Currently we can only set uri. There is a workaround for that: if getMetadata() is aware of this it can prepare metadata value in a right way. I don't think that getMetadata() should care what the destination of the value will be.
  • Pre-processing of the metadata value is not possible: Example are tags. They usually come in form of a comma-separated list. If you throw that into an entity reference field nothing useful will happen. You would generally want to preprocess that to convert that comma-separated list to a list of entities and use their IDs. Same workaround as in the first item applies.
  • Multi-value fields: Only single value fields are supported. Same workaround as in the first item applies.
  • Updating mapped values: Currently we map values on initial save only. If metadata changes upstream at some later time we won't update. There is currently no workaround for that (except custom code).

Proposed resolution

It seems that we could use a separate plugin type to solve this problem (MediaMetadataMapperInterface or ComplexFieldMappingInterface something along those lines). It is true that it isn't the job of the MediaSource to solve complex media-metadata-to-field mappings. By introducing a new plugin type we separate responsibilities and clearly document them.

Doing this would not break BC. This was explicitly validated. See #2831274-310: Bring Media entity module to core as Media module for the full explanation. Quoting #2831274-311: Bring Media entity module to core as Media module to make this very clear in this issue too:

I find #310 sufficiently convincing that I'd be in favor of postponing "complex field mapping" functionality to a follow-up.

It'd change this:

function preSave() {
  …
   // FIELD MAPPING LOGIC
   …
}

to:

function preSave() {
  …
  if ($this->getSource() instanceof ComplexFieldMappingInterface) {
    $this->getSource()->doComplexfieldMapping();
  }
  else {
    // FIELD MAPPING LOGIC
  }
   …
}

That's a trivial change, which does not require API changes.

Remaining tasks

- Discuss the mapping features.
- Implement mapping

User interface changes

API changes

TBD

Data model changes

CommentFileSizeAuthor
#10 step4.png205.83 KBseanB
#10 step3.png158.26 KBseanB
#10 step2.png209.55 KBseanB
#10 step1.png381.35 KBseanB
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

seanB created an issue. See original summary.

Wim Leers’s picture

slashrsm’s picture

It feels that this issue could address the problems that were brought up in #2855630: Field mapping not working properly in contrib.

geek-merlin’s picture

seanB’s picture

Here are some thoughts based on what I've learned so far. We need to support at least the following features (or at least be able to do it in the future):

  • Mapping metadata to simple text fields. This is already possible.
  • Mapping metadata to simple number fields. This is already possible.
  • Mapping metadata to simple boolean fields. This is already possible.
  • Mapping complex metadata to multivalue fields. Not sure where, but this was mentioned for sure.
  • Mapping metadata to taxonomy terms, creating new terms and/or reusing existing terms. You need to be able to configure a vocabulary.
  • Mapping metadata to datetime fields. This came up in #2831937-70: Add "Image" MediaSource plugin. You need to be able to configure the timezone.
  • Mapping metadata to entity references. This came up #2831274-291: Bring Media entity module to core as Media module. Probably related to taxonomy terms.

Right now we don't want metadata to care about storage. I do think we should have some configuration to make sure only logical mapping options are presented to the user. For instance it wouldn't make sense to map metadata containing a date to a boolean field. Since media sources define the metadata, it is probably the best place to store such configuration.

At a minimum we should require metadata to define a data type. For date we can allow text/number/date fields and handle mapping accordingly. For booleans we can allow text/number/boolean fields etc.

Like I said, just some thoughts, feedback is more than welcome!

geek-merlin’s picture

Some random thoughts:
* From a distant point of view, we want a generalized verison of computed_field (while that only computes / denormalizes a special field type, we want to compute arbitrary field types).
* Even if the concrete implementations are currently for media, there is nothing that prevents using this for other entity types (pluginification is #2879969: Make Media field mappings reusable between media types)
* Mapping metadata to taxonomy terms, creating new terms and/or reusing existing terms. - So it is not only about mapping, but some values want to be preprocessed.

My gut feeling what can happen is
a) we either pick some use cases and bake them in which feels a bit creepy and random but might just work(tm)
b) we build a something along the lines of feeds mapper and feeds tamper (huh)
c) we leverage the token system for field default values
d) ???

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

geek-merlin’s picture

@SeanB #5:

My first gut feeling was that the mapping is an important part of media, but in the current state a really odd child.

After some sleeping over this i think the most drupal-ish way is to leverage typed data api for this.
Imagine a configuration like this:

Map your fileds
---------------------

field_mimetype : [media.field_media_source.mimetype]

field_colorspace: [media.field_media_source.colorspace]

Leveraging typed data, we can stay typesafe, be it text, term or whatever. We just need property providers and no additional API.
We just need a typed-data browser (like rules has, or was proposed in #514990: Add a UI for browsing tokens for tokens)

We can even do something like:

Map your fileds
---------------------

field_colorspace_term: [media.field_media_source.colorspace.autoterm]

(Just last week i hacked that use case to have terms for extracted text properties.)

So
* Closed my #2879969: Make Media field mappings reusable between media types in favor of this
* Added #2924821: Help media with typed data browser? in Rules to ping fago&friends about this
* Added #2924819: Make Tokens a special case of Typed Data to further the idea

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

seanB’s picture

FileSize
381.35 KB
209.55 KB
158.26 KB
205.83 KB

We just had an interesting discussion about how this could look. Here is a summary:

  • Move mapping from media type form to separate tab
  • You can create new mappings from the 'Manage mappings' tab. This allows for multi-step mapping configuration, re-using existing functionality for creating fields as much as possible. This also allows metadata attributes to be mapped to multiple fields (which could be a nice bonus feature).
  • Media sources should be able to specify the field types for each metadata attribute. Mapping a name to een date field doesn't make sense and should not be possible.
  • To transform the metadata for different field types, we need something like metadata processors per field type. These could return a text when using a textfield, but fetch a target ID when using an entity reference. Through a weight and an isApplicable() method, modules can provide custom processor to do even more complex mappings. For entity references, this could lean on the entity reference selection plugins.
  • A metadata attribute can only map to existing fields of the supported type and if the field instance is not yet used by another metadata attribute. The field types should also have a metadata processor.
  • Newly created mapping fields should be disabled for the media form (since they are automatically filled when saving media). If a user adds them to the form manually, we should add a message in the description letting the user know the field data could be overwritten when saving media.
  • We only allow mappings to be deleted, editing a mapping would add more complexity without really adding value.

Some screens of how this could look.

STEP 1
STEP 2
STEP 3
STEP 4

geek-merlin’s picture

Nice writeup! That child looks to me as if the father is the feeds mapper and the mother a reinvention of typed data. ;-)

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

seanB’s picture

eelkeblok’s picture

Issue summary: View changes

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

phenaproxima’s picture

Priority: Normal » Major
Issue tags: +Media Initiative, +Triaged Media Initiative issue

This is a major feature request, so escalating the priority here.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.