Problem/Motivation

With #1921558: Convert file_get_mimetype() to use Symfony MimeTypeGuessers, the function file_mimetype_mapping() was removed. The mapping was placed as a protected property of the ExtensionMimeTypeGuesser class, as its main purpose was for the guesser to use it. But there are cases when the mapping, or just the list of MIME types, should be used in itself, outside the context of guessing. For instance, a form in the File entity module should present known MIME types for the user.

Proposed resolution

Move the default mapping to a new service and class, Drupal\Core\File\MimeType\MimeTypeMapper, with getter/setter methods.

Introduce an alterMapping() method which invokes the mimetype alter hook upon service instantiation to allow modules to play with MIME type<->extension mapping.

Change the ExtensionMimeTypeGuesser class to use the new mapper; deprecate its ::setMapping method.

Deprecate the file_mimetype_mapping alter hook.

Add tests.

Add tests.

Remaining tasks

Commit

User interface changes

None

API changes

A new service file.mime_type.mapper.

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Bug because functionality in previous version is now missing
Issue priority Major because it is a small regression. In D7 one could access the mapping out of the context of guessing, and could eg get a list of all known mimetypes . After #1921558: Convert file_get_mimetype() to use Symfony MimeTypeGuessers its impossible without extending the class, because the mapping is a protected property
Disruption None, a BC layer covers the case of custom/contrib classes extending from ExtensionMimeTypeGuesser.
CommentFileSizeAuthor
#164 2311679-nr-bot.txt90 bytesneeds-review-queue-bot
#148 2311679-nr-bot.txt10.61 KBneeds-review-queue-bot
#146 2311679-nr-bot.txt90 bytesneeds-review-queue-bot
#116 2311679-nr-bot.txt150 bytesneeds-review-queue-bot
#103 2311679-103.patch53.13 KBpcambra
#98 2311679-98.patch74.21 KByogeshmpawar
#93 2311679-93.patch74.13 KBmondrake
#93 interdiff_91-93.txt3.94 KBmondrake
#91 interdiff_88-91.txt5.92 KBmondrake
#91 2311679-91.patch71.16 KBmondrake
#88 2311679-88.patch71.85 KBmondrake
#80 interdiff_79-80.txt4.2 KBmondrake
#80 2311679-80.patch64.88 KBmondrake
#79 2311679-79.patch62.69 KBmondrake
#79 interdiff-77-79.txt680 bytesmondrake
#77 2311679-77.patch62.69 KBmondrake
#77 interdiff_75-77.txt873 bytesmondrake
#75 2311679-75.patch62.85 KBmondrake
#75 interdiff_73-75.txt4.35 KBmondrake
#73 interdiff_70-73.txt516 bytesmondrake
#73 2311679-73.patch62.97 KBmondrake
#70 2311679-70.patch62.98 KBmondrake
#70 interdiff_67-70.txt5.72 KBmondrake
#67 2311679-67.patch66.97 KBmondrake
#67 interdiff_65-67.txt1.71 KBmondrake
#65 2311679-65.patch66.84 KBmondrake
#65 interdiff_64-65.txt15.64 KBmondrake
#64 2311679-64.patch75.03 KBmondrake
#59 separate_mime_type-2311679-59.patch75.01 KBmondrake
#59 interdiff_54-59.txt1.41 KBmondrake
#54 separate_mime_type-2311679-54.patch74.24 KBmondrake
#54 interdiff_52-54.txt4.59 KBmondrake
#52 separate_mime_type-2311679-52.patch69.66 KBmondrake
#47 separate_mime_type-2311679-47.patch69.73 KBmondrake
#47 interdiff_43-47.txt6.35 KBmondrake
#43 separate_mime_type-2311679-43.patch68.16 KBmondrake
#40 separate_mime_type-2311679-40.patch68.19 KBmondrake
#40 interdiff_37-40.txt1.48 KBmondrake
#37 separate_mime_type-2311679-37.patch67.22 KBmondrake
#37 interdiff_36-37.txt4.9 KBmondrake
#36 separate_mime_type-2311679-36.patch67.89 KBmondrake
#36 interdiff_35-36.txt4.43 KBmondrake
#35 interdiff_32-35.txt2.51 KBmondrake
#35 separate_mime_type-2311679-35.patch65.53 KBmondrake
#32 separate_mime_type-2311679-32.patch64.96 KBmondrake
#32 interdiff_27-32.txt4.12 KBmondrake
#27 separate_mime_type-2311679-27.patch65.42 KBmondrake
#27 interdiff_25-27.txt6.05 KBmondrake
#25 separate_mime_type-2311679-25.patch64.28 KBmondrake
#25 interdiff_23-25.txt8.51 KBmondrake
#23 separate_mime_type-2311679-23.patch59.8 KBmondrake
#23 interdiff_8-23.txt8.21 KBmondrake
#18 separate_mime_type-2311679-18.patch54.63 KBmondrake
#18 interdiff_17-18.txt1.53 KBmondrake
#17 interdiff.txt4.21 KBParisLiakos
#17 separate_mime_type-2311679-17.patch53.97 KBParisLiakos
#16 separate_mime_type-2311679-16.patch53.93 KBmondrake
#16 interdiff_12-16.txt882 bytesmondrake
#12 interdiff.txt2.58 KBmondrake
#12 separate_mime_type-2311679-12.patch53.07 KBmondrake
#12 separate_mime_type-2311679-12-reroll-do-not-test.patch53.94 KBmondrake
#8 separate_mime_type-2311679-8.interdiff.txt4.63 KBarla
#8 separate_mime_type-2311679-8.patch57.03 KBarla
#4 separate_mime_type-2311679-4.interdiff.txt5.11 KBarla
#4 separate_mime_type-2311679-4.patch55.12 KBarla
#2 separate_mime_type-2311679-2.patch54.56 KBarla

Issue fork drupal-2311679

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

arla’s picture

Issue summary: View changes
arla’s picture

Assigned: Unassigned » arla
Status: Active » Needs review
StatusFileSize
new54.56 KB

This patch moves the mapping from ExtensionMimeTypeGuesser to the new MimeTypeMapper, which is a service, and lets ExtensionMimeTypeGuesser use that mapper.

ParisLiakos’s picture

thanks for opening the issue

  1. +++ b/core/lib/Drupal/Core/File/MimeType/ExtensionMimeTypeGuesser.php
    @@ -921,13 +58,13 @@ public function guess($path) {
    +  public function getMapper() {
    
    +++ b/core/modules/system/src/Tests/File/MimeTypeTest.php
    @@ -84,7 +84,7 @@ public function testFileMimeTypeDetection() {
    +    $extension_guesser->getMapper()->setMapping($mapping);
    

    I do not see the reason to have this public getter, just retrieve the mapper from the $this->container

  2. +++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php
    @@ -0,0 +1,921 @@
    +class MimeTypeMapper {
    

    Needs an interface, so that its swappable

arla’s picture

Adjusted from the feedback in #3.

berdir’s picture

  1. +++ b/core/lib/Drupal/Core/File/MimeType/ExtensionMimeTypeGuesser.php
    @@ -912,22 +49,11 @@ public function guess($path) {
         //   - awesome.image.jpeg
         while ($additional_part = array_pop($file_parts)) {
           $extension = strtolower($additional_part . ($extension ? '.' . $extension : ''));
    -      if (isset($this->mapping['extensions'][$extension])) {
    -        return $this->mapping['mimetypes'][$this->mapping['extensions'][$extension]];
    +      if (isset($mapping['extensions'][$extension])) {
    +        return $mapping['mimetypes'][$mapping['extensions'][$extension]];
           }
         }
    

    One thing that we were discussing was if and how much of this logic should move to the mapper?

    Right now, there is just getMapping(). But that's a very arbitrary structure, wouldn't it be nice to have getMimeTypes() too, if you just need those?

    And if we go there, would it be useful if the mapper would possibly even have a method like getMimeTypeForExtension($extension)? Then this would be re-usable if you'd just have a extension and not an actual file. Not sure if we should consider that at all or in a separate issue...

  2. +++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapperInterface.php
    @@ -0,0 +1,36 @@
    +  /**
    +   * Replaces the mimetypes/extension mapping.
    +   *
    +   * @param array|null $mapping
    +   *   Passing a NULL mapping will reset to the default mapping.
    +   */
    +  public function setMapping(array $mapping = NULL);
    

    Why do we need a setter? If you want to alter it, use the hook, this seems strange.

arla’s picture

2. I'm not sure why setMapping() would be needed, but it is tested in MimeTypeTest. I guess testing the alter hook would be more appropriate, and remove setMapping()?

ParisLiakos’s picture

totally agreed with 1.

2. is strange indeed and was there pre-conversion, see also #1921558-17: Convert file_get_mimetype() to use Symfony MimeTypeGuessers.
In other words, the hook will only fire if the default mapping is used: The setter overrides the hook.

I very much support to make the hook fire *always* and get rid of the setter...or the other way around

arla’s picture

Here the Mapper has also getMimeTypes() and getMimeTypeForExtension(). The test for custom mapping is removed. Is it desirable to add a test for the alter hook?

The alter hook is only run on the first call to getMapping(). Unless the guesser is used before registering all hook implementations, this should not be a problem.

Status: Needs review » Needs work

The last submitted patch, 8: separate_mime_type-2311679-8.patch, failed testing.

alexpott’s picture

mondrake’s picture

Priority: Normal » Major
Status: Needs work » Needs review
StatusFileSize
new53.94 KB
new53.07 KB
new2.58 KB

#2388749: Register symfony's mime guessers if they are supported has been committed, and it removed hook_file_mimetype_mapping_alter().

Rerolled patch in #8 (see do-not-test patch), and removed the alter call here too. Interdiff against the reroll.

Would love to see this in as there was no straightforward way in contrib to map extensions to mimetypes before #2388749: Register symfony's mime guessers if they are supported and also hacks like this

/**
   * {@inheritdoc}
   */
  public function getMimeType($extension) {
    return $this->mimeTypeGuesser->guess('dummy.' . $extension);
  }

now no longer (legitimately I'd say) work.

dave reid’s picture

What is the reasoning for removing the alter hook? This hook has been *very* useful for contrib to easily provide support for new file types when adding them to core would take much longer.

mondrake’s picture

Status: Needs review » Needs work

The last submitted patch, 12: separate_mime_type-2311679-12.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new882 bytes
new53.93 KB

Fixed PHPUnit test failure.

ParisLiakos’s picture

StatusFileSize
new53.97 KB
new4.21 KB

I dont think we should expose the mapping array, it is implementation details.
Instead lets provide methods.
Also added a method addMapping($mimetype, $extension) which adds/overwrites mappings.
If we agree on it i ll go ahead and write tests

mondrake’s picture

Related issues: +#2388749: Register symfony's mime guessers if they are supported
StatusFileSize
new1.53 KB
new54.63 KB

#17

I dont think we should expose the mapping array

+1, but then I suggest also to have a method to return supported extensions given a MIME type, that would be possible to be done elsewhere with the mapping array exposed, but not otherwise. Done here.

Also added a method addMapping($mimetype, $extension) which adds/overwrites mappings.

Great! It's what I was suggesting in #2388749-39: Register symfony's mime guessers if they are supported.

If we agree on it i ll go ahead and write tests

+1 :)

Status: Needs review » Needs work

The last submitted patch, 18: separate_mime_type-2311679-18.patch, failed testing.

alexpott’s picture

mondrake’s picture

#21 new patch coming soon.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new8.21 KB
new59.8 KB

New patch - combining parts of #8 and parts of #18. Interdiff against #8.

This keeps new methods introduced by @ParisLiakos in #17 and by me in #18, and keeps (but changed to protected) the getMapping method in #8 which is needed to allow the alter call to hook_file_mimetype_mapping_alter(). I'd say it would be probably better to change the signature of the hook to pass the MimeTypeMapper object and use addMapping() in hook implementations instead of letting hooks manipulate the array by reference, as pointed out in #17. But I guess that needs discussion.

Also added some tests for MimeTypeMapper::getExtensionsForMimeType() and MimeTypeMapper::addMapping().

ParisLiakos’s picture

+++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php
@@ -0,0 +1,944 @@
+    if (!in_array($mimetype, $this->getMapping()['mimetypes'])) {
...
+    $key = array_search($mimetype, $this->getMapping()['mimetypes']);
...
+    $extensions = $this->getMapping()['extensions'];
+    return isset($extensions[$extension]) ? $this->getMapping()['mimetypes'][$extensions[$extension]] : NULL;
...
+    if (!in_array($mimetype, $this->getMapping()['mimetypes'])) {
...
+    $key = array_search($mimetype, $this->getMapping()['mimetypes']);
+    $extensions = array_keys($this->getMapping()['extensions'], $key, TRUE);

lets only call getMapping() once and store it in a local variable instead of doing unnecessary function calls in the same method

I'd say it would be probably better to change the signature of the hook to pass the MimeTypeMapper object and use addMapping() in hook implementations instead of letting hooks manipulate the array by reference

Good idea!

mondrake’s picture

StatusFileSize
new8.51 KB
new64.28 KB

#24 done

ParisLiakos’s picture

  1. +++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php
    @@ -0,0 +1,946 @@
    +    $this->getMapping();
    

    can we add a comment here? why this is needed

  2. +++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapperInterface.php
    @@ -0,0 +1,57 @@
    +  public function getMimeTypes();
    

    and maybe add a small test case for this method

  3. +++ b/core/modules/system/file.api.php
    @@ -102,25 +102,30 @@ function hook_file_url_alter(&$uri) {
    + * Invoked by \Drupal\Core\File\MimeType\MimeTypeMapper::getMapping(). It
    ...
    + * should use the \Drupal\Core\File\MimeType\MimeTypeMapper::addMapping()
    ...
    + * \Drupal\Core\File\MimeType\MimeTypeMapper::getMimeTypeForExtension() and
    + * \Drupal\Core\File\MimeType\MimeTypeMapper::getExtensionsForMimeType()
    ...
    + * @see \Drupal\Core\File\MimeType\MimeTypeMapper::addMapping()
    + * @see \Drupal\Core\File\MimeType\MimeTypeMapper::getMimeTypeForExtension()
    + * @see \Drupal\Core\File\MimeType\MimeTypeMapper::getExtensionsForMimeType()
    

    Maybe those docs should refer to the interface methods ;)

mondrake’s picture

StatusFileSize
new6.05 KB
new65.42 KB

#26 done.

#26.1 - this is needed to make sure that alter hook is invoked before adding any mapping. Maybe I am overcomplicating, but it could be a case that code outside of the hook implementation add a mapping, then the first time any other method is called the hook would fire and may override that mapping.

Also fixed a couple of doc issues.

ParisLiakos’s picture

Nice, thanks..so i guess all we need now is a beta evaluation template and updating https://www.drupal.org/node/2258015 with alter hook change.
other than that, this is rtbc

mondrake’s picture

#28 updated change record draft

https://www.drupal.org/node/2258015/revisions/view/8067869/8077401

can anybody take care of the beta eval?

ParisLiakos’s picture

Category: Task » Bug report
Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

thank you
I think this is a bug since its a regression

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php
    @@ -0,0 +1,961 @@
    +    if (!$this->isMappingAltered) {
    ...
    +    $this->getMapping();
    

    There's a weird recursion going on here when the alter hook is fired.

  2. +++ b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php
    @@ -0,0 +1,961 @@
    +
    +    return TRUE;
    

    This seems pointless - just return $this for a fluent interface.

  3. I really like the new way of adding mappings in the alter - much nicer.
  4. +++ b/core/modules/system/file.api.php
    @@ -102,25 +102,31 @@ function hook_file_url_alter(&$uri) {
      * Alter MIME type mappings used to determine MIME type from a file extension.
    

    This is not quite correct anymore.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new4.12 KB
new64.96 KB

Thanks @alexpott.

#31.1 - different approach here, alter hook is invoked at construction time before any method is called. Not sure it is OK.
#31.2 - done
#31.3 - :)
#31.4 - changed

also changed a small doc issue 'Contains \Drupal...' instead of 'Contains Drupal'

ParisLiakos’s picture

no, we should avoid doing stuff on the constructor other than injecting dependencis, especially invoking a hook.

There's a weird recursion going on here when the alter hook is fired.

which is fine since

$this->isMappingAltered = TRUE;

is called before the hook is called, so the function will just return $this->mapping, it wont do anything else.

alexpott’s picture

I agree that doing the alter in the constructor is wrong. The alter should only be fired when necessary but also the recursive call during an alter is also weird. Plus we've lost the ability to remove mappings.

mondrake’s picture

StatusFileSize
new65.53 KB
new2.51 KB

We are indeed facing recursion here, because if we call e.g. getMimeTypeForExtension, this will call getMapping which, on first run, will invoke the hook. Hook implementations then will call e.g. addMapping who in turn will call again getMapping. So here we need measures to avoid endless loops. I made a change here to make things I hope more explicit, starting back from #27. If any idea how to circumvent this, feel free :)

Next - adding removeMapping and removeMimeType, will post a patch soon.

mondrake’s picture

StatusFileSize
new4.43 KB
new67.89 KB

Added ::removeMapping() and ::removeMimeType() methods.

mondrake’s picture

StatusFileSize
new4.9 KB
new67.22 KB

I found an alternative, i.e. using the 'calls' parameter in service definition to execute the alter hook separately from other methods, but not in the constructor (kind of 'setter injection' according to Symfony docs). Sorry, I'm learning here...

alexpott’s picture

@mondrake that looks like a really nice solution.

mondrake’s picture

#2260061: Responsive image module does not support sizes/picture polyfill 2.2 just introduced the below

+++ b/core/modules/responsive_image/responsive_image.module
@@ -328,17 +445,38 @@ function responsive_image_get_image_dimensions($variables) {
+  // The MIME type guesser needs a full path, not just an extension, but the
+  // file doesn't have to exist.
+  $fake_path = 'responsive_image.' . ImageStyle::load($image_style_name)->getDerivativeExtension($extension);
+  return Drupal::service('file.mime_type.guesser.extension')->guess($fake_path);
+}

which is OKish, but a workaround. With the patch here this could become

  $extension = ImageStyle::load($image_style_name)->getDerivativeExtension($extension);
  return Drupal::service('file.mime_type.mapper')->getMimeTypeForExtension($extension);

is it something to be added to this issue?

mondrake’s picture

Based on @yched's comment in #2426757-5: responsive_image_build_source_attributes() calls mime type guesser over and over, added #39. Also added a strtolower($extension) in MimeTypeMapper::addMapping to keep consistency with other methods.

Status: Needs review » Needs work

The last submitted patch, 40: separate_mime_type-2311679-40.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new68.16 KB

Rerolled.

mondrake’s picture

Any reviewers?

We were at RTBC back in #30 and @alexpott's feedback in #31 was addressed, see #38 also.

BTW we also have a published change record which is giving this for done, so it would have to be at least set back to draft while this is sorted out.

mondrake’s picture

Issue summary: View changes
StatusFileSize
new6.35 KB
new69.73 KB
  • Added lazy: true tag to the file.mime_type.mapper service, to match mime type guessing services definition
  • fixed a glitch in the responsive_image module
  • Fixed and added more dcoumentation to hook_file_mimetype_mapping_alter
  • Changed String::format to SafeMarkup::format in MimeTypeTest
slashrsm’s picture

Issue tags: +sprint
dave reid’s picture

Issue tags: -sprint +D8Media

Status: Needs review » Needs work

The last submitted patch, 47: separate_mime_type-2311679-47.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new69.66 KB

Reroll.

Status: Needs review » Needs work

The last submitted patch, 52: separate_mime_type-2311679-52.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new4.59 KB
new74.24 KB

Added/updated the proxy classes.

arla’s picture

Assigned: arla » Unassigned
mondrake’s picture

IMO, this needs decision before RC:

  • the changes to hook_file_mimetype_mapping_alter() signature (passing a MimeTypeMapperInterface object instead of the internal mapping array) will be hard to get in during 8.0 because of BC
  • the published change record give the changes in this issue for done, so if this gets postponed, the CR will have to be reverted in some parts

Status: Needs review » Needs work

The last submitted patch, 54: separate_mime_type-2311679-54.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new1.41 KB
new75.01 KB

Rerolled, conficts due to #2534066: Allow selecting the original image when creating a responsive image style, then made the logic in responsive_image_get_mime_type simpler to follow, and added comments.

mondrake’s picture

mondrake’s picture

This is a soft blocker for ImageMagick, see #2612590: Allow configuring in the UI the image formats supported by the toolkit. It would allow the ImageMagick toolkit to be configurable in terms of image formats accepted. Now the supported file extensions are hardcoded in ImagemagickToolkit::getSupportedExtensions() similarly to GD core toolkit, but the difference with ImageMagick is that if you want to add support for an image format you do not need to write specific code for that format as in GD - you just have to have a version of the executable that supports the format you need.

mondrake’s picture

Version: 8.0.x-dev » 8.2.x-dev

Status: Needs review » Needs work

The last submitted patch, 59: separate_mime_type-2311679-59.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
Issue tags: -Contributed project soft blocker +Contributed project blocker
StatusFileSize
new75.03 KB

Just a reroll. However this is now no longer acceptable I think, it breaks BC. This issue is still blocking a stable release for the ImageMagick module.

mondrake’s picture

StatusFileSize
new15.64 KB
new66.84 KB

An attempt to move on without BC break. This restores the hook_file_mimetype_mapping_alter to its original signature, still keeping the new service file.mime_type.mapper.

Status: Needs review » Needs work

The last submitted patch, 65: 2311679-65.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new1.71 KB
new66.97 KB

Status: Needs review » Needs work

The last submitted patch, 67: 2311679-67.patch, failed testing.

alexpott’s picture

+++ b/core/core.services.yml
@@ -1554,6 +1554,11 @@ services:
+    lazy: true

Are we sure this needs to be lazy? Given this is already injected into a lazy service that always needs it I'm not sure.

mondrake’s picture

StatusFileSize
new5.72 KB
new62.98 KB

@alexpott thanks, patch here removes lazy flag and proxy class.

mondrake’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 70: 2311679-70.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
Issue tags: -rc deadline
StatusFileSize
new62.97 KB
new516 bytes

Whoops..

Status: Needs review » Needs work

The last submitted patch, 73: 2311679-73.patch, failed testing.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new4.35 KB
new62.85 KB

Let's see this.

mondrake’s picture

Issue summary: View changes
mondrake’s picture

StatusFileSize
new873 bytes
new62.69 KB

Removed no longer needed file docblocks. Anyone up for a review?

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mondrake’s picture

mondrake’s picture

StatusFileSize
new64.88 KB
new4.2 KB

A problem with #79 was that, even if existing API were not changed, still any class that was extending from ExtensionMimeTypeGuesser and trying to access its protected $mapping and $defaultMapping properties would fail, since the mapping is moved to the new class.

Here I am using a magic __get method in ExtensionMimeTypeGuesser, so that any get attempt from ExtensionMimeTypeGuesser::$mapping or ExtensionMimeTypeGuesser::defaultMapping will be redirected to the mapper service, and fetched through two public methods there, introduced for BC and marked @internal.

mondrake’s picture

Issue summary: View changes
mondrake’s picture

This issue is pretty much stalled :(

I wonder whether it would make sense to take a different approach in the short/medium term, i.e. just add the methods that would be needed in contrib to the ExtensionMimeTypeGuesser service:

getMapping()
addMapping($mimetype, $extension)
removeMapping($extension)
removeMimeType($mimetype)
getMimeTypes()
getMimeTypeForExtension($extension)
getExtensionsForMimeType($mimetype)

and postpone introducing the separate service MimeTypeMapper to D9.

Any feedback?

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mondrake’s picture

Assigned: Unassigned » mondrake
mondrake’s picture

StatusFileSize
new71.85 KB

Two years later... I would still like this to move on. Getting the list of available MIME types or the list of file extensions mapped to a MIME type is not possible, and workarounds are not good.

In the ImageMagick module, at one point I was extending a class from the guesser, but this was forcing the alter hook to be called again. Lately I am using reflection to get the value of the $mapping array after it has been processed by the alter hook, but again this smells bad.

Relaunching here. This patch (sorry no interdiff):

  1. Defines a new file.mime_type.mapper service, implemented through a Drupal\Core\File\MimeType\MimeTypeMapper class (with a supporting Interface), that advocates the execution of the hook_file_mimetype_mapping_alter hook and provides OO methods to retrieve and manipulate the MIME type to extension mapping array.
  2. Provides a Kernel test for that.
  3. Changes the ExtensionMimeTypeGuesser class to use the new mapper; the ::setMapping method is deprecated, and deprecation tests are added.
  4. Changes the signature of the hook_file_mimetype_mapping_alter hook to pass the mapper object in addition to the $mapping array. This for BC, in D9 the $mapping array should be removed. Hook implementations can, as of the release after which the changes are made, either manipulate the array directly or use the OO methods to alter the mapping.
andypost’s picture

+++ b/core/modules/file/tests/file_test/file_test.module
@@ -291,16 +292,13 @@ function file_test_file_url_alter(&$uri) {
-function file_test_file_mimetype_mapping_alter(&$mapping) {
+function file_test_file_mimetype_mapping_alter(&$mapping, MimeTypeMapperInterface $mime_type_mapper) {

this means no BC, so not sure it is possible in minor versions

mondrake’s picture

Status: Needs review » Needs work

OK, I agree it is confusing. Let me do sth else.

mondrake’s picture

Status: Needs work » Needs review
Issue tags: +Needs tests
StatusFileSize
new71.16 KB
new5.92 KB

Instead of changing the hook signature, this is now deprecating hook_file_mimetype_mapping_alter and replacing it with a hook_mimetype_alter, where the mapper is passed as an argument.

If we agree this is the sustainable way forward, I will add a test for the hook deprecation.

mondrake’s picture

Issue summary: View changes

Updated IS.

mondrake’s picture

Issue tags: -Media Initiative, -D8Media, -Contributed project blocker, -Needs tests
StatusFileSize
new3.94 KB
new74.13 KB

Added deprecation tests for hook_file_mimetype_mapping_alter.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mondrake’s picture

FTR, if anyone interested, I released today a beta1 version of the Sophron module:

Features

  • Enhances Drupal's MIME type detection based on file extension to recognise 1200+ MIME types from 1600+ file extensions (vs Drupal's 360 MIME types and 475 file extensions).
  • Provides an extensive MIME type management API through FileEye/MimeMap.
  • Optionally replaces Drupal's core MIME type extension-based guesser.
mondrake’s picture

Assigned: mondrake » Unassigned
Status: Needs review » Needs work
Issue tags: +Needs reroll
yogeshmpawar’s picture

Assigned: Unassigned » yogeshmpawar
yogeshmpawar’s picture

Assigned: yogeshmpawar » Unassigned
Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new74.21 KB

Re-roll of #93.

Status: Needs review » Needs work

The last submitted patch, 98: 2311679-98.patch, failed testing. View results

yogeshmpawar’s picture

Status: Needs work » Needs review

Seems like these test fails are unrelated so setting back to Needs Review & Triggering bots.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

avpaderno’s picture

Status: Needs review » Needs work
pcambra’s picture

Status: Needs work » Needs review
StatusFileSize
new53.13 KB

Re-rolled as patch was not applying on core.services.yml
Patch is smaller due to the similarity from ExtensionMimeTypeGuesser with MimeTypeMapper

diff --git a/core/lib/Drupal/Core/File/MimeType/ExtensionMimeTypeGuesser.php b/core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php
similarity index 86%
copy from core/lib/Drupal/Core/File/MimeType/ExtensionMimeTypeGuesser.php
copy to core/lib/Drupal/Core/File/MimeType/MimeTypeMapper.php

Status: Needs review » Needs work

The last submitted patch, 103: 2311679-103.patch, failed testing. View results

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

This was a bugsmash triage target. kim.pepper agreed it was a bug. So let's get a reroll here.

Bhanu951 made their first commit to this issue’s fork.

bhanu951’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
bhanu951’s picture

One last test failing.

ExtensionMimeTypeGuesserDeprecationTest::testConstructorDeprecation test is failing due to ExtensionMimeTypeGuesser::$mapping or ExtensionMimeTypeGuesser::$defaultMapping are not being redirected to the mapper service.

I am not sure if we should add setters or getters as added in #80 to maintain backwards compatibility.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new150 bytes

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

kim.pepper made their first commit to this issue’s fork.

kim.pepper changed the visibility of the branch 11.x to hidden.

kim.pepper’s picture

Status: Needs work » Needs review

Did a fair amount of cleanup on this issue.

  • ExtensionMimeTypeGuesser is now using the MimeTypeMapper
  • Removed the getMapping() method which was temporary anyway and refactored tests to not call it
  • Split up tests and moved them to the Drupal\KernelTests\Core\File\MimeType namespace to make the namespace under lib/
  • Handle NULL returned from ExtensionMimeTypeGuesser::guessMimeType()
  • Updated the changes to the default map that went in since this was last worked on a year ago
  • Added typehints
  • Added more dependency injection
  • Fixed linting errors
  • Updated the deprecation message to refer to 11.1.0 and 12.0.0
kim.pepper’s picture

mfb’s picture

Why not allow contrib modules to call getMapping() so they can arbitrarily alter the mapping?

kim.pepper’s picture

Can't already alter the mapping with the alter hook?

kim.pepper’s picture

Also different mapping implementations could use different data structures, so what gets returned from getMapping() could be different.

mfb’s picture

Well I'm looking at how to still allow File MIME module to work without using the deprecated hook_file_mimetype_mapping_alter(). It looks like the new alter hook doesn't allow a contrib module to get the mapping so it can make various changes, and then apply it via setMapping().

And different data structures? Well that would make it even more difficult for a contrib module to alter the mapping, lol sigh. I confess to not actually following this issue.

kim.pepper’s picture

function hook_mimetype_alter(?MimeTypeMapperInterface $mime_type_mapper = NULL) gives you the ability to add/remove/update mappings by using the API interface rather than the raw data structure, which is better IMO.

mfb’s picture

Yes I see it's possible to call addMapping(). But I'd have to iterate through and make hundreds of method calls (if someone configures File MIME module to parse their /etc/mime.types file), with an array_search() in each one, so it just seems slower, that's all. And I'd have to double check there is no behavior change from how it worked previously.

mfb’s picture

Realized I was being lazy and I should benchmark it. So, I found that implementing the new hook in File MIME module (configured to add all the MIME types from the /etc/mime.types file) adds about 10 milliseconds of execution time. This is probably fine for those cases where the hook is invoked during file uploads - such requests are going to be extremely slow regardless - but not ideal for other pages where the hook might be invoked (e.g. managing display of an entity with a file field). Admittedly, File MIME module doesn't have very many installs (and more installs of the Drupal 7 branch than the current branch).

kim.pepper’s picture

#128 Thinking about this further, we should probably remove setMapping() to hide the underlying data structure. Projects like https://www.drupal.org/project/sophron that use the https://github.com/FileEye/MimeMap library have a different data structure.

#129 hmm. Shouldn't that hook only get fired on container build? It should be cached after that.

mfb’s picture

@kim.pepper no, there is nothing caching the map, whether in the container or otherwise. Both the old and new alter hooks run whenever the map is used.

mfb’s picture

Projects like https://www.drupal.org/project/sophron that use the https://github.com/FileEye/MimeMap library have a different data structure.

I looked at Sophron module, and it has a totally different mechanism for allowing other modules to alter its mapping - it dispatches an event. I guess you are saying that a future version of Sophron module could start invoking this alter hook?

kim.pepper’s picture

#131 I don't think that's right.

  file.mime_type.mapper:
    class: Drupal\Core\File\MimeType\MimeTypeMapper
    calls:
      - [alterMapping, ['@module_handler']]

The hook gets fired when this service is created.

mfb’s picture

alterMapping is called whenever the file.mime_type.mapper service is created (for example when ExtensionMimeTypeGuesser is constructed). So the alter hook fires on any request that needs to create that service.

E.g. if a page renders an entity that displays an audio file field, the service is needed and the alter hook fires. But once that entity is cached in the render cache, it will no longer render, and therefore the service won't be created, so the alter hook won't fire. Maybe that is the caching you were referring to, I'm not sure? There are also admin pages where the service is created, and the alter hook fires, without any sort of cache being involved (e.g. managing display of an entity with a file field).

kim.pepper’s picture

#134 Yeah I guess I was talking about the container cache. I misunderstood what you meant by 'Both the old and new alter hooks run whenever the map is used.' thinking you meant when we call guessMimeType() but it's when the service is created.

I did some more work removing setMapping(). I don't think that's going to cause too many issues.

mfb’s picture

I was timing the alter hook on requests where the service is created (e.g. because guessMimeType() was called at least once). Since it's when the service is created, the 10ms slowdown only happens at most once per request.

kim.pepper’s picture

I've done a fair bit of rework to replace the new hook with tagged services. I think triggering hooks during container build is a bit of an antipattern, and this approach uses a more symfony native approach.

mfb’s picture

@kim.pepper I don't understand why you say "triggering hooks during container build" - the hook was triggered when the service was created, i.e. when a request needs to use it for the first time, not when the container is built.

kim.pepper’s picture

the hook was triggered when the service was created

Fair enough.

As for the test fail, I have no idea why \Drupal\Tests\block\Functional\BlockCacheTest::testCachePerPage() is failing. It's passing locally for me. Could it be a random fail?

kim.pepper’s picture

Found the source of the test fails was a deprecation warning on the legacy hook_file_mimetype_mapping_alter() in file_test.module. We already test this in file_deprecated_test.module so I removed it.

Ready for reviews again!

mondrake changed the visibility of the branch 2311679-separate-mime-type to hidden.

mondrake’s picture

Related issues: +#3477346: ExtensionMimeTypeGuesser::guessMimeType returns less accurate MIME type when file extensions have multiple parts

I did a review. I am definitely biased by having developed Sophron, feel free to push back.

Also, adding a related issue that should be committed before this IMHO

kim.pepper’s picture

Thanks for the detailed review @mondrake. Much appreciated. I've responded to all your comments and made changes where I agree. I resolved a few where I did not agree, but feel free to re-open those threads if you disagree.

kim.pepper’s picture

I created a POC MR with a FileEye MIME map implementation to check if our interfaces and factory approach works with 3rd party libs.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

kim.pepper’s picture

Status: Needs work » Needs review

Rebased.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new10.61 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

kim.pepper’s picture

Status: Needs work » Needs review

Tests are passing.

mondrake’s picture

A bunch of nitpickeries from me. I think this is almost ready, cannot RTBC it though.

kim.pepper’s picture

Addressed all feedback. Thanks again.

godotislate’s picture

Status: Needs review » Needs work

One small deprecation docblock issue with version numbers.

kim.pepper’s picture

Status: Needs work » Needs review

Thanks. Feedback addressed.

godotislate’s picture

Status: Needs review » Needs work
kim.pepper’s picture

Status: Needs work » Needs review

Added tests and answered question.

godotislate’s picture

Status: Needs review » Reviewed & tested by the community

lgtm

godotislate changed the visibility of the branch 2311679-separate-mime-type-11.x to hidden.

godotislate changed the visibility of the branch 2311679-separate-mime-type-11.x to active.

godotislate changed the visibility of the branch 2311679-separate-mime-type-fileeye to hidden.

mondrake’s picture

Status: Reviewed & tested by the community » Needs work
mondrake’s picture

Assigned: Unassigned » mondrake

working on this

mondrake’s picture

Assigned: mondrake » Unassigned
Status: Needs work » Needs review

rerolled

godotislate’s picture

Status: Needs review » Reviewed & tested by the community

lgtm

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

kim.pepper’s picture

I've merged 11.x into this MR. Looks like the Hook Attributes work that got merged is triggering some deprecations that will need looking at.

kim.pepper’s picture

Status: Needs work » Reviewed & tested by the community

This was just a rebase on 11.x so putting back to RTBC

kim.pepper’s picture

Spoke to @xjm at DrupalCon Singapore contribution day, and she recommended tagging as Needs framework manager review.

kim.pepper’s picture

Category: Bug report » Task
Priority: Major » Normal

Discussed with @larowlan and this should be a task.

kim.pepper’s picture

I squashed all the commits in the MR to make rebasing on 11.x easier.

kim.pepper’s picture

Status: Reviewed & tested by the community » Needs work

Looks like we are triggering deprecation errors in Drupal\Core\File\MimeType\DefaultMimeTypeMap::getMapping() and Drupal\Core\File\MimeType\DefaultMimeTypeMap::setMapping()

kim.pepper’s picture

Status: Needs work » Needs review
Issue tags: -Needs framework manager review

Removing 'Needs framework manager review' tag as @larowlan has reviewed.

godotislate’s picture

Status: Needs review » Needs work

A couple comments.

kim.pepper’s picture

Status: Needs work » Needs review

Thanks for the review. Addressed feedback.

godotislate’s picture

Status: Needs review » Needs work

Apologies, seeing a couple new things with fresh eyes that I missed last pass:

kim.pepper’s picture

Status: Needs work » Needs review

Applied suggestions.

mondrake’s picture

Status: Needs review » Needs work
Issue tags: +Needs change record

Just noticed the CR links redirect to this issue and not to a CR. A CR for this is actually missing.

kim.pepper’s picture

Created a draft CR. Will look at changing the links in the morning unless someone else gets there first.

kim.pepper’s picture

Status: Needs work » Needs review
Issue tags: -Needs change record

Updated links to CR.

godotislate’s picture

Status: Needs review » Reviewed & tested by the community

LGTM. Nice work, @kim.pepper. Thanks for sticking with this.

mondrake’s picture

Assigned: Unassigned » mondrake
Status: Reviewed & tested by the community » Needs work

Working on BC fixes.

mondrake’s picture

Assigned: mondrake » Unassigned
Status: Needs work » Needs review

Now Sophron is passing tests with this MR applied (deprecation errors are thrown but that is OK). https://github.com/mondrake/d8-unit/actions/runs/12844774697/job/3581814...

We need to keep the protected properties as they are and cannot rely on the new ones (the map and the file system) to be initialised when the methods are called, as the classes that today extend ExtensionMimeTypeGuesser overriding the constructor will not have done that.

kim.pepper’s picture

I think we have resolved all threads except one question for @alexpott about changes to MimeTypeMapFactory

godotislate’s picture

Status: Needs review » Reviewed & tested by the community

Looks like all the latest feedback has been addressed. Back to RTBC,

alexpott’s picture

I've updated the MR to leverage \Drupal\Core\DependencyInjection\DeprecatedServicePropertyTrait - it'll allow us to remove the module handler property and have less complexity in the constructor.

godotislate’s picture

There are test failures in Drupal\Tests\system\FunctionalJavascript\Form\TriggeringElementTest. I'm guessing unrelated, and the tests need re-running.

kim.pepper’s picture

I re-ran the pipeline a couple of times to see if it was a random failure, but still getting the fails.

catch’s picture

Head was broken, might need a rebase.

godotislate’s picture

Maybe the MR branch needs a rebase? Looks like there was a commit to HEAD that broke TriggeringElementTest, but it was reverted. See #3461309: Refactor FormTestClickedButtonForm::buildForm #16.

kim.pepper’s picture

Rebased on 11.x

kim.pepper’s picture

Tests back to green.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed 37737af and pushed to 11.x. Thanks!

  • alexpott committed 37737af8 on 11.x
    Issue #2311679 by kim.pepper, mondrake, bhanu951, arla, alexpott,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

cmlara’s picture

Coming from #3528246: [META] Add support for Drupal 10.5/11.2 where we now have a circular reference fault related to this change..

S3fs implements a decorator of the core file_system service primarily to prevent silent data loss with an added bonus of better performance.

However the decorator needs to obtain the mime type of files in order to set the relevant delivery headers in the S3 bucket.

We hit this once in the past and as such we created a duplicate stand alone instance of the guesser service to avoid, however now that the guesser itself directly requires the file_system service we encounter the circular reference errors again.

xjm credited larowlan.

xjm’s picture

Saving a couple missing credits from the sprint.