This patch pulls the default value of the mime_extension_mapping variable out to its own function so it's accessible to module developers who want to use or alter the mapping.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Anonymous’s picture

Status: Needs review » Needs work

You need to move the variable_get to the file_mime_extension_mapping and store the array data in a static variable.

Actually, why are you using the variable system anyway? Why not just store the data in a static variable and allow the static to change if a value is given?

mfb’s picture

@earnie: I don't understand your comment about moving the variable_get() to file_mime_extension_mapping(). The whole point of the patch is to make the default value of the variable accessible, not the configured value of the variable.

A variable is used so sites can permanently modify their mime_extension_mapping, although possibly it could have been implemented using drupal_alter() instead, allowing sites to modify it dynamically.

drewish’s picture

really needs a test.

mfb’s picture

ok.. I'd say a few more assertions for the existing file_save_data() and file_save_upload() tests would be a good enough test coverage but let me know if someone has ideas for more exhaustive tests.

Anonymous’s picture

@mfb: Moving variable_get() to the file_mime_extension_mapping() function will allow for what you want but I don't have to specify it in the call.

So

+    $mapping = variable_get('mime_extension_mapping', file_mime_extension_mapping());

becomes

+    $mapping = file_mime_extension_mapping();

If you do

+  static $map = array(
+    'ez' => 'application/andrew-inset',
+    ...
+  );
+
+  return variable_get('mime_extension_mapping', $map);

in the function.

The static $map will cause the run time build of the array to occur once in each session refresh instead of each call to the function. The return of the variable_get values allow the use of the function for the map value without the user needing to worry with where the values might come from.

There also needs to be an administration UI if you keep the variable_get.

mfb’s picture

Status: Needs work » Needs review
FileSize
33.43 KB

I added some mime-type related assertions to the existing file and upload tests.

@earnie, the new function is intended to provide access to the *default* value of the variable; your code would only give me access to the configured value. The idea is that developers could use this array in their own functions, besides file_get_mimetype(). Admittedly, it might be I'm the only person in the world interested in this...

Anonymous’s picture

Status: Needs review » Needs work

Then you don't need the function. A defined constant would be better.

mfb’s picture

I thought about a constant, but this value isn't needed on the vast majority of page requests, and is a rather large array so takes a fair amount of memory. It seems to me like too much overhead to make it a constant.

Anonymous’s picture

Too much overhead is why I think it either needs to be static or and constant. Building the large array every time you need it is huge. Then you're storing it in the variable system as well. If it's a constant then the PHP optimization engines have a chance to cache it.

mfb’s picture

Another reason it cannot be a constant, PHP does not allow arrays as constants.

mfb’s picture

Status: Needs work » Needs review
FileSize
33.46 KB

use a static variable in case the function is called a second time (e.g. multiple file uploads in one request)

drewish’s picture

Status: Needs review » Needs work

tests are an improvement but still needs some dedicated tests for the new and altered functions. we need to:
- call file_mime_extension_mapping() and make sure it returns an array
- call file_get_mimetype() with the default and a made up mapping and check that the right type is returned

i know that these functions are tested indirectly by other tests but we need some tests that specifically target these functions and their edge cases.

drewish’s picture

i transposed the function names in that last comment.

we should also test loading mappings from variables vs. parameters in file_get_mimetype().

mfb’s picture

Question about core unit tests: Should we be using "UnitTest" as the class name suffix?

Anonymous’s picture

mfb’s picture

What I was asking about is the test writers guidelines at http://drupal.org/node/325974 suggest using *UnitTest for unit tests. But right now there is only one *UnitTest class in core, DrupalErrorHandlerUnitTest.

drewish’s picture

Status: Needs work » Needs review
FileSize
36.33 KB

Here's some tests. Also improved the PHPDoc and renamed the variable to match the function name (giving it some namespace protection).

c960657’s picture

I assume the typical use case for modifying the default mapping is adding or replacing a few entries, not supplying a completely different mapping. Wouldn't it be better to let the setting be applied on top of the default mapping?

mfb’s picture

I'd agree that this would be preferable. Possibilities include making the default mapping an internal static array and applying the setting array, if it exists, on top of it, or using drupal_alter() in the unlikely event that various modules need to merge their additions/replacements.

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review
FileSize
37.49 KB

here's another possibility: get rid of the variable, and just use drupal_alter to allow the mapping to be altered. This seems like the "right" way to allow large arrays to be altered by contrib modules, rather than using a variable.

Damien Tournoud’s picture

Status: Needs review » Needs work

I like this last approach.

Only one thing: the static in file_mime_extension_mapping() serves no purpose except hogging memory. It should be removed.

mfb’s picture

Status: Needs work » Needs review
FileSize
37.35 KB

I did find that the static offered a speedup if the function is called twice (e.g. two file uploads in one request) presumably because PHP only has to parse the large array statement once. But, now that I think about it, file uploads are not the requests you worry about speeding up by a fraction of a second so I would agree. another thing is there was a leftover variable_del('file_mime_extension_mapping')

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review

Does PIFR tell us which assertions failed? I don't have any failures.

Status: Needs review » Needs work

The last submitted patch failed testing.

c960657’s picture

Status: Needs work » Needs review
FileSize
39.66 KB

Reroll.

c960657’s picture

FileSize
38.56 KB

The last patch contained an unrelated change to registry.inc.

mfb’s picture

FileSize
36.76 KB

Looks good. One minor tweak, in some cases file.test uploads a jpeg rather than png (it selects a random image) so we can only check that MIME type is image/something.

c960657’s picture

Your updates to the patch look sane.

Berdir’s picture

Copy paste of my "crazy" idea which I posted at http://drupal.org/node/370958#comment-1435888

a idea that was mentioned in the hook_variable_info() issue was to create a table to store the mimetype array. I've also seen that there is work for a file.module, it could even provide a interface to add additional mimetypes. Or a contrib module could do it.

Some advantages such a table would imho have:
- Better handling of large amount of entries, does need less memory
- Doesn't need to be cached/saved in the variables table as a big blob
- Possibilites to manage the mime types and create a UI to manage the mimetypes
- We could reference to that table, instead of storing the mimetype for each file entry

mfb’s picture

@Berdir: I don't have anything against a table, but note this patch already gets rid of the variable.

Berdir’s picture

@mfb
Oh, I see, I havent really looked at the patch before. A table might be a bit more flexible, but this is easier and don't need a update function and so on. I'm happy as long as we don't have to load the whole array from the cache on each request.

mfb’s picture

FileSize
37.02 KB

chasing HEAD

Berdir’s picture

Status: Needs review » Needs work

I really want this in, simply because it will cut the hook_variable_info() patch in half :)

I'd suggest adding the explanation in #29 to those tests. Just from looking at the tests, I'ts not obvious why substr() is needed there.

$this->assertEqual(substr($file1->filemime, 0, 5), 'image', t('A MIME type was set.'));

Other than that, it looks good to me.

mfb’s picture

Status: Needs work » Needs review
FileSize
37.17 KB

OK, I added code comments for the substr() calls.

Berdir’s picture

Status: Needs review » Needs work

Ok, another small point.

+    $this->assertResponse(array(200), 'Uploaded file is accessible using public download.');
+    $this->assertEqual(file_get_contents($file), $this->drupalGetContent(), 'Contents of the file are correct.');
+    // Verify file using private download method.
+    $this->drupalGet('system/files/' . $filename);
+    $this->assertResponse(array(200), 'Uploaded file is accessible using private download.');
+    $this->assertEqual(file_get_contents($file), $this->drupalGetContent(), 'Contents of the file are correct.');
+    $this->assertEqual($this->drupalGetHeader('Content-Type'), 'text/plain', t('A MIME type was set.'));

AFAIK, the assert messages at the end should all be translated, but some of these aren't...

mfb’s picture

Status: Needs work » Needs review
FileSize
37.18 KB

Thanks. Looks like upload.test needs further cleanup in this area, but that's out of scope for this patch.

Berdir’s picture

Status: Needs review » Reviewed & tested by the community

Agreed.

Ok, patch looks fine to me, All tests pass and there are tests for the new features. This makes it easier to extend these mimetypes for other modules. Also, this removes a big part of the hook_variable_info() patch (and cache) and therefor simplifies that patch severly.

This is imho RTBC.

chx’s picture

Status: Reviewed & tested by the community » Needs work

I do not get or agree with the array_reverse. You are altering, so alter the array instead of appending.

mfb’s picture

Status: Needs work » Needs review
FileSize
37.04 KB

@chx: the rationale for array_reverse() is that it allows for slightly simpler code on the contrib module side, because you can append to the array rather than prepending.

Without the array_reverse, if you wanted to map 'ogg' => 'audio/ogg', you would need to prepend it to the array -- i.e. $mapping = array('ogg' => 'audio/ogg') + $mapping; -- otherwise the built-in 'ogg|ogx' => 'application/ogg' would have precedence. Also, if you altered one of the mappings and the built-in set of extensions for the mapping was later modified, then your alteration would end up being appended to the array and have no effect.

For now I left the array_reverse() but I did need to re-roll.

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review
FileSize
37.07 KB

chasing HEAD

Status: Needs review » Needs work

The last submitted patch failed testing.

catch’s picture

mfb’s picture

Status: Needs work » Needs review
FileSize
5.9 KB

The main thrust of the original feature request has already been resolved by file.mimetypes.inc :) However, I rerolled all the extra bits in this patch, including the ability for contrib modules to alter the mimetype mapping and some extra tests.

mfb’s picture

FileSize
5.89 KB

Fixed a typo in function documentation.

mfb’s picture

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review
pwolanin’s picture

in function file_get_mimetype() you load the mapping and call drupal_alter() on each function invocation?

mfb’s picture

Issue tags: +Performance

@pwolanin: Yes. Were you suggesting a static cache of the default mapping? If so see comment #22 and #23.

I'm adding a Performance tag as I think there's already a performance boost (if a module needs to alter the default mapping) in not storing a huge mime_extension_mapping array in the variable table.

pwolanin’s picture

@mfb - if this is only used at upload time, then I'd agree it's fine.

Berdir’s picture

It is used for every request, for the favicon... http://api.drupal.org/api/function/template_preprocess_page/7

But it might make sense to remove that check and only do it when the favicon is changed.

mfb’s picture

@Berdir: Hmm, interesting :( Including file.mimetypes.inc every time we render a page is not ideal. Yes this should be cached the same way we cache an uploaded file's mime type.

mfb’s picture

FileSize
8.61 KB

OK this patch adds another performance optimization, replacing the call to file_get_mimetype() on every page load with theme_get_setting('favicon_mimetype'). The favicon_mimetype is set whenever a custom favicon is configured.

mfb’s picture

FileSize
8.89 KB

Moved a few more lines of logic from page preprocess to theme settings submit.

mfb’s picture

FileSize
10.6 KB

Oops the merge conflict in file.test needed more fixups than I thought, hopefully passes now.

mfb’s picture

Title: Give module developers access to the default mime_extension_mapping » Allow modules to alter the MIME extension mapping rather than setting a huge variable
aaron’s picture

subscribe

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review
FileSize
10.63 KB

Chasing HEAD.

aaron’s picture

No, no, no. We don't want people to have to manually enter the favicon MIME type. Look at #531476: Add favicon mimetype theme setting first, which would take care of that. It's RTBC as soon as someone can give it a look-over.

mfb’s picture

@aaron, there is nothing manual in this patch about favicon MIME type, it's automatic. Can you explain what's wrong with this patch? I do think it's a good idea to move it to separate issue since it's not directly related to altering the mapping.

aaron’s picture

Ah, misread the patch. Yeah, that looks okay. Though I think using getimagesize info for favicons would be better anyway. Mine's just a different approach, and we could redo that patch afterwards.

aaron’s picture

How about you remove the favicon MIME type out of this patch, since it's not really related, and we can give it a proper read, looking at the different options (drewish and i came up with two alternates at the other issue already).

aaron’s picture

For instance, I'm pretty sure (without testing) that this will end up with no favicon MIME type on maintenance pages, despite best intentions, as there's no access to the db. That's the benefit of using variable_get; we could set that in $conf in settings.php.

mfb’s picture

FileSize
7.57 KB

Well my patch uses theme_get_setting() which is used elsewhere in theme.maintenance.inc. Also it sets a default value for favicon_mimetype which should work fine if the theme setting doesn't exist. Anyways I've removed the favicon stuff from this patch so let's address it at #531476: Add favicon mimetype theme setting.

aaron’s picture

This looks really good to me now. You should probably put something in the changelog.txt about it, then I'd say it's RTBC. (Of course, we'll also need to update upgrading dox to note the change from the variable_get.)

mfb’s picture

FileSize
9.43 KB

added entry to changelog and documentation for the new hook to system.api.php

aaron’s picture

Status: Needs review » Reviewed & tested by the community

awesome work! looks rtbc to me.

drewish’s picture

Status: Reviewed & tested by the community » Needs work

I'd love to avoid changing these lines:

-    $this->assertResponse(array(200), 'Uploaded ' . $filename . ' is accessible.');
+    $this->assertResponse(array(200), t('Uploaded %filename is accessible.', array('%filename' => $filename)));
     $this->assertTrue(strpos($this->drupalGetHeader('Content-Type'), 'text/plain') === 0, t('MIME type is text/plain.'));
-    $this->assertEqual(file_get_contents($file), $this->drupalGetContent(), 'Uploaded contents of ' . $filename . ' verified.');
+    $this->assertEqual(file_get_contents($file), $this->drupalGetContent(), t('Uploaded contents of %filename verified.', array('%filename' => $filename)));

since it's not in the scope of the issue and there's still discussion about using t() in assertion messages—translators hate them since they clutter things up and anyone doing core dev work has to be able to read english.

Berdir’s picture

Yeah, I wrote in #331171-37: Allow modules to alter the MIME extension mapping rather than setting a huge variable that they should be translated, but in the meantime, the remove translation from tests issue has landed, so these should be removed again.

mfb’s picture

Status: Needs work » Needs review
FileSize
8.7 KB

OK, removed the t() calls from *.test

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review
FileSize
9.43 KB

Chasing HEAD

Status: Needs review » Needs work

The last submitted patch failed testing.

mfb’s picture

Status: Needs work » Needs review
FileSize
9.38 KB

Chasing HEAD

aaron’s picture

Status: Needs review » Reviewed & tested by the community

i don't think there's anything else holding this up now.

pwolanin’s picture

Status: Reviewed & tested by the community » Needs work

this doesn't make sense to me to have in the stream wrapper:

+      $mapping = file_default_mimetype_mapping();
+      // Allow modules to alter the default mapping.
+      drupal_alter('file_mimetype_mapping', $mapping);

Why would we call drupal_alter on it every time it's used, instead of just the first time?

pwolanin’s picture

Status: Needs work » Needs review
FileSize
10.08 KB

something like this.

drewish’s picture

that makes way too much sense. mfb any objections?

mfb’s picture

It looks good to me. This function is typically only called when file(s) are uploaded so the memory-hogging static variable shouldn't be a problem, and does speed things up (if anyone is actually benchmarking multiple-file uploads).

aaron’s picture

Status: Needs review » Reviewed & tested by the community

As we introduce a new hook_file_mimetype_mapping_alter(), I wonder if we should document that in system.api.php? Otherwise, looks great to me.

This review is powered by Dreditor.

pwolanin’s picture

Status: Reviewed & tested by the community » Needs review
FileSize
10.85 KB

yes it should be added. New patch.

aaron’s picture

Status: Needs review » Reviewed & tested by the community

I was being blind, Peter, sorry about that... :P The patch at #81 has that. Thus, #81 is RTBC, and #85 now has a duplicate API description (which would probably break the API module -- should an issue be created to test that API hook_functions aren't duplicated?).

pwolanin’s picture

FileSize
10.4 KB

hmm, I wrote more extensive doxygen, so I merged my doxygen with the existing code example for the hook.

mfb’s picture

Looks good.

pwolanin’s picture

Issue tags: +API change

tagging

Dries’s picture

Status: Reviewed & tested by the community » Fixed

Reviewed this patch and it looks good. Committed to CVS HEAD. Thanks mfb, pwolanin, drewish et al. Good work.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

mattgilbert’s picture

Is there a Drupal 6 version?

mattgilbert’s picture

Is there a Drupal 6 version?

mfb’s picture

@mattgilbert: In Drupal 6, http://drupal.org/project/filemime module provides a UI to set the mime_extension_mapping variable to whatever you want (can also read in a mime.types file from the server file system).

jhodgdon’s picture

Status: Closed (fixed) » Needs work
Issue tags: +Needs documentation updates

Was this change announced on the 6/7 module update guide? If it was, or if it didn't need to be, please revert status/tag change.

jhodgdon’s picture

Status: Needs work » Fixed

I couldn't find anything about MIME types on the 6/7 module page, so I added this:
http://drupal.org/update/modules/6/7#mime-types
If someone wants to review, that could be useful -- I based it on the patch in #87, which appears to be the only one that was committed. I think it's OK...

mfb’s picture

Looks fine. Technically speaking, the mime type mapping couldn't really be altered in Drupal 6, it could just be replaced (the default mapping is in code as the variable's default value). So this issue added the ability to alter the mapping rather than just replace it.

jhodgdon’s picture

Right. I figured most modules would have done a variable_get(), changed the array, and then done a variable_set() probably, which is kind of a "poor man's alter".

Status: Fixed » Closed (fixed)
Issue tags: -Performance, -DX (Developer Experience), -API change, -Needs documentation updates

Automatically closed -- issue fixed for 2 weeks with no activity.