uploading of files fails with mime functions enabled.

It would be nice to understand the cause.

*sigh* Given the number of problems that MIME magic caused I say, let's get rid of it. Dries, please commit this against all releases. On http://www.php.net/manual/en/ref.mime-magic.php one can read: This extension has been deprecated as the PECL extension fileinfo provides the same functionality (and more) in a much cleaner way. So we better not use it, now.

May I point to how others solved this? Here's an example from the phpBB attachment MOD.

markus, I read that code. I fail to see how that applies here as that one deals with a few specific file types. We need a generic solution.

I'm with Dries on this. If an upload is being processed 3 times there's something seriously wrong with the upload process. The mime magic error is just a symptom of the real problem.

Then that's another issue. The mime magic determination is flawed, deprecated in short not to be used. Get rid of it. I introduced it a few months back, that was an error, obliterate it.

chx, I agree with you.

My previous comment was simply aimed to pointed to something I know it works (without mime magic), only I choose a library (not really related to this issue, true), as a pointer.

If we decide not to remove the current fix, we should fix it. Right now, it is broken as can be seen from the many bugreports.
I'm tempted to remove it now, and to reintroduce a proper fix if it emerges. I'll hold off committing this patch one or two days so we can discuss this some more. I haven't make up my mind yet so I'm looking for constructive feedback.

Dries, you won't get any good feedback here. According to http://www.php.net/~derick/meeting-notes.html#id31

The mime_magic extension doesn't work very well

Are we really debating that we want to use an extension which is marked as deprecated and admitted by the authors not to work well ?

The same code has been in a stable release since 4.6.4, so this is something that might need to be in a 4.6.6

You already get a mime type from the browser, but it can be trusted. So we need something more, but those mime magic extensions just check for a few bytes in the file stream. Sadly, these days, that can't be trusted either. It has happened before and it will happen again and again.

Aside from something like an anti-virus enabled to check all file uploads, the best protection would be to move uploaded files to a place the could not be accessed from a browser (off the htdocs dir or protected via .htaccess rules, but security is best in layers, so off the htdocs sounds wiser). Anyway, no problem then, anything that has been uploaded can be downloaded, there is little you can do to make sure it is all safe.

The problem is then isolated to the download side, ie. files that could be processed by the browser itself. So, you could create your own set of functions (such as those that I pointed to before) to check for a set of known file types such as images, pdfs, flash or video. You could then use the proper mime type for them or application/octet-stream for the rest.

while it's true that mime_magic stuff isn't entirely fool proof ... and yes, it does look to be slated for removal ... it does give us an extra bump over just trying to rely on the mimetype headers sent by the browser (these are quite often wrong from IE anyway - often application/octet-stream).

i'd have to say -1 for it's removal. there is still a bug in drupal's use - namely for non-JS uploads ... upload_nodeapi 'validate' get's called twice (since upload_save actually calls it manually).

i've not had a chance yet to roll the patch, but the issue is better solved (imo) by fixing that issue, as the mime magic stuff does work in other scenarios.

mind you... just removing the mime_magic stuff is a simpler (already rolled) patch. just my $0.02 CAD

Well. We have two issues here. One is a broken upload.

Another is that mime magic should be removed or not. Well, it'll be real cool when we will answer the 1-2 daily support requests that it's not our task to set up Apache's MIME magic correctly. This will nicely drive away people from Drupal. Yes I know we do not want those users, but this goes a bit too far...

Oh and it should be removed from 4/5.x / 4.6.x too.

I agree with James here. All our upload stuff is not exactly extremely secure anyway (a post 4.7 task though). Mime magick is just a little extra.

What about a 'hidden variable' taht gets variable_set('use_mime', TRUE); when mime was found correctly set. If not, we should set it to FALSE.
Then in the upload we can check for this var. And on the admin/settiings we can prompt a big red warning about this.

Is there a fix for this or just a work-around. I've read these posts a couple of times but my upload functions are broken. I tried the first patch and it did nothing. The second patch - no_mime_magic.patch was rejected when I tried to patch the includes/file.inc file.

I'd just like to be able to upload files without Drupal appending .TXT to them. If anyone can help.

metapunk, look at the patch at http://drupal.org/node/44165, works for me (marked as duplicate to this one).

do we have any resolution here?

I found this problem with drupal versions prior to 4.7 while trying to upload a .swf file. mime reads it as a txt file. I just applied chx's patch to cancel the use of mime_content_type function for one of my sites running 4.6.5. I would apply this patch to the actual 4.6.5 version and make it 4.6.6, along with other patches there might be arround.

Just a note that the patch attached by killes on January 16 did not work for me, while the patch sent by chx on January 17 did work.

I am running Drupal 4.6.5 with PHP 5 under Ubuntu 5.10.

OK - we really need to nail this one down!
I ran an xdebug trace on file upload (attached - makes great bedtime reading) which shows how file_check_upload() is called each times.

Here is a simplified stack:

1st.Time
....0.0025......44560...->.{main}()./home/owen/sites/cvs/index.php:0
....0.5416....5592248.....->.menu_execute_active_handler()./home/owen/sites/cvs/index.php:15
....0.5774....5712128.......->.call_user_func_array()./home/owen/sites/cvs/includes/menu.inc:414
....0.5775....5712128.........->.node_page()./home/owen/sites/cvs/includes/menu.inc:0
....0.5783....5712128...........->.node_add()./home/owen/sites/cvs/modules/node.module:2032
....0.5867....5712608.............->.node_form()./home/owen/sites/cvs/modules/node.module:1758
....0.6796....5736000...............->.drupal_get_form()./home/owen/sites/cvs/modules/node.module:1641
....0.8998....5765784.................->.upload_form_alter()./home/owen/sites/cvs/includes/form.inc:102
....0.9049....5765920...................->.upload_nodeapi()./home/owen/sites/cvs/modules/upload.module:169
....0.9059....5765920.....................->.file_check_upload()./home/owen/sites/cvs/modules/upload.module:214
....0.9062....5765920.......................->.is_object()./home/owen/sites/cvs/includes/file.inc:140
....0.9063....5765920.......................->.is_uploaded_file()./home/owen/sites/cvs/includes/file.inc:145
....0.9064....5765920.......................->.file_exists()./home/owen/sites/cvs/includes/file.inc:174
.
2nd.Time.(from.drupal_get_form().above)
....1.0919....5822432.................->.drupal_validate_form()./home/owen/sites/cvs/includes/form.inc:108
....1.0922....5822528...................->._form_validate()./home/owen/sites/cvs/includes/form.inc:146
....1.0930....5822744.....................->.call_user_func_array()./home/owen/sites/cvs/includes/form.inc:208
....1.0932....5822768.......................->.node_form_validate()./home/owen/sites/cvs/includes/form.inc:0
....1.0934....5822768.........................->.node_validate()./home/owen/sites/cvs/modules/node.module:1617
....1.1081....5828648...........................->.node_invoke()./home/owen/sites/cvs/modules/node.module:1600
....1.1127....5828736.............................->.upload_nodeapi()./home/owen/sites/cvs/modules/node.module:317
....1.1137....5828736...............................->.file_check_upload()./home/owen/sites/cvs/modules/upload.module:214
....1.1140....5828736.................................->.is_object()./home/owen/sites/cvs/includes/file.inc:140
....1.1141....5828736.................................->.is_uploaded_file()./home/owen/sites/cvs/includes/file.inc:145
....1.1142....5828736.................................->.file_exists()./home/owen/sites/cvs/includes/file.inc:174
.
3rd.Time.(from.drupal_get_form().above)
....1.2655....5832544.................->.drupal_submit_form()./home/owen/sites/cvs/includes/form.inc:110
....1.2660....5832640...................->.call_user_func_array()./home/owen/sites/cvs/includes/form.inc:158
....1.2660....5832640.....................->.node_form_submit()./home/owen/sites/cvs/includes/form.inc:0
....1.2855....5834616.......................->.node_save()./home/owen/sites/cvs/modules/node.module:1872
....1.3696....5839072.........................->.node_invoke_nodeapi()./home/owen/sites/cvs/modules/node.module:493
....1.4024....5843232...........................->.upload_nodeapi()./home/owen/sites/cvs/modules/node.module:317
....1.4034....5843232.............................->.upload_save()./home/owen/sites/cvs/modules/upload.module:320
....1.4037....5843232...............................->.upload_nodeapi()./home/owen/sites/cvs/modules/upload.module:378
....1.4195....5844416.................................->.file_check_upload()./home/owen/sites/cvs/modules/upload.module:214

The next question is to figure out when the file is getting moved and to either remove or fix the calls to file_check_upload after that point.
Any patches?

[+1 for removing the mime_content_type tests (false sense of security... etc), but we need to fix this bug too!]

- Owen

ok gang. i'm gonna take this on. I had a half working patch before vancouver, need to refresh my memory and finish it off. (assigning to myself so i keep track of the nid).

Possible fix, but I'm having trouble druplicating

 if (is_object($source)) {
    if (is_file($source->filepath)) {
      return $source;
    }
    //nothing was stopping execution here...
    return false;
 }

.darrel.

The patch submitted by chx on #5 works for me. I'm using Drupal 4.6.5 on Ubuntu 5.10 with Lighttpd. It fix the .txt problem in the upload module and also the image module which gave me three warnings about the file mime type, leaving the screen only with that warnings.

I'm using a debian sarge apache 1.3 php 4.4 and the patch for Chx posted on #5 seems to have cleared up the problem for me. I don't know what happened that introduced this problem to all debian derived servers. Did anyone figure this out ?

I'm tempted to commit chx' patch (#5) unless James comes up with something in the next day or two.

The patch for http://drupal.org/node/5961#comment-76338 also resolves this issue as far as I can tell.

dopry, while it may solve on your server, this may not be the case for everyone's server.... well, i have expressed my views on using mime magic and even admitted it was my fault to introduce... rant done.

The issue at hand wasn't mime_get_content, it was an invalid file path being returned from file_check_upload.

I removed the mime type magic for now (using chx' patch which seems to be supported). When a better solution comes along, we'll commit that! Let's work with dopry on this one!

Opening this report to say that the patch provided was not commited to 4.6.x series!

Hi,

I'm not sure if this is the right place to post this, but this could be useful for Drupal Users running Ubuntu:

I used to have the .txt upload problem in Drupal 4.6.5 on Ubuntu 5.10, and chx's patch helped... but in 4.6.6 the problem arose again.

All these troubles come from a broken mime_magic support support in the Ubuntu php5 package, as phpinfo() returns these values:

mime_magic support: invalid magic file, disabled
mime_magic.debug: Local Value = off, Master Value = off
mime_magic.magicfile: /usr/share/misc/file/magic.mime, Master Value /usr/share/misc/file/magic.mime

We simply have to set the right path to the mime file... but due to a bug in php (http://php.linux.hr/manual/es/ref.mime-magic.php#50724) this setting must be changed in php.ini, not httpd.conf or anywhere else.

I simply added the following lines to php.ini and everything was working in 4.6.6 without any patch:

[mime_magic]
mime_magic.magicfile = "/etc/magic"

Backported to 4.6.

Marked issue 74542 (".txt added to all file extensions") as a duplicate of this issue.

When the code for file type checking was removed, some code that depends on it was left in, resulting in .txt being added to every file. I'm attaching patches to remove this code.

Patch for 4.7 and CVS.

Patch for 4.6.

Do not re-open old / closed issues to address current issues. Create a new issue.

This is an old issue. The committed patch was incomplete. I have opened a new issue to deal with this patch.