Enhance filefield mapper to perform more validation on remote URLs

I still maintain this is a valid enhancement, and I am finding a need to make further improvements. I have a feed that doesn't give me a normal image URL such as:

http://theirsite.com/file.jpg

Instead, it gives me this:

http://theirsite.com/image.ms?product=1234&width=768&height=1024

The script on their side outputs straight image/jpeg as a content type (not application/octet-stream like we currently expect.) We also don't cleanly handle creating a filename. We create something called "FeedsEnclosure-image.ms?product=1234_width=768_height=1024" which doesn't work very well.

I'm going to enhance two things from my original patch.

1) My URL checking function that does the HEAD, filefield_feeds_valid_url, is now going to return a content type instead of TRUE. If we determine a type from the HEAD request, we pass that back. Otherwise we pass back application/octet stream as a default. This will ensure that FeedsEnclosure is called with the correct content type.

2) In the event we don't get a filename, I'm going to figure out a way to account for that and create one.

OK, I'm done with something that seems to work. I can roll a patch, but it's probably easier to just drop in the function. There were other commits since my first patch.

FeedsParser.inc looks like the following. We accept a passed variable $mimetype, which if it is available comes directly from the HTTP HEAD information on the remote file we are downloading. This allows us to accurately set the mimetype on the file.

Basically, this function is different from the one in the tree now as follows:
1) Accepts passed $mimetype variable
2) Removes/replaces whitespace in filenames with underscores
3) Decodes HTML entities in the basename/filename
4) Determines mimetype extensions if available using a new library called mime.inc.
5) Searches the proposed filename for URL query strings, which indicate we might not have a file. If it finds query strings, we manually create the filename based on the date, some random characters, and the mimetype extension from #4 above.

  /**
   * @return
   *   The file path to the downloaded resource referenced by the enclosure.
   *   Downloads resource if not downloaded yet.
   *
   * @todo This is not concurrency safe.
   */
  public function getFile($mimetype) {
    if(!isset($this->file)) {
      // First set the default filename. Replace spaces and decode all HTML entities.
      $replace = array("%20");
      $filename = str_replace($replace, "_", htmlspecialchars_decode(html_entity_decode(basename($this->getValue()))));
      // Grab the normal file extensions if we have a mimetype
      if($mimetype) {
        feeds_include_library('mime.inc', 'mime');
        $mimetypes = array_flip(feeds_drupal_mime_types());
        if (array_key_exists($mimetype, $mimetypes)) {
          $extensions = explode("|", $mimetypes[$mimetype]);
        }
      }
      // If we see query strings, assume we don't have a good name and create our own
      $qstrings = array("?", "=", "&", ";");
      foreach($qstrings as $qstring) {
        $isqstring[] = strpos($filename, $qstring);
      }
      if(!empty($isqstring)) {
        if($extensions) {
          $filename = date("YmdHis") . '_' . mt_rand() . '.' . $extensions[0];
        } else {
          $filename = date("YmdHis") . '_' . mt_rand();
        }
      }

      $dest = file_destination(file_directory_temp() .'/'. get_class($this) .'-'. $filename, FILE_EXISTS_RENAME);
      if (ini_get('allow_url_fopen')) {
        $this->file = copy($this->getValue(), $dest) ? $dest : 0;
      }
      else {
        $this->file = file_save_data($this->getContent(), $dest);
      }
      if ($this->file === 0) {
        watchdog('content', 'Cannot write content to %dest', array('%dest' => $dest));
      }
    }
    return $this->file;
  }

I also attached my modified filefield module, as described in comment #1 above. This includes the HTTP HEAD checks and the returned content types, which dovetail into the enhancements in this post.

I attached the new mime.inc, which is a slightly changed version of the same include file from the fileframework module.

I'd love some feedback on this.

Thanks for the reply Alex.

1) Good thought and I just did a bit more research. Within the filefield module, the widget for the field stores allowed file extensions in the field definition. The actual extensions seem to be stored in the database within the content_node_field_instance table, CCK definition for the filefield, widget_settings field. Speaking strictly of the filefield module, upon upload the function in filefield.module called filefield_widget_upload_validators() runs. This checks the uploaded file against the maximum filesize and the allowed extensions.

My initial thought was perhaps we can just call that function to perform the same validation on files that are created from a feed import. The difference in the case of feeds vs. a normal upload is we can't enforce the same level of consistency at the remote end. With filefield, we can just deny a user from uploading a file with the wrong extension. In the case of feeds we may run into a remote site that links us to a file with an extension OR does what I described above and instead sends us a URL mapper such as theirsite.com/getmyfile.php?id=1234&width=500&height=500 with a mimetype of image/jpeg. It's not a 'real' file but is more of a web service serving us a file and we have to deal with the aftermath of trying to check it and save it. They could also send back various HTTP statuses of 403/404 or even return a 'themed' 404 page that just looks like a regular webpage.

This was the rationale behind trying to do a HEAD on the remote 'file' to determine what we are dealing with. Perhaps taking this a step further, we could do an HTTP HEAD, figure out the mimetype of the remote file, compare it against the allowed filetypes from the field definition, and then proceed if we succeeded with those checks. Or we could do this, save the file locally, check the actual mimetype based on php fileinfo, etc.

What do you think?

2) How would this work in the above scenario? To accurately determine mime type, would we would have to fetch it from the remote end first? Or, would we just try to grab whatever is at the remote end, save it somewhere, and then post-process it. There would still have to be logic involved to work around 403/404 and other errors that might come back as text/html. I'm working under the belief that to use FeedsEnclosure::getMIMEType(), we would have to have the file first from getFile().

3) Yes, I think we should have some kind of error handling ability. Some of this will probably depend on the answers to the above as it's somewhat inter-related.

I'm pretty motivated to write whatever is needed to implement this assuming we come to a final approach. I'm dealing with very large files/imports and all of them use filefield.

I'm still working on this code to perform better detection of the remote file via an HTTP HEAD request. I ran into an interesting situation on one of my feeds. I am feeding in an XML file with lots of image links that are to be turned into Drupal filefield/imagefields upon import.

One of the feeds I am using is hosting images at Amazon. Apparently they are using some kind of Amazon service where they only allow for 1 inbound request per second from a single IP. It's not a strict enforcement of that rule and a lot of items work, then you run into an image that instead gets this:

 HTTP/1.1 403 Forbidden
Server: squid
Content-Type: text/html
Content-Length: 1307
X-Squid-Error: ERR_ACCESS_DENIED 0
X-Cache-Lookup: NONE from cdn-images.amazon.com:10080
Cneonction: close
Expires: Sun, 18 Jul 2010 23:52:09 GMT
Date: Sun, 18 Jul 2010 23:52:09 GMT
Connection: close

If you request the same image a few seconds to a minute later, it works fine. So basically you get throttled down with 403's when you ask for too many images in a short period of time. The solution presented in the original issue intentionally makes two requests - one HEAD and one GET. This is part of what's creating the intermittent failures. The end result is you get an import with some working images, and some broken images.

I'm reworking it to support a delay if you get one of those 403s. That seems to have cured most of my problems. I'm also thinking of forward proxy support to reduce the possible load on the remote server. I'm just putting it out there as an FYI in case others are running in to the same thing.