BMU does not respect the schema of the file upload field

In my experience i am using Amazon s3 and it is doing what it is supposed to. It reads the files in as a stream to the database then it uploads them to s3 when it is generating the nodes. There are no files left on the server.

What are you using for your integration with s3?

I'm changing the title to better reflect the issue.

The problem is that BMU does not default to the schema of the field on which it operates. Rather, it initializes the upload to the sitewide default for all file uploads, a rather heavy-handed approach, IMO. This is particularly problematic for modules such as Storage API because (for performance reasons) it ignores everything except for its own schemas. For sites requiring scalability, Storage API is invaluable (I wrote the DreamHost DreamObjects integration module, for example). These are the types of sites that will benefit greatly from BMU.

I don't have much time, but will try to help with a patch. If anyone is wanting to jump in, the problem is in the schema selection in the _bulk_media_upload_generate_entity() function, in bulk_media_upload.upload.inc, line 155

I'm too tired to do a proper patch (in the middle of moving across the country, please have mercy!!!), but here is the change: Replace line 155 of _bulk_media_upload_generate_entity() with these two lines:

before:

  // Save media file.
  $scheme = variable_get('file_default_scheme', 'public') . '://';
  $source = $tmpfile['tmppath'];

after:

  // Save media file.
  $mediawidget_info = field_info_field($mediafield_info['field_name']);
  $scheme = (!empty($mediawidget_info['settings']['uri_scheme']) ? $mediawidget_info['settings']['uri_scheme'] : variable_get('file_default_scheme', 'public')) . '://';
  $source = $tmpfile['tmppath'];

In my tests, this fixes the integration issues with Storage API and opens up BMU to support a wider variety of configurations.

For clarity, I will post the entire function below which demonstrates the state of the function after the change has been applied:

/**
 * Internal function for entity generation.
 */
function _bulk_media_upload_generate_entity($tmpfile, $form, $form_state, &$context) {
  global $user;
  $entity_type = variable_get('bulk_media_upload_entity_type');
  $entity_info = entity_get_info($entity_type);
  // Default label field to 'name'.
  $label_field = isset($entity_info['entity keys']['label']) ? $entity_info['entity keys']['label'] : 'name';
  $mediafield_name = variable_get('bulk_media_upload_mediafield');
  $bundle = variable_get('bulk_media_upload_bundle');
  $mediafield_info = field_info_instance($entity_type, $mediafield_name, $bundle);

  // Save media file.
  $mediawidget_info = field_info_field($mediafield_info['field_name']);
  $scheme = (!empty($mediawidget_info['settings']['uri_scheme']) ? $mediawidget_info['settings']['uri_scheme'] : variable_get('file_default_scheme', 'public')) . '://';
  $source = $tmpfile['tmppath'];

  $directory = '';
  if(isset($mediafield_info['settings']['file_directory'])){
    $directory .= token_replace($mediafield_info['settings']['file_directory']) . '/';

    // If the directory isn't writable, or doesn't exist and can't be created,
    // the upload will fail.
    $prepare_directory = file_stream_wrapper_uri_normalize($scheme . $directory);
    if (!file_prepare_directory($prepare_directory, FILE_CREATE_DIRECTORY)) {
      drupal_set_message(t('The file directory @dir does not exist or is not writable. Please contact an administrator.', array('@dir' => $prepare_directory)), 'error');
      return;
    }
  }

  $destination = file_stream_wrapper_uri_normalize($scheme . $directory . $tmpfile['name']);
  $destination = file_unmanaged_move($source, $destination, FILE_EXISTS_RENAME);

  // Create the file object.
  $uri = file_stream_wrapper_uri_normalize($destination);
  $wrapper = file_stream_wrapper_get_instance_by_uri($uri);
  $file = new StdClass;
  $file->uid = $user->uid;
  $file->filename = basename($uri);
  $file->uri = $uri;
  $file->filemime = file_get_mimetype($uri);
  $file->filesize = @filesize($uri);
  $file->timestamp = REQUEST_TIME;
  $file->status = FILE_STATUS_PERMANENT;
  $file->is_new = TRUE;
  $file->status = FILE_STATUS_PERMANENT;
  file_save($file);

  // Create the new entity.
  $entity = entity_create($entity_type, array(
    $entity_info['entity keys']['bundle'] => $bundle,
    $label_field => $file->filename,
  ));

  // Taxonomy terms need to have their vocabulary ID added.
  // See http://drupal.org/node/1409256.
  if ($entity_type == 'taxonomy_term') {
    if ($vocabulary = taxonomy_vocabulary_machine_name_load($entity->vocabulary_machine_name)) {
      $entity->vid = $vocabulary->vid;
    }
  }
  // If the entity has a property tied to UID, set it so that, e.g., nodes will
  // be assigned an author.
  $info = entity_get_property_info($entity_type);
  foreach ($info['properties'] as $property) {
    if (isset($property['schema field']) && $property['schema field'] == 'uid') {
      $entity->uid = $user->uid;
      break;
    }
  }
  entity_save($entity_type, $entity);

  // Create the media field.
  $fieldinfo = field_info_field($mediafield_name);
  $filefields = array('image', 'file');

  $settings = array();
  $settings['fid'] = $file->fid;

  // Additional values for filefield (and imagefield).
  if(in_array($fieldinfo['module'], $filefields)){
    $settings['display'] = 1;
    $settings['description'] = '';
  };

  $entity->$mediafield_name = array(LANGUAGE_NONE => array(0 => $settings));

  // Default values.
  if (isset($form['default_values'])) {
    field_attach_submit($entity_type, $entity, $form['default_values'], $form_state);
  }

  entity_save($entity_type, $entity);

  // Replace tokens in title - this has to be done after entity_save.
  $label = token_replace($form_state['values']['title'], array($entity_type => $entity, 'file' => $file));

  $entity->$label_field = $label;

  entity_save($entity_type, $entity);

  $context['message'] = t('Importing: @filename', array('@filename' => $file->filename));
  $context['results']['ids'][] = $entity->$entity_info['entity keys']['id'];
}

I have an image field that is set to store in Drupal's own private file storage, but the files are being uploaded to Public file storage. So this is not just a problem when using external storage.

Trying to upload a proper patch.

Agreed, it's not just Storage API, but Private file storage as well (which means this could be interpreted as a security issue!).

Hey folks - I'm no drupal genius, so I can't say much to the efficacy of this patch other than the fact that I just tried it with a site I'm working with, and it fixed up the problem for me (using private file system).

The patch solves the problem. Please put in in the next dev version!

That patch didn't work for me in #7.
I don't seem to have a field type that has "uri_scheme" in its field instance settings.

I've created this patch that worked in my case. It pretty much checks the allowed_schemes of the file's field instance.
If it contains the default file scheme or if allowed_schemes is not set, it will use the default file scheme.
Otherwise, it will use the first scheme in the allowed schemes array of the file's field instance.

Patch in #7 did not solve everyone's problem. My patch needs review.

I think the solution can be smaller.
As field_info_field() is called later anyway we can just move it up in the function and read the schema value.
Tested this with S3 (s3fs) and works fine.

Tested #7 and works here using bulk_media_upload-7.x-1.x-dev. I haven't tested 11 and 13 unless for some reason #7 is no good to use