File Size Limit

This is definiatly needed. I have hit the same problem. Has anyone looked at it?

I guess not.

Marked the following as duplicates:

• #1648016: Allowed memory exhausted
• #1706542: Out of memory

Here is a patch that adds a filesize limit. There is currently no UI for setting this, but can easily be done via drush, settings.php, or strongarm. The default is no limit, so existing sites will behave as they do now. This is a very important feature for sites that have no upload size limit, as even with enormous amounts of memory, large enough files will still fatal error cron.

I think it is better to use the exclude hook. We might get to a point with multiple environments and different limitations of filesizes (who knows right?) and we do not want to mark this file as invalid and/or incomplete. We just want to ignore/exclude it.

/**
 * This is invoked for each entity that is being inspected to be added to the
 * index. if any module returns TRUE, the entity is skipped for indexing.
 *
 * @param integer $entity_id
 * @param string $entity_type
 * @param integer $row
 *   A complete set of data from the indexing table.
 * @param string $env_id
 * @return boolean
 */
function hook_apachesolr_exclude($entity_id, $entity_type, $row, $env_id) {
  // Never index media entities to core_1
  if ($entity_type == 'media' && $env_id == 'core_1') {
    return TRUE;
  }
  return FALSE;
}

I've hacked my the module to fix it on my system, but I can do a more general fix if someone can help me with a bit of code.

Essentially,

   $mem_left = amount_of_memory_still_available_to_php();
   $mem_needed = size_of_file_about_to_be_read($file);
   if ($mem_needed > ($mem_left + $some_fudge_factor)) {
         // do want needs to be done
       }
     else {
        log_an_error("File to large: ".$file);
     }

What I need is that first function, amount_of_memory_still_available_to_php().

This patch is based on the patch from #4 and uses the hook_apachesolr_exclude hook. A new variable is added to the configuration screen to control the filesize limit.

Update on patch from #7 with tiny bit of validation for variable to limit filesize.

Re-roll of #8 without removing the newline at end of file (which caused the patch to break against HEAD).

Made the default 40MB as we see most of the problems coming from files above that size. It's still configurable. Also moved the exclude hook to the hook that was already implemented. (apachesolr_attachments_apachesolr_file_exclude)

Committed it to dev branch. Let me know if it works for you

+++ b/apachesolr_attachments.module
@@ -388,6 +388,19 @@ function apachesolr_attachments_apachesolr_file_exclude($entity_id, $row, $env_i
+  // Exclude files above the configured limit.
+  $filesize_limit = variable_get('apachesolr_attachments_filesize_limit', '40');
+  // load the entity.
+  $entities = entity_load('file', array($entity_id), NULL, TRUE);
+  // Take the first item.
+  $entity = reset($entities);
+
+  // Check if the filesize is higher than the allowed filesize.
+  if (isset($entity->filesize) && $filesize_limit > 0 && $entity->filesize > $filesize_limit) {
+    return TRUE;
+  }
+

It looks like the variable is an integer that means 'megabytes', but $entity->filesize is kept in bytes. So this works BUT the default of '40' will then skip all files except those smaller than 40 bytes :(

Patch for 7.x-1.x.

@janusman - I think you posted an interdiff only?

I didn't.. my diff was against a fresh checkout of master :)

My changes are pretty tame and seem to work, so committing to 7.x-1.x-dev

(As soon as I get permissions to do so!)

Committed.

I still have a "PHP Fatal error: Allowed memory size", when I index a file with tika with the 7.x-1.3+9-dev version (2015-02-13).

In, fact the error occurs because tika try to extract the content of the file before the hook_apachesolr_file_exclude.

For fix this bug, i have alter the apachesolr_attachments_get_attachment_text() function, as bellow :

function apachesolr_attachments_get_attachment_text($file) {
  $indexer_table = apachesolr_get_indexer_table('file');
  if (!apachesolr_attachments_is_file($file)) {
    return FALSE;
  }
  // Exclude files above the configured limit.
  $filesize_limit = variable_get('apachesolr_attachments_filesize_limit', '41943040');
  // Check if the filesize is higher than the allowed filesize.
  if (isset($file->filesize) && $filesize_limit > 0 && $file->filesize > $filesize_limit) {
    return false;
  }
....

New patch, basically the same as n.dhuygelaere's from #21, but I added logging.

Committed to 7.x-1.x

This is somehow STILL NOT RELEASED... I was bit by this problem this week and installing the dev version of the module did the trick. 👍 Module maintainers please release!