Batch API export error

This error, however, doesn't get logged.

It probably isn't logged in Drupal's DB Log, because if PHP has crashed, Drupal won't be able to do anything, including log to the database. You should be able to find a log message in your server's PHP or Apache logs when this happens.

My guess is that you're running into a PHP memory limit. We try to guess what a reasonable limit is for exporting in webform_results_download_form_submit(), but my guess is that the batch size is guessing too large and you're running out of memory. If it's possible to increase the memory limit on your server, you might try that.

It'd be good to get some real-world adjustment on our algorithm as well though, if you could try adjusting some of the numbers in the webform_results_download_form_submit() function I'd be interested in hearing the results. Here's the current code that does the guessing:

    // Start the batch size at 50,000 per batch, but divide by number of
    // components in the form. e.g. If a form had 10 components, it would
    // export 5,000 submissions at a time.
    $batch_size = ceil(50000 / max(1, count($form_state['values']['node']->webform['components'])));

    // Every 32MB of additional memory after 64MB adds a multiplier in size.
    $memory_limit = parse_size(ini_get('memory_limit'));
    $mb = 1048576;
    $memory_modifier = max(1, ($memory_limit - (64 * $mb)) / (32 * $mb));
    $batch_size = ceil($batch_size * $memory_modifier);
    $options['range']['batch_size'] = $batch_size;

You might start just by reducing the 50000 number until things start working. If you could then report what your PHP limit is and what number worked for you (as well as how many components are in your form) that would help us adjust this guessing so we don't overshoot for other users.

Sorry for re-open it. I had the same problem (with 256Mb) and I can download large files with thousand of rows putting 500 in the $batch_size var (in place of 5000). I'm not sure the correct way for do it.

Do you think that is a good idea create a form field in the download form for configuring the $batch_size? I can create a fieldset with advanced config or something similar. Do you think that is a nice solution? Any other idea for resolve it? Really I can resolve my particular situation easily but I can contribute a solution for this problem if I can. ^^

(I change the version of the issue, really the issue is still active)

I create an extra fieldset with advanced option and added a textfield too for configuring the number of rows that the batch have to load in each process. I test it and in my project it run perfectly and it resolve my problem. I hope that it is a good solution for all.

@stijndmd please, if you already have the problem. Can you test it?

Outch, I made the patch for the dev project. Try to retest it.

Did you try the approach outlined in #2 by quicksketch? If so, what values worked for you.

We don't add esoteric configuration options unless there is no other choice. It makes the UI even more complex than it already it.

Hi @danchadwick,

You are right with the "esoteric" configuration. It's a estrange value... but I can't resolve it without it.

Some data about my case:

I tried to adjust the function but I didn't have good result, it only run correctly with very lower values 50-100-200 (in production), more less. I don't know why. I can't test with all values (in production, sorry). Obviously I can't change the production memory_limit and it isn't necessary if we can resolve this issue.

In my local environment, with 256M of memory_limit , I can use 500~2000.

The code says that we start with 50,000 per batch and it's the minimal size, for me is too hight. Do you have any other solution? Some ideas?

How many components? What is the calculated batch size using the algorithm above.

With 256Mb, you have a memory modifier of 6. So your batch size is 300000 / # components. That sounds like a big batch size.

Are you running out of memory or time?

out of memory.

I only have 9 components.

Really I don't know the problem. Can I contribute any other data?

What message to you get in the PHP server log?

What happens if you set your PHP timeout to a very long time?

My guess is that a low number of components and the high memory may be causing a timeout -- just not enough time to output all 33,333 submissions per batch.

EDIT: What is your PHP timeout?

What value of batch_size works for you (not the 50000 number, but the number after adjusting for components and memory)?

I just tested with -1 in max execution time and 1024M of memory_limit.

Result:

Hubo un error HTTP AJAX. Código de Resultado HTTP: 500 A continuación se detalla la información de depuración. Ruta: /batch?id=1272&op=do StatusText: error ResponseText:

In english...

There was an AJAX HTTP error. HTTP Result Code: 500 then debug information is detailed. Path: /batch?id=1272&op=do StatusText: error ResponseText:

I think that it is a extreme configuration for php, but I think that is a nice test.

Now we have 6666 rows with 9 fields per row. One more time I tested with 50000, 500 and it fails. With 200 it has run ok.

Can I do any other test?

EDIT: Thanks for helping me. :-) It's a little bit difficult for me resolve this kind of error.

I'm not sure whether you are testing with "200" as the $batch_size BEFORE or AFTER the adjustments for the number of components and memory size.

200 seems incredibly low to run out of memory unless you have barely enough memory to start with.

Are you sure you aren't running out of time? Did you check your PHP server log? The error you posted is just the apache internal server error (500) when PHP presumably crashed or ran out of time.

What is your PHP timeout from php.ini?

200 BEFORE adjustments. It's higher with the adjust (but not too hight really).

Ok, I found an error:

PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 32 bytes) in /var/www/MYSITE/httpdocs/sites/all/modules/contrib/views/views.module on line 2148

This is the code of the function:

2144 function _views_query_tag_alter_condition(QueryAlterableInterface $query, &$conditions, $substitutions) {
2145   foreach ($conditions as $condition_id => &$condition) {
2146     if (is_numeric($condition_id)) {
2147       if (is_string($condition['field'])) {
2148         $condition['field'] = str_replace(array_keys($substitutions), array_values($substitutions), $condition['field']);                                                   
2149       }
2150       elseif (is_object($condition['field'])) {
2151         $sub_conditions =& $condition['field']->conditions();
2152         _views_query_tag_alter_condition($query, $sub_conditions, $substitutions);
2153       }
2154       // $condition['value'] is a subquery so alter the subquery recursive.
2155       // Therefore take sure to get the metadata of the main query.
2156       if (is_object($condition['value'])) {
2157         $subquery = $condition['value'];
2158         $subquery->addMetaData('views_substitutions', $query->getMetaData('views_substitutions'));
2159         views_query_views_alter($condition['value']);
2160       }
2161       elseif (isset($condition['value'])) {
2162         $condition['value'] = str_replace(array_keys($substitutions), array_values($substitutions), $condition['value']);
2163       }
2164     }
2165   }
2166 }

I have the last view version (7.3.8). I'm searching the error trying to debug it. Any idea about the it?

This patch does three things:

1) The maximum suggested batch size in limited to no larger than 10000 to respect PHP timeout, regardless of memory.

2) The batch_size may be overriden (actual number of rows, not the 50000 that is used as a starting value before adjustment) by adding this line to your site's settings.php file:

$conf['webform_export_batch_size'] = 500;

There is no UI for this value.

3) Fixed a copy/paste error in a batch-related export doc block comment.

Committed to 7.x-4.x and 8.x.

OK, really it's a better solution :-).

Thanks!

After update, receiving same error as above. Prior to update I could download thousands of webform submissions, after the update, I can't download more than 500 to 700 submissions... PHP Memory Limit 256. Webform 7.x-4.1. Drupal 7.32. Trying to download a form with 31k submissions and 5k submissions.