out of memory on large CSV exports of submissions

I am all for using something else or having something else as an option.

But I think it would be better as entity type agnostic module. I could see other use case where you might need a export of other entity types to CSV file.

Do you still get the out of memory if you use the Batch option on Views Data Export?

Hi all, I can propose an alternative way for export submissions to csv file.
This module in sandbox, but I think this module will be useful to you Entityform Submission export

@semihin_kirill, thanks for the link and an interesting module. I am sure many will find it useful if they don't want to deal with Views Data Export.

I think Views Data Export is working well in most cases.

This may not be joachim's original problem but there is still a problem with batching. The symptoms include slow batching, out of memory errors and duplicate fields in output when using views_data_export and the autofields function.

The problem occurs because entityform_views_pre_view() is executed each batch segment rather than once per export.

_entityform_view_add_all_fields() is then fired off for each batch and adds the fields without checking if they're already present.

A quick and dirty method to get around this is to surround $view->add_item() with an if statement:

if (!isset($view->display_handler->display->display_options['fields'][$field['field_name']])) {
  $view->add_item($display_id, 'field', "field_data_{$field['field_name']}", $field['field_name'], $field['options']);
}

A nicer approach might be better integration with Drupal's batching system.

If you see in the 7.x-2.dev version I have added a hook so that others modules can change the fields that will be auto added. I also but some settings in the display object to flag that "autofields" were added via Entityform: http://cgit.drupalcode.org/entityform/tree/entityform.module?h=7.x-2.x&i...

_entityform_view_add_all_fields() is then fired off for each batch and adds the fields without checking if they're already present.

I haven't tested the batch yet but if this is being re-run every time then I could potentially just check for this flag in entityform_views_pre_view

@imclean I have also thought about adding your "quick and dirty" code but for another reason. In the "autofields" logic I added all the fields for the view mode. But if they View they are using already has a particular field it would be added twice. I could see a case where you want most fields auto added but want to add 1 or 2 field your self because you want a specific setting in the field in Views(like re-write).

Re-closing this issue and created another issue to handle duplicate fields #2340989: Fields added multiple times when using Views Data Export batch mode

There is a patch there that needs review.

> A nicer approach might be better integration with Drupal's batching system.

Can you explain more about what you mean by that

The view doesn't need updating each iteration, it appears to be cached. This has taken me a little while to track down and I haven't really thought about the specifics of making entityform "batch aware".

Items to be worked on should be retrieved inside each iteration of callback_batch_process().

The view can be retrieved each iteration but the extra fields don't need to be added each time.

Looking at views_data_export.module it does appear to cache the view in the table "views_data_export_object_cache". This would make sense as it shouldn't need to generate the view each iteration.

I haven't tested the batch yet but if this is being re-run every time then I could potentially just check for this flag in entityform_views_pre_view

This could be a good approach. I'll review the new issue.