Export is incomplete when using batch mode

I have a similar problem:

1. This does not occur unless run in batch mode
2. In my case it is an XML export
3. If the batch size is 100, then at approximately every 475 lines of XML output the current node element is closed and the same node element is opened. The export continues with whatever was not completely put on on the current node. It then runs for another 475 lines and repeats the same behavior.
4. If I increase the batch size to 200 then the above occurs after 948 lines of XML have been printed.
5. If I decrease the batch size to 50 then the above occurs after 238 lines of XML have been printed, then again after 487 lines are printed, then again at 735 lines, then again at 983 lines, etc.

Subscribing.

I'm also seeing this issue on large XLSX exports. What's especially strange is the error is not happening consistently. I've lowered my batch size to 10 in the meantime, in case the database load is part of the issue, but it would be good to know where the error lies exactly.

My exports are really large, so I'm sure that's not helping. The view pulls together nodes that have something like 80+ fields that are being referenced. No small job!

I get this exact error using the steps in the original post to recreate. The outputs are always in round number of batches.

Report outputs vary in size and in fact once the batch report has been run and the 'click here to download' page is shown, clicking the same download link repeatedly gives a different file each time with a varying number of rows.

I watched the process through a debugger and see what is happening:

The call stack is all based on batch.inc. In my case the batch size is 100. So the system reads 100 lines from the view and then fires off code in views_data_export.
The call stack will eventually get to line 277 of plugins/views_data_export_style_export.inc
This will call the theme, and since I'm exporting to XML, it will get to theme/views-data-export-xls-body.tpl.php
That script reads those 100 lines that have been stored in an array and prints out XML.

The problem is that since my elements are about 23 lines long, the 100th line is in the middle of the fifth element. So four elements print fine, but the 5th element gets clipped to whatever has been read up the 100th line. All the script can do is open an element, print the XML for whatever it's got and close the element.

The call stack is repeated and lines 101 to 200 are read. When it gets to the theming and printing, it then prints an element for whatever had been clipped from the previous cycle and goes on with the next four elements. At line 200 it then will print another clipped element.

Here is the work around we used. It's specific to our case: batch size 100, exporting to XML and overwriting the views-data-export-xml-body script. So it won't help you unless you adapt it to your own case. I hope it's useful to someone or that it can help in developing a patch.

Having this issue too. This is hugely necessary for running a custom sales report with Commerce.

Setting to critical because essentially this means Views Data Export is useless.

Can the maintainer take a look and help us on a solution?

Many thanks.

Forgot to set to critical.

Having this issue too, wanna to find a way out.
Now my data size is not that huge, still can use non-batch mode as workaround.
But is there any solution to fright for not stable about the batching?
Anyone knows the root cause?

Subscribing.
I too am having a similar issue and need a fix. In my case I've made my own bulk action programatically to create a PHPExcel excel export of user information for a view. Out of 2600ish records I only get 924 or 1001. If I find any thing out I'll post back here.

I am also facing same issue. Not all of the records are getting download and also I am facing one more issue.
The other issue I am facing is 'No active Batch' error and my download stops with Ajax error message. But this is not frequent

Please create another thread for the other issue. It may not be related to this discussion.

@charubachi: That "other" issue is related to these: #961508: Dual http/https session cookies interfere with drupal_valid_token(), #471970: DrupalWebTestCase->getAbsoluteUrl should use internal browser's base URL and the proposed fix in #2245003: Use a random seed instead of the session_id for CSRF token generation

Figured out what my specific issue was. PHP has a setting in the .ini file that limits the maximum number of post requests.

An example. If I had 10,000 items I was trying to bullk export. If you count the post requests you will still get all 10,000 items, but only 1,000 of them are populated.

In my specific case before Prod and Dev max_input_vars were set to different values.

PHP 5.3.9 brought in max_input_vars. This defaults to 1000.

Hopefully this helps someone else.

I can confirm that this bug appears when using view data export in batch mode. I have around 350 entities and sometimes it exports 300, sometimes 200, sometimes 50 and so on.

If your site is running in multiple servers, I think this problem is related with this another issue: #2352763: Files corruption using multiple servers/containers + batch
The default folder to save the file is the temporary path, and this folder normally is not shared with multiple servers. Setting the variable "views_data_export_directory" to "public://" or "private://" can solve this problem ('public://' and 'private://' need to be shared with all servers).

Thank you,
I noticed the other issue after posting this and yes most probably it is because we are using a load balancer for which we removed the session cookie. I will know for sure in a few days when I set the public:// folder as temporary folder for views export data.

Anyone can help this? I got the issue same with #15, when I use view data export in batch mode to export the all users(total 14000), sometimes exports 8000,sometimes 6000, sometimes 7000 .

@ben600324 if you are dealing with multiple servers try #16. For me setting the export directory as explained in #16 solved the issue.

Using #16, and going to the link, it looks like Pantheon had an issue with this. So on the link from #16, #6 solved it for me, needed to set the segments to 100 as well.

#20 didn't work for me in pantheon production server. Version: 7.x-3.0-beta7

Has anyone reproduced this on PHP 7?

See #2352763: Files corruption using multiple servers/containers + batch for Acquia servers.

I have the same issue and just debugged it. The problem is that the Excel file is written to in theme_views_data_export_xlsx_body function, which basically means during every single batch iteration a new excel file is created and filled with the result set of that batch only. Thus the final file you get as download only contains the very last batch.

So mvonfrie were you able to identify the solution to this issue? I am facing a similar issue.

See #16.

I confirm that #16 works when the site is running on more then one server. I did:
drush vset views_data_export_directory "public://"

Thanks, could this at very least get documented in the read me of this module.
#16 worked a charm.

In my particular case, batched exports were falling over when they got to a row containing a special character:
'Exception: SQLSTATE[HY000]: General error: 1366 Incorrect string value: \'\\xC5\\xB5p Ll...\' for column \'users_name\' at row 1' in views_data_export_plugin_query_default_batched->execute()

This was due to incompatible character set settings on the database, the default was latin1 (and i assume that batch creates a temp table with this default character set).

I fixed this by updating default character set to utf8:
ALTER DATABASE CHARACTER SET utf8 COLLATE utf8_unicode_ci;
(NOTE: i do not know what possible side effects this may have, so please don't run directly on production DB)

This may be a very specific case, but hopefully could help someone.

I'm trying to export the user list. If I use the Batch option, Role column is empty in CSV. any idea?

Same issue as rrishnajee. When I turn on batch mode, role column is empty in CSV.

Drupal 7.88
Views data export 7.x-3.2
PHP 7.4.27
MariaDB 10.4.22
nginx 1.16.1