Files corruption using multiple servers/containers + batch

I am seeing similar behavior in Pantheon Live environments. The export seems to work well in Dev environments. This might be related to their use of multiple containers in the Live environment.

I just encountered the exact same thing @jayhaase noticed. Dev output produces all my 10,000+ rows in csv. Live produces maybe 4,000. And the data is spordic. Sometimes missing random rows, sometimes only 200 rows, sometimes also missing the headers. Very inconsistent.

I seem to have solved the problem by setting the export directory using devel.
variable_set('views_data_export_directory', 'public://views_plugin_display');

Using the temporary directory my export would contain between 4,000 and 6,000 entries. Changing to the public gave me full 11,660 rows in the downloaded file.

I haven't found any place to actually set the variable other than through devel.

The underlying issue here is that with multiple servers or containers the batch export process is distributed between the available web heads.

In cases where the servers have different temp paths the end result is the server that served the initial batch process can be different than the second server that handles the next batch process. As a result this can fail without setting the stream wrapper to a shared space or as the DB streamwrapper as noted above.

It would be good to update the title of the issue to highlight the problem with multiple server handling the batch vs a Pantheon specific issue as Jim noted in the referenced thread https://www.drupal.org/node/1782038.

This would help users who experience this outside of Pantheon with more than one web server. That way it could be possible to converge on a solution that works for everyone as the module is quite useful.

Found same issue on Pantheon live, fixed it by adding this line in settings.php file
$conf['views_data_export_directory'] = 'public://views_plugin_display';

This hasn't worked for me. I added the variable views_data_export_directory as in #5 (with drush vset), and it looks fine in the variables table, but in a export that should have 1632 rows, I am only getting about 1425 records.

In several places on the site, I have places where users can download lists. When the page is for an internal user, I just tell them to use the test server, which we update pretty regularly, and that works, though not without grumbling. But I have some pages where outside users need to download CSVs, and of course I can't have them using test.

I am also getting the same issue as @smichel. Are there any fixes for this?

#6 solved it for me, needed to set the segments to 100 as well.

I had the same issue with Acquia just on prod server which had multiple web nodes, after setting the variable as mentioned in #3 it worked for me.

We had missing rows only in Prod in Acquia. Setting drush vset views_data_export_directory public:// fix the issue.

Probably not a good idea to set it to public://. According to Acquia documentation you should use Alternative Stream Wrappers. Or failing that private:// would be better than public://

Since this seems to fix the issue, there needs to be a UI for setting it. Some government sites are not allowed to use Drush.

I experienced this issue on Acquia servers when exporting to a file with 15,000 records in production only, which is because there are multiple servers in production. Running it on dev, in which there is only one server, worked fine.

I would get 8,000 or 7,600 or various other amounts. I hacked the module plugin and changed the temporary to private and it worked, but I am uneasy making changes into contributed modules. Are there plans to surface this into a configuration screen? And is there an alternative files directive (other than public or private) that would work across servers that Acquia recommends?

@mitchems, see comment #11.

I had the same issue with pantheon live server, after setting the variable as mentioned in #3 it worked for me. Thanks

Upping to critical since this results in data loss.

I am experiencing this issue as well and #3 seems to have fixed the problem.

Sorry for the lack of attention to your issue, please accept my apologies.

Drupal 7 is going to be end-of-life'd by the community in approximately 1 month.

As such, I am closing all non-critical looking, non-PHP compatibility issues for Views Data Export to tidy up the issue queues and reduce the noise. You can read about this on #3492246: Close down Drupal 7 issues.

If you feel like this issue has been closed by mistake, please do comment about re-opening it.
If you feel like the ticket is still relevant for the 8.x-1.x version of the module, the please search for a duplicate issue first, and if there really isn't one (and you've looked properly) then change the version on the ticket and re-open.

Thanks to everyone involved in this issue: for reporting it, and moving it along, it is truly appreciated and the Drupal community wouldn't be what it is today without your involvement and effort, so I'm sorry that we couldn't get this issue resolved. Hopefully we'll work together in a future issue though, and get that one resolved :)