Inconsistent results when doing batch exports

I have the same problem but turning off the batch export isn't an option because we need to be able to export large datasets (~8000 rows).
When I export more than a certain amounts of rows (it's difficult to know how much exactly, maybe 2000 rows) the problem appears.
As for @danharper the amount of rows exported is correct but some rows replace others.

I reproduce the problem with xlsx, xls and csv.

We solved our problem : the only sort criteria for our view was the creation date of the users.
This date was imported via migrate and similar for a lot of users so when each query generated by VDE was executed, it retrieved the users in an order that was not reliable.
So for example the batch n°150 retrieving the list of users with and offset of 500 and sorted only by creation date could very well find the same id as the batch n°151.

We needed to add another sort criteria with an unique id (here the uid) to make the batched export work.
So I don't know for @danharper but for us it was a mysql technicality and not a bug.

Indeed #3 helped me fix the view, thanks.

Duplication issues also raised here on the original batch support tread but not addressed
https://www.drupal.org/project/views_data_export/issues/2789531#comment-...

I'm also experiencing duplications (5-10%) which don't alter the total number of results, eg results are being overwritten by duplicates in the export. I tried switching the view sort order from a user created date o a UUID sort, which stopped the duplicates appearing in the export, but this is just a workaround for a bug which needs attention.

I have the same problem and I apply solution of #3 and it's works for me.

I think the problem is generated by order of view, and number of iteration of batch. In my case, I have 1 view with content type, and reference 9 paragraphs. This generate 9 rows.

Content 1 Pragraphs 1
Content 1 Pragraphs 2
Content 1 Pragraphs 3
Content 1 Pragraphs 4
Content 1 Pragraphs 5
Content 1 Pragraphs 6
Content 1 Pragraphs 7
Content 1 Pragraphs 8
Content 1 Pragraphs 9

The iteration of batch is defined to 5.
And the view is order date created of Content.

I think that, when batch get 5 elements with sql consult, return this.

Content 1 Pragraphs 2
Content 1 Pragraphs 4
Content 1 Pragraphs 3
Content 1 Pragraphs 8
Content 1 Pragraphs 6

Then the next sql return me 5 elements more, ordened by date of content type, and this for me, generate dupicate rows of paragraphs, and remove other paragraphs.

Content 1 Pragraphs 2
Content 1 Pragraphs 4
Content 1 Pragraphs 3
Content 1 Pragraphs 8
Content 1 Pragraphs 6
Content 1 Pragraphs 1
Content 1 Pragraphs 2
Content 1 Pragraphs 4
Content 1 Pragraphs 9

I solved order the view with ID paragraphs unique value. And it's works for me.

Same story here, an additional order criteria helped to solve the issue. see #3
The bug is nasty and hard to grasp: The export is working and even the number of records is matching. You have to drill into your data to see the problem.

This bug does really need more love. Also, in the first place I was wondering, why we need to batch execute an export with only 10k rows anyway ...

We have encountered the same issue, and #3 fixes our export.

We had to add two IDs, the Content ID and the Author ID, for the export to work correctly.

We were seeing this problem with an export too, only our export was losing hundreds of records. The table was sorting by creation date and the problem was that migrated data from a number of years back had the same creation date, and due to problems with the pagination query it would show a provide result set over time. I added an extra sort field for the node ID, which is after the primary sort mechanism (the previously mentioned creation date), and now it works properly.

It might be worth adding some official documentation about this, maybe even add something in the display plugin's form, as I suspect there might be lots of sites having problems with this without even knowing it.

Retitling the issue as it's a general problem of inconsistent data, rather than just duplicate records.

I've identified the source of the batch export limit problem. It has to do with the sequence of query results.

In order to prevent changes in the query result sequence, add the unique fields to the sort.

I updated the project page with a note about this issue. Thanks all for digging in and figuring this out!

This MR updates the README as well.

Looks good, this was helpful for me too!

Committed the change to the readme.

1- Will this solution still work if i am already having sort criteria like changed,published and on top of that if i applied nid,uid to that sort criteria. Or it should be only nid or uid sort criteria.

2- will this help in cron job also giving inconsistent result if i am using batch process dataexport::processBatch

I have a case where my view is based on a Search API index. I encountered a similar problem with the Batch option. However, there seems to be no sort option I can apply through this view that will fix the issue. It's possible that this is because it's Search based. I updated my search index to include NID as a field, but adding this NID as a sort option did not fix the problem with the batch output. It's possible this is not fixable for Search-based views without an actual software fix.

Hello,
Just for the record I am facing the same issue as @aaronpinero comment #3120563-22: Inconsistent results when doing batch exports. If I find something relevant I'll post it here