Problem/Motivation

Hi everyone, hope you're all well

I'm having some trouble downloading submissions from my webform.

My form has over 20,000 submissions and when I download them, the csv file is incomplete: not all submissions were included. I've already configurated it to use batch (default_batch_export_size = 100), but it didn't work.

Somebody have any idea how can I solve this problem? I need to download all submissions (through interface) into a single file.

Thank you!

Issue fork webform-3576340

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

larisse created an issue. See original summary.

velmir_taky made their first commit to this issue’s fork.

velmir_taky’s picture

Status: Active » Needs review
StatusFileSize
new898 bytes

Problem/Motivation

When exporting submissions from a webform with a large number of entries (20,000+), the resulting CSV file is incomplete — not all submissions are included.

The root cause is that batchProcess() in WebformResultsExportController loads submissions with loadMultiple() on every batch iteration, but the entity static cache is never cleared between iterations. Memory keeps growing — roughly 5 MB per 100 submissions — until PHP hits its memory_limit and the process dies silently mid-export.

Steps to reproduce

1. Create a webform with a few fields (textfield, email, textarea).
2. Have 1,000+ submissions (the issue becomes critical around 20,000+).
3. Go to Results → Download and export as CSV.
4. The downloaded CSV will be missing submissions from the end.

Fix

Added resetCache() on the webform_submission storage after writing each batch to free the loaded entities from the static cache.

Testing

Tested locally with 1,100 submissions and batch_limit of 500:

- Before fix: memory grows from 50 MB to 96 MB across 11 iterations, never stops growing.
- After fix: memory stays around 66–74 MB, stable after the first iteration.

All 1,100 rows present in the CSV, no duplicates, no gaps in serial numbers.

cilefen’s picture

This looks similar to two existing bug reports. Can you verify whether this is a duplicate of those?

liam morland’s picture

Version: 6.3.0-beta7 » 6.3.x-dev
velmir_taky’s picture

Thanks for the references. I reviewed both issues:

- #3369136 is about incomplete ZIP archives due to ZipArchive not being closed between batches — a different root cause.
- #3570311 is about Chrome blocking CSV downloads triggered via meta-refresh — a browser security issue.

This issue is specifically about entity static cache not being cleared during batch CSV export, causing memory to grow until PHP silently fails. The fix is a single resetCache() call after each batch write. Different root cause and different fix from both referenced issues.

larisse’s picture

Version: 6.3.x-dev » 6.3.0-beta7
Category: Bug report » Support request
Status: Needs review » Needs work

Hello everyone! Thank you for taking a look at this!

@velmir_taky, I applied your patch and, the first time, the CSV file downloaded completely. I tested the download again, but the CSV was no longer complete. So, I cleared the cache and tried downloading again, and the CSV downloaded completely.

I will investigate to see if something on the site is affecting the cache.

larisse’s picture

Category: Support request » Bug report
liam morland’s picture

Version: 6.3.0-beta7 » 6.3.x-dev
Issue tags: -Possible duplicate

Thanks for working on this. Bug reports are always against the development branch.

Based #7, it sounds like the other issues are not duplicates.

velmir_taky’s picture

Status: Needs work » Needs review
StatusFileSize
new5.47 KB

1. Added \Drupal::service('entity.memory_cache')->deleteAll() after each batch iteration to also clear cached non-submission entities (users, files, source entities) that accumulate during export processing. This follows the same pattern used in Drupal core's SqlFieldableEntityTypeListenerTrait.

2. Added a kernel test (WebformSubmissionBatchExportTest) that verifies both entity memory cache and persistent cache are properly cleared after a batch export iteration. The test fails without the patch and passes with it.

@larisse Could you test this updated patch?

velmir_taky’s picture

CI results: 301/302 tests passed. The single exception (WebformAccessSubmissionPermissionsTest) is a pre-existing infrastructure issue (RecursiveDirectoryIterator failure on twig template directory), unrelated to this change.

larisse’s picture

Status: Needs review » Reviewed & tested by the community

Hii @velmir_taky! I tested it and it worked perfectly. Patch #11 solved the problem.
Thank you so much for working on this and for explaining the problem :)

liam morland’s picture

Status: Reviewed & tested by the community » Fixed

Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.