Problem/Motivation
Hi everyone, hope you're all well
I'm having some trouble downloading submissions from my webform.
My form has over 20,000 submissions and when I download them, the csv file is incomplete: not all submissions were included. I've already configurated it to use batch (default_batch_export_size = 100), but it didn't work.
Somebody have any idea how can I solve this problem? I need to download all submissions (through interface) into a single file.
Thank you!
| Comment | File | Size | Author |
|---|---|---|---|
| #11 | webform-reset_entity_cache_on_batch_export-3576340-11.patch | 5.47 KB | velmir_taky |
Issue fork webform-3576340
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #4
velmir_taky commentedProblem/Motivation
When exporting submissions from a webform with a large number of entries (20,000+), the resulting CSV file is incomplete — not all submissions are included.
The root cause is that
batchProcess()inWebformResultsExportControllerloads submissions withloadMultiple()on every batch iteration, but the entity static cache is never cleared between iterations. Memory keeps growing — roughly 5 MB per 100 submissions — until PHP hits itsmemory_limitand the process dies silently mid-export.Steps to reproduce
1. Create a webform with a few fields (textfield, email, textarea).
2. Have 1,000+ submissions (the issue becomes critical around 20,000+).
3. Go to Results → Download and export as CSV.
4. The downloaded CSV will be missing submissions from the end.
Fix
Added
resetCache()on thewebform_submissionstorage after writing each batch to free the loaded entities from the static cache.Testing
Tested locally with 1,100 submissions and
batch_limitof 500:- Before fix: memory grows from 50 MB to 96 MB across 11 iterations, never stops growing.
- After fix: memory stays around 66–74 MB, stable after the first iteration.
All 1,100 rows present in the CSV, no duplicates, no gaps in serial numbers.
Comment #5
cilefen commentedThis looks similar to two existing bug reports. Can you verify whether this is a duplicate of those?
Comment #6
liam morlandComment #7
velmir_taky commentedThanks for the references. I reviewed both issues:
- #3369136 is about incomplete ZIP archives due to ZipArchive not being closed between batches — a different root cause.
- #3570311 is about Chrome blocking CSV downloads triggered via meta-refresh — a browser security issue.
This issue is specifically about entity static cache not being cleared during batch CSV export, causing memory to grow until PHP silently fails. The fix is a single
resetCache()call after each batch write. Different root cause and different fix from both referenced issues.Comment #8
larisse commentedHello everyone! Thank you for taking a look at this!
@velmir_taky, I applied your patch and, the first time, the CSV file downloaded completely. I tested the download again, but the CSV was no longer complete. So, I cleared the cache and tried downloading again, and the CSV downloaded completely.
I will investigate to see if something on the site is affecting the cache.
Comment #9
larisse commentedComment #10
liam morlandThanks for working on this. Bug reports are always against the development branch.
Based #7, it sounds like the other issues are not duplicates.
Comment #11
velmir_taky commented1. Added
\Drupal::service('entity.memory_cache')->deleteAll()after each batch iteration to also clear cached non-submission entities (users, files, source entities) that accumulate during export processing. This follows the same pattern used in Drupal core'sSqlFieldableEntityTypeListenerTrait.2. Added a kernel test (
WebformSubmissionBatchExportTest) that verifies both entity memory cache and persistent cache are properly cleared after a batch export iteration. The test fails without the patch and passes with it.@larisse Could you test this updated patch?
Comment #12
velmir_taky commentedCI results: 301/302 tests passed. The single exception (
WebformAccessSubmissionPermissionsTest) is a pre-existing infrastructure issue (RecursiveDirectoryIteratorfailure on twig template directory), unrelated to this change.Comment #13
larisse commentedHii @velmir_taky! I tested it and it worked perfectly. Patch #11 solved the problem.
Thank you so much for working on this and for explaining the problem :)
Comment #16
liam morlandThanks!