After digging into webform module I have found, that it spending a lot of memory and time when trying to get all submissions (tested on prod where are ~20K submissions)
Peak memory usage was 292.55 MB [317.47 sec, 41.89 MB]
Now memory decreased to ~140MB and time to 27 seconds for webform with ~20000 submissions.
So it is >10x speedup.
- I have used subquery ability for retrieving submissions.
- Also added static cache for function, that was called multiple times.
- Fixed trouble, when there are FALSE submissions within database and dblog dying from Notices.
- Fixed submissions counter wrong count displaying bug
needs review
| Comment | File | Size | Author |
|---|---|---|---|
| #10 | webform-webform_components_cleanup-2442917-8.patch | 1.78 KB | danchadwick |
| 10x_speedup.patch | 6.59 KB | podarok |
Comments
Comment #1
podarokComment #2
podarokComment #3
podarokComment #5
danchadwick commented@podarok - Thank you very much for your efforts at optimizing webform. In reading your patch, I have a few comments:
Please don't be deterred. I am very enthusiastic about committing effective optimizations. Also, I've adjusted the issue metadata to correspond to Issue Queue Handbook standards.
Comment #6
podarokPlease, add me to co-maintainers, I'll support 3x branch, because we have hi-load project that looks like never be updated to 4x branch
I'll reroll this patch asap
Comment #7
danchadwick commented@podarok -- As luck would have it, user torotil has just volunteered to be webform's new 7.x-3.x branch maintainer. The good news is that since both you and he are interested in maintaining this branch, you can review each other's work. He has another issue in the queue that I'm sure he'd like eyes on (#1332820: Parts of nested tokens (in fieldsets) printed when left empty). And I discussed this issue with him and he's ready to help review patches after the review in #5 is attended to.
I'm happy to see the enthusiasm for 7.x-3.x and for optimization.
Comment #10
danchadwick commentedNo activity in a month, so I went through this patch again in the context of the active branches.
1) I see static caching added to webform_component_feature(). It doesn't do much, except call webform_components(), which is itself cached. I'm unconvinced of a big win here, but i didn't do any testing to confirm this. I'd be very happy to be proved wrong.
I did make a tiny optimization to webform_components, eliminating a array_diff_key. With only a dozen component types, I doubt it helps much, but I think it makes the code clearer.
2) There is a probably big win optimization to webform_get_submissions when a header is used. Due to changes in the 4.x branch, webform_get_submissions is no longer called for the submissions or results table, as these are done with views. The 8.x branch should remove the header option probably.
3) I don't understand how the change to webform_get_submission_count would help. I didn't check the 3.x branch, but the 4.x branch has indices on webform_submissions to make queries with nid and optionally uid very fast. I did see a line of left-over code, which I deleted.
Committed to 7.x-4.x and 8.x.
Alas the big win is an optimization in code no longer used in the current branch.
Comment #11
danchadwick commentedMy thanks to @podarok for the webform_get_submissions optimization. It will end up in webform as part of the optimization and fixes for batch download. See the related issue.