This is a Drush command that is capable of very large exports (it is in production use on a site with 137k records) with limited memory overhead and pretty reasonable performance (the 137k records take ~1m10s to export). It has a few options to allow you to tune the batch size, and also select specific start/end batches (and skip the header, in case you want to keep appending to a single file). It can output to stdout, or a named file. There is not yet support for selecting exporters, delimiters etc, but that could be added fairly easily (to do it right, exporters should each define their own available options in a hook - the Drush "engine" system could perhaps be used for this). The file itself could either live in the top level directory (perhaps good to have it a bit more visible, given that it is an "entry point") or in /includes.
This is dependent on the webform.report.inc refactoring patch in #1275468: Improve webform.report.inc abstraction to support drush/batch processing.
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | webform.drush_.inc_.txt | 4.14 KB | owen barton |
Comments
Comment #1
quicksketchThanks Owen! Sounds cool, but looks like we're missing the patch?
Comment #2
owen barton commentedd.o. file name rule messages catch me out every time (it shows ext warning, but then tells you it has renamed it for you...).
Comment #3
quicksketchThis feature looks cool but we should merge it with this request: #1327186: Use BatchAPI to Export very large data sets to CSV/Excel. I don't want to have this drush command using specialized code when we could use the same code to solve two problems.
Comment #4
quicksketchLooks like this patch has been merged into #1275468: Improve webform.report.inc abstraction to support drush/batch processing, which is fine with me since it shows *why* the changes are needed. Though I'd still prefer that the UI gets batch API first... oh well. In any case let's convene in that other issue.