When webform results have multiple files attached, it's cumbersome and time consuming to try and download all attached files even when using table view. Since results can be downloaded, it would be nice to also include all attached files in the export. We propose an archive option in the download tab. This will create a zip file that includes the exported CSV/TSV and any attached files.

We've implemented this functionality in the attached patch which can be applied against git branch 6.x-3.x (6.x-3.9). We would like to help get this into the next 6.x-3.x release if possible.

Comments

dharmatech’s picture

StatusFileSize
new4.33 KB

Patch attached.

quicksketch’s picture

Thanks, this is a nice-looking patch. If you can roll a D7 version that'll speed things along a bit, but either way I'm a bit swamped and I'm not sure I'll get to new Webform features any time soon. To make things easier yet, this patch also needs to follow a few Drupal coding standards (I use the D7 standards for both D6 and D7 for consistency): use a space on either side of concatenation dots; put else and elseif statements on the next line; use elseif instead of else if.

dharmatech’s picture

StatusFileSize
new4.29 KB
new4.34 KB

Thanks for the quick response. Attached is the patch for 6.x and 7.x. Updated to follow code convention and is now consistent with existing code.

quicksketch’s picture

Status:Needs review» Needs work

Thanks @dharmatech! These patches look great. Unfortunately I'm having a few issues after applying this patch:

- On D7, the .zip file is 0 bytes. Works great in D6 though.
- After downloading the .zip, the included file is named sample_webformcsv (where "sample webform" is the name of the node). So the dot before the extension is missing. Plus, I actually used tabs as the separator, "tsv" should be the extension, not "csv".
- The .zip option only works with the delimited text file option. If you select "Excel", it just downloads the Excel file.

This is a great patch and I'd love to include it in the project. Feel free to adjust the base class and functions if needed to accompany this new functionality.

dharmatech’s picture

StatusFileSize
new5.17 KB
new5.13 KB

Patches attached fixes the following issues:
- D7 0 byte zip file
- Add dot for file extension
- Set proper file extension depending on export type (CSV or TSV)
- Zip file generated when Excel format is selected

The diffs were generated against git branch 6.x-3.x (6.x-3.9) and master (7.x-3.9).

dharmatech’s picture

Status:Needs work» Needs review
Owen Barton’s picture

Version:6.x-3.9» 6.x-3.x-dev
Status:Needs review» Needs work

It appears that this was previously committed to at least the 6.x-3.x branch (haven't checked which version or if it is 7.x also).

I am setting back to CNW though, as the code references $base_path, which is not set anywhere (shouldn't this use something like http://api.drupal.org/api/drupal/includes--file.inc/function/file_direct... in any case?):

<?php
$zh
->addFile(realpath($base_path . $file), basename($file));
?>
quicksketch’s picture

Wow, you're right Owen! Looks like I might have accidentally committed this as part of another issue.

According to Git history, It looks like I did this as part of a security fix (issue #54554), and I only did it to the Drupal 6 version. We should roll back this change ASAP, it wasn't ready for prime-time yet.

quicksketch’s picture

StatusFileSize
new4.49 KB

I've rolled back the 6.x-3.x branch with this patch. I'm sort of upset with myself for letting this one go through in such a shady manner. It was entirely unintentional, sorry to anyone who may have had troubles caused by this (or are upset by its sudden removal). We'll get this working and included in a release soon.

quicksketch’s picture

Regarding this question:

I am setting back to CNW though, as the code references $base_path, which is not set anywhere (shouldn't this use something like http://api.drupal.org/api/drupal/includes--file.inc/function/file_direct... in any case?)

I think we can just remove the $base_path variable entirely. The file is being manipulated in the Drupal temp directory, so using of file_directory_path() is unnecessarily. Similarly, $base_path shouldn't be needed because the temp path is already relative or absolute and doesn't need a prefix.

dharmatech’s picture

Actually you do need $base_path because the ZipArchive handler needs the full path to the files. ZipArchive doesn't have a concept of "working directory" that I could find.

quicksketch’s picture

Actually you do need $base_path because the ZipArchive handler needs the full path to the files.

Hmm, sounds like we need to use realpath() first then, because the tmp file directory may be either relative or absolute when defined by the user.

Sheldon Rampton’s picture

I experienced a problem when downloading submission results as a delimited text (tab delimiter) report using the 6.x-3.14 version of the webform module. I'm documenting my experience here in case it may be helpful for debugging this feature.

The webform which gave me the issues included a field for uploading a file. Our website received more than 3,000 submission results. Upon attempting to download with the "include files" option checked, I experienced errors in the form of an invalid .zip archive with a filesize of Zero KB. I tried expanding the zip archive anyway, and it produced a file with a filesize of 4KB and an extension of ".zip.cpgz".

On the theory that webform was choking on the large number of files in our submission results, I tried downloading them in batches by using the "All submissions starting from: to (optional) end" option and entering a partial range of submission IDs. This produced fatal PHP errors if the range between the starting and ending sids was too large, but I was able to successfully download everything in 13 separate batches by experimenting with sid ranges until I found ranges that worked. I seemed to be able to download successfully if the filesize of the resulting .zip archive was 100MB or smaller.

quicksketch’s picture

but I was able to successfully download everything in 13 separate batches by experimenting with sid ranges until I found ranges that worked.

We need to implement #1327186: Use BatchAPI to Export very large data sets to CSV/Excel to help solve this problem, though I'm surprised just 3,000 submissions caused this problem for you. What's your PHP memory_limit as reported by phpinfo()?

Sheldon Rampton’s picture

Our PHP memory_limit is 256M. I think the issue probably has more to do with the total size of all of the uploaded files, which adds up to roughly 1 gigabyte.

dharmatech’s picture

In this case, is the problem w/the size of the attachments or the size of the CSV? Maybe we should be splitting the files into chunks and provide a download link rather than sending the file immediately?

quicksketch’s picture

Yeah, the issue in #1276098: Add Drush command for bulk export took this approach of writing to a tmp file over multiple requests. I'd prefer to avoid a separate download link and instead stream the file, since we'll need to protect that file somehow and automatically delete it after some period of time.

Owen Barton’s picture

I can't think how we would use Batch API and stream the file, since Batch API ties up the browser (with the JS/reload code), adn we can't produce large files without Batch API due to PHP timeouts. Might be possible with crazy iFrame stuff, but I think the download link approach is probably more elegant. I don't think there is a way to store the file contents in Batch API or cache tables without loading it all into RAM at some point, which is problematic, so I think a filesystem file is the way to go. The file could still live in /tmp and be streamed (using the same technique as private files, but could be without depending on it), which prevents us needing to load the whole thing into RAM, and means we know when it has been successfully downloaded, so we can clean it up.

dharmatech’s picture

Any progress? Batch API seems a bit of a diversion unless someone can provide a patch. There have been 6 3.x releases since the original patch, it would be great to see this in an official release.

aaronpinero’s picture

Hello all. I am looking to update to the latest D6 webform release. However, I wanted to include (if possible) the patch for downloading files uploaded as part of webform submissions. Is it possible for me to use one of the patches in this issue? If so, which one? Thanks...

dharmatech’s picture

I doubt any of the patches will apply cleanly now. The last one I posted #6 works quite well unless you have very large attachments (upwards of 1GB based on Sheldon's experience). We are approaching a year now since first posting a patch and looking back through the comments it's a stand still because there isn't a clear direction. This makes it hard for others (including me) to continue to contribute patches for this functionality.

The download link makes the most sense to me and I think Owen walks through most of it #19.

dromansab’s picture

Hi,

what about Drupal 7 version? I've tried it and it doesn't work. The system downloads an empty file...

I have to install a PHP library?

Thanks

marameodesign’s picture

Version:6.x-3.x-dev» 7.x-3.9

Me too. I am looking for this approach in D7 - anyone has successfully used this patch ?

Thank you

DanChadwick’s picture

Issue summary:View changes
Status:Needs work» Closed (won't fix)

Closing for lack of activity. In all these years, there aren't that many people that want this. With the changes in download options, this would be a bit more complicated now.