Webform currently hardcodes an LF in delimited exports.
A lot of people using Windows want to open these exported CSV/TSV files in software that expects CRLF line-endings.
We recently had this request from a client. Instead of extending the exporter class I though it might be a better idea to provide this as a configuration option, as I guess more people would want this functionality.

Patch will follow.

Comments

rv0’s picture

Status: Active » Needs review
StatusFileSize
new6.09 KB
danchadwick’s picture

Status: Needs review » Needs work

I'm a bit plus/minus on this. Most windows applications can open text files with unix LF line endings. For example WordPad will (but Notepad won't).

Assuming for the moment that we want to add this feature, I don't think it warrants a global default. We have a number of settings that one might want a global default for, and the list keeps expanding. I'm not sure what the right way to handle that is, so I hesitate to add a little-used option to the admin options.

I also question the need for CR. This was the format used by the Macintosh before OS-X. Who would use this options? This would allow the option to be a simple question: Line endings: (x) Windows (CRLF) ( ) Unix/universal (LF) -- for example.

Last, the options as described in the drush command should match.

rv0’s picture

Status: Needs work » Needs review
StatusFileSize
new5.94 KB

Hi DanChadwick, thanks for your review!

The need for CRLF: (sadly) a lot of companies and organizations rely on software and protocols that aren't that flexible when it comes to how data is handled. In my specific usecase, the whole backend administration of a town (±60000 inhabitants) runs on these CSV files with CRLF. They can't just change the software to use LF because that would break all other programs that feed csv to it. In general, most applications that do csv exports have 2 options: delimiter and line endings, so it does make sense to me (hence me writing this as a patch instead of just another contrib add-on module.)

The global default: I guess I can remove that if you want.. It just made sense as the delimiter setting is also global and both are kinda related. Perhaps I just hardcode it?

The need for CR: none really.. I removed it.

Not sure what the issue is with the drush command?

In any case, here's a new patch

danchadwick’s picture

Status: Needs review » Needs work

I installed your patch. Thank you for the effort. I just can't get beyond the amount of extra UI complexity it adds that hardly anyone needs. Webform has 400,000+ installations, and as far as I know this is the first issue to request CRLF export files.

Am I correct that for your client, 100% of the webform downloads need to be CRLF (or at least can be CRLF)? I'm thinking that we just introduce a blind (i.e. no UI) config variable for the terminator. Its value would either be "\n" or "\r\n" -- i.e. the actual terminator.

Further, I might be inclined to change the default to CRLF on the theory that just about anyone can accept a Windows file because they are so ubiquitous. I'm not sure about this though.

So if someone wants to override this, they can either put a variable into the database or edit settings.php with $conf['webform_csv_line_ending'] = "...";.

Thoughts?

rv0’s picture

I'm ok with a blind config variable!
As for changing the default to CRLF: NOOoooo!! I bet that would break 1000s of workflows, as you say yourself: 400,000+ installations!

Shall re-roll a patch soon, got some other stuff to do first!
How/where should I document it?

danchadwick’s picture

Status: Needs work » Fixed
StatusFileSize
new2.51 KB

I took care of the patch, in part because I wanted to tweak the variable name. Thanks for your work.

Re documentation: if you could find a place (or make a place) on the community documentation pages for Webform and document this, that would be great. There are other "blind" configuration variables to, such as using legacy tables rather than views for the results tables.

Committed to 7.x-4.x.

danchadwick’s picture

Version: 7.x-4.x-dev » 8.x-4.x-dev
Status: Fixed » Patch (to be ported)

Needs D8 port. Because CMI.

  • DanChadwick committed 63cde39 on 7.x-4.x
    Issue #2533010 by rv0, DanChadwick: Support exporting with alternative...
rv0’s picture

Version: 8.x-4.x-dev » 7.x-4.x-dev
Category: Feature request » Bug report
Status: Patch (to be ported) » Active

Sadly, this change doesn't work the way it should.
if you variable_get the line_ending, you get a literial string (ie.: you get the equivalent of '\n' instead of "\n")
The resulting CSV file wil just contain the string \n at the end of every line, except when using the default setting.
This is why in my original patch I hardcoded the line endings.
This is also why the webform_exporter_delimited class contains code like this:

    // Convert tabs.
    if ($this->delimiter == '\t') {
      $this->delimiter = "\t";
    }

I did a little research and it seems like there isn't a good fix for this, it cannot just be converted: https://bugs.php.net/bug.php?id=43901
I also read:

you can call functions within string escapes, so this is just as dangerous as eval() anyway.

So basically, I think the variable method of storage should be changed back to CR/LF/CRLF/.. strings like I did in my original patch, and do the conversion in a switch case.

I'm willing to provide a new patch for this, what do you think @DanChadwick?

danchadwick’s picture

Version: 7.x-4.x-dev » 8.x-4.x-dev
Category: Bug report » Task
Status: Active » Patch (to be ported)

Webform provides no UI to save this variable. You get back from variable_get exactly what you set using variable_set or $conf in settings.php. The issue is that *you* set a literal backslash rather than the desired newline character.

So you want something like $conf['webform_csv_line_ending'] = "\r\n", rather than with single quotes.

rv0’s picture

@DanChadwick
Sorry about that, you are right. I had set the variable using drush vset, it appears there's no way to do it like that (or at least, not a way I know of).
See:

variable_set("test", "test\n");
rv0$ drush vget test
test: "test\n"

rv0$ drush vset test "test\n"
test was set to "test\n".                                                                                                                   [success]
rv0$ drush vget test
test: test\n

Funny! See how the output is different the 2nd time I run vget.. I'll see what I can find about this, at least it is documented here already ;)

fenstrat’s picture

Version: 8.x-4.x-dev » 7.x-4.x-dev
Category: Task » Feature request
Status: Patch (to be ported) » Fixed

Committed and pushed to 8.x-4.x. Thanks.

  • fenstrat committed 2a799e2 on 8.x-4.x
    Issue #2533010 by rv0, DanChadwick: Support exporting with alternative...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.