Use Reply-To instead of From e-mail headers (Google/Yahoo/etc anti-spoofing policy marks e-mails as spam)

I agree we need to update how Webform handles headers to better deal with spoofing measures taken by mail servers. They're quite right to enforce such policies, but right now Webform leaves the burden of setting up the correct FROM addresses to users, which is becoming increasingly complicated. Over at webform.com, we use hook_mail_alter() to modify all outgoing e-mails to send all e-mails from the @webform.com domain, but the reply-to header is the FROM address the user provided. I don't think that's necessarily a good solution for everyone however, as it depends on how you have e-mail configured.

For now, I think the best generic solution is to use https://drupal.org/project/webform_reply_to, which lets you manually specify a separate FROM and Reply-to header. Set FROM to be your site's domain (i.e. noreply@example.com) and Reply-to as the user's address.

This is becoming more and more common, and another reason that I think Allow setting Reply-To email header is a reasonable request.

I re-closed #872382: Allow setting Reply-To email header so we can focus on this issue here.

I'd like to avoid requiring users to understand both FROM and REPLY-TO headers in order to set up a Webform that delivers e-mail. I don't think many users would know that the FROM address needs to be the same domain as the site itself.

I'd like to propose that we add a new site-wide setting that affects how Webform handles the From address. Something like this:

From address behavior:
(•) Change the "From" address to "Reply-To" when the site domain does not match
( ) Use the entered "From" address regardless of domain
To avoid e-mails being marked as spam, e-mails usually must be sent from the same domain as the current site. Webform will always use the entered e-mail address as the From address if it matches the current domain.

And then a second option, shown only if the first option is selected above:

From address fallback:
[ noreply ]@example.com
If the From address does not match the current domain, this address will be used. Usually a "noreply" or a catch-all address.

Then when sending e-mails, we do a quick check comparing the current site's domain to the FROM address domain. If they match, we send the FROM header exactly as entered. If they don't match, we move the FROM header to REPLY-TO and replace FROM with the fallback address. This behavior would be the default, and users who wanted to send e-mails from any address regardless of the spam problem could use the second option to restore the old behavior.

This approach keeps us from adding any new fields to individual webforms, all webforms would start "working" immediately after the setting was added, and new installations would work out-of-the-box without any configuration.

How does that sound?

Updating the issue title to describe the situation a bit more.

quicksketch said,

Over at webform.com, we use hook_mail_alter() to modify all outgoing e-mails to send all e-mails from the @webform.com domain, but the reply-to header is the FROM address the user provided. I don't think that's necessarily a good solution for everyone however, as it depends on how you have e-mail configured.

That's exactly how I was thinking about handling this.

Do you feel like letting lazy programmer plagiarize your code here? A peek at how you handle the mail headers would probably save a few hours of experimentation.

Thanks,
- Tim.

Do you feel like letting lazy programmer plagiarize your code here? A peek at how you handle the mail headers would probably save a few hours of experimentation.

Haha, that's all I've ever wanted to do. ;) Here's the gist: https://gist.github.com/quicksketch/11124800

If this were to go into the Webform module itself, we could save some guessing by setting these headers directly. Though if we use the approach I described above, it would also add some kind of domain-matching test to allow FROM headers to pass through if possible.

The Webform Reply To module fixed this for me! I was completely baffled as to why we were not getting emails that were not Gmail. Glad I found this thread!

If this were to go into the Webform module itself, we could save some guessing by setting these headers directly. Though if we use the approach I described above, it would also add some kind of domain-matching test to allow FROM headers to pass through if possible.

@quicksketch -- Is this something we want to do in webform, or leave this as a site-wide issue for all e-mail? I am using something similar to your webform.com code for my domain. I suggest we either close this issue, or fix it right away, in 7.x-4.0 if possible.

Is this something we want to do in webform, or leave this as a site-wide issue for all e-mail?

Hmm, I hadn't considered a site-wide fix. I suppose we could make this some kind of stand-alone module that affected all outgoing e-mails and not just those from Webform. On the other hand, I can't think of any other e-mails in Drupal that allow the FROM address to be manually set, so for my situation, the ONLY place I'd need this to be fixed is in Webform. I imagine that's the case for most other sites out there as well; so I think it'd make sense to fix this in Webform.

I'd love to get this into Webform 4.x, but it doesn't need to be a release blocker for 4.0, since the behavior would have an option to make it backwards compatible (though most sites would want to use the new approach, considering Gmail, Yahoo, Hotmail, etc are probably marking their mails as spam right now).

@quicksketch yeah, I'm thinking "something" needs to be done sooner than later... I did some digging and it appears that yahoo implemented the reject spf back around April. It wasn't until June or so that I started receiving bounce notices from Google Apps that the yahoo and aol emails were being rejected... Who knows how many email's were black hole'd and I never knew about them.

Thing is, it's a customer service issue and ultimately "bad for business". Really bad. The sender, with the yahoo email address, has no idea that entering their email address will cause their webform email to possibly go into a black hole. The customer simply thinks the company doesn't care enough about them to respond back, so they either call with the complaint of "I sent an email", OR even worse, they find a different company.

I get why the big email companies are doing it, but in the end it hurts the legitimate folks a lot more than it hurts the spammers...

Going to look at the separate module you referenced and cross my fingers it works for 4.x.

I agree that this issue shouldn't be a stable release blocker, but if yahoo and aol email's make up 15% of all email addresses (number I pulled from thin air) then we're missing out on 15% of potential sales, customer leads etc.

Update: Yes, Webform Reply To module does with with 4.x after applying update from comment #17

Webform Reply To 7.x-2.0 has now been posted with the aforementioned Webform 7.x-4.x compatibility patch included.

For reference, the Print modules provides:

(x) Use Reply-To header
When enabled, any email sent will use the provided user and username in the 'Reply-To' header, with the site's email address used in the 'From' header (configured in Site information). Enabling this helps in preventing email being flagged as spam.

I would support a patch to roll Webform Reply To into webform core. @jerry -- Would would be interested in posting a patch??

The function would simply place the From address into Reply-To address only when the From address isn't the site address.

Maybe something like this (swiped from one of my site's hook_mail_alter):

  $site_name = variable_get('site_name', 'Drupal');
  $site_mail = variable_get('site_mail', ini_get('sendmail_from'));
  
  // To help avoid messages being flagged as spam, all message should be from
  // this domain.
  if (stripos($message['from'], $site_mail) === FALSE) {
    // Message is not already being sent from the site e-mail address.
    $message['headers']['Reply-To'] = $message['from'];
    // Match e-mails of the form 'My Name <email@domain.com>'
    preg_match('/^([^<]*?) *(?:<(.*)>)?$/', $message['from'], $matches);
    if ($matches) {
      $message['from'] =t('!name via !site_name <!site_mail>',
                                                         array('!name' => empty($matches[1]) ? $matches[2] : $matches[1],
                                                               '!site_name' => $site_name,
                                                               '!site_mail' => $site_mail));
    }
  } else {
    // Make all messages from the site be from the site name.
    $message['from'] =  "$site_name <$site_mail>";
  }
   $message['headers']['From'] = $message['from'];

^^ Forgive the whitespace /line breaks above that are not Drupal coding standards. You get what I mean. :)

The guts of this patch is:

    // If requested and not already set, set Reply-To to the From and re-format From address.
    if (webform_variable_get('webform_email_replyto') &&
        empty($email['headers']['Reply-To']) &&
        ($default_from_name = webform_variable_get('webform_default_from_name')) &&
        ($default_from_address = webform_variable_get('webform_default_from_address')) &&
        ($default_from_parts = explode('@', $default_from_address)) &&
        count($default_from_parts) == 2 &&
        $default_from_parts[1] &&
        stripos($email['from'], '@' . $default_from_parts[1]) === FALSE) {
      // Message is not already being sent from the domain of the default webform from address.
      $email['headers']['Reply-To'] = $email['from'];
      // Match e-mails of the form 'My Name <email@domain.com>' as follows:
      // ^          = beginning of string
      // "?         = optional quote
      // ([^<]*?)   = match optional characters that aren't a < (non-greedy)
      // "?         = optional quote
      // SPACE*     = optional spaces
      // (?:<(.*)>) = < matching stuff > (without the angle brakets)
      // $          = end of string
      preg_match('/^"?([^<]*?)"? *(?:<(.*)>)?$/', $email['from'], $matches);
      $email['from'] = $matches && webform_variable_get('webform_email_address_format') == 'long'
                          ? t('"!name via !site_name" <!site_mail>',
                              array('!name' => empty($matches[1]) ? $matches[2] : $matches[1],
                                    '!site_name' => $default_from_name,
                                    '!site_mail' => $default_from_address))
                          : $default_from_address;
    }

Committed to 7.x-4.x. Need port to 8.x, which I didn't make due to config differences.

Patch needs porting to 8.x

Now that this is in place, is there any reason to need this module:
https://www.drupal.org/project/webform_reply_to

Can someone who knows these modules well advise if the Webform Reply To module can be deprecated once Webform 7.x-4.x-dev is used? Or is Webform Reply To providing some other functionality?

@DarrelDuane -- From reading the recently-updated webform_reply_to project page, it would seem like that module is no longer needed if you are using webform 7.x-4.3 or later.

It seems the module is no longer required (see linked issue).

But updating to the lastest release of webform actually disabled that module automatically... somehow? Unfortunately none of my reply-to settings have been carried across and this needs to be done manually it seems: #1462192: Integrate into the Webform core module

Committed and pushed to 8.x-4.x. Thanks!