In the fight against spam more and more ISP's require mail send though the php function to originate from the main domain. The contact form however uses the senders email as "from:" address.
I'd like to see a field in settings to optionally set a fixed "from address" (for example contactform@domain.tld) and use "reply-to" to enable replies to the sender, while still using the fixed from address which is accepted by the isp.
Current Status
This issue was fixed for Drupal 8: committed to the 8.3.x branch on 2 August 2016 and the 8.4.x branch on 27 January 2017.
As of 29 October 2017, the issue is RTBC for Drupal 7, so it may be fixed in an upcoming release of Drupal 7.
Until then, you can either apply the patch from this issue or use a contrib module to get the same effect. One module mentioned in the comments below is Contact Reply-To. There may be other options.
Comment | File | Size | Author |
---|---|---|---|
#216 | core-use_reply_to_instead_of_from-111702-216.patch | 6.22 KB | pandaski |
Issue fork drupal-111702
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #1
Crell CreditAttribution: Crell commentedDrupal 5 has been released, so new features go to Drupal 6. That said, it sounds like a reasonable feature.
Comment #2
Oceria CreditAttribution: Oceria commentedLooking into the code I think this is a matter of adding only 3 lines extra, although I am not a php/xhtml coder (hence the lack of code in my post). However, it should be relatively simple and might be added in version 5.1. However I agree I should have set it to 5.x-dev, and not 5.0.
PS, I see my link is malformed: this is the correct link:
PCExtreme
Comment #3
Crell CreditAttribution: Crell commentedIt doesn't matter how many lines of code it is. A stable release is feature-frozen by definition. Drupal 5 will only be accepting bug fixes from now on.
Comment #4
Oceria CreditAttribution: Oceria commentedFair enough. But could somebody at least look into the code and help me to set a fixed From address and use the reply-to for the mailers email? I realy would like to upgrade my ageing 4.6 site to 5.0, but I think a site should have a working* feedback mechanism. The feedback module was easier to hack myself ;) and I don't want to wait until 6 comes out!
* Mind you, this is not a Drupal problem, but a hack to circumvent ISP policies.
Comment #5
PasqualleComment #6
Damien Tournoud CreditAttribution: Damien Tournoud commentedThis is a sensible feature request.
Were duplicates:
#195298: From addresses showing Recipients addresses in e-mails sent to contact requesters.
#350062: Site wide contact wrong from address
Also please keep in mind this recent change in the D7.x branch: #330084: Setting Reply-To in mails to the same as From is redundant
Comment #7
Dave ReidFeature requests are closed in D7 now.
Comment #8
GuyPaddock CreditAttribution: GuyPaddock commented#611176: Big spam block issue. From address is user's input address. appears to be a duplicate of this as well.
Comment #9
notrab CreditAttribution: notrab commentedI am new to drupal and am having the problems with gmail blocking contact form emails due to the from address spoofing, etc. Is there any update on this? Is there a patch I can apply to fix the from address and make the reply-to address be the users address? I thought this was a patch, but I don't see a patch file. Does the comment related to D7 above mean that this will be fixed in Drupal 7? Any suggestions for a newb would be great.
Comment #10
Crell CreditAttribution: Crell commentedFeature requests go only against the development version, and Drupal 7 is in feature freeze. This may be fixed in Drupal 8. Please do not change the version tag on an issue back to a stable release.
This could be justified for Drupal 7 if it could be argued that it's security related, but otherwise it's like so many issues that just never made it onto anyone's radar for this release.
Comment #11
Oceria CreditAttribution: Oceria commentedFixed for a 4.x module ( http://drupal.org/node/70505 ), originally requested in this topic for 5.x in 2007, rightfully bumped up to 6.x-dev, juped to 7.x-dev in 2008 without a word and now again in 2010 to 8.x-dev.
This problem seriously limits the usefulness of the mailsystem in Drupal for a lot of people using shared hosting. I cannot believe none of the developers runs into this issue. Are you all running dedicated servers?
Comment #12
Crell CreditAttribution: Crell commentedOceria: "none of the developers" implies that there's some small cadre of people who are the only ones that touch code. :-) If it's a problem you're having, patches are welcome from anyone. If you don't know how to write the patch, find a friend who does.
None of the shared hosts I've been on have had an issue here, so I couldn't really test it either way.
[Edit: Wow that was bad grammar. Fixed.]
Comment #13
Dave ReidMoving to the mail system component, as this can only be addressed in drupal_mail() itself, not contact.module.
It actually wouldn't be that hard to write a small module that implements hook_mail_alter() and adjusts the mail headers itself.
Comment #14
Oceria CreditAttribution: Oceria commentedThere you have it Crell: I am not a good coder and simply do not know where to begin. I probably have been looking for the code in the wrong place all along. I have asked for a pointer as to where to look for the code (http://drupal.org/node/111702#comment-496357) and given at least a set up for a patch (http://drupal.org/node/70505). I'm willing to look into it again, if I know where to look into the code. I was not able to grep it out myself back then (which on it's own is proof of my lack of skils).
Comment #15
chriscjcj CreditAttribution: chriscjcj commentedThank you for posting about this issue.
I host my site at Bluehost.com. They have a very strict mailing policy which effectively breaks Drupal's e-mail functionality. Bluehost's policy dictates the following:
1. You MUST use bluehost's servers to send e-mail. SMTP modules for Drupal which attempt to use an external SMTP server (like Gmail) are blocked by Bluehost.
2. E-mail addresses in the "FROM" field MUST be populated with an address that is configured as a valid e-mail address in your Bluehost account. You can only add e-mail addresses to your Bluehost account with domain names that you have legitimate control of. (For instance, I could not configure a Yahoo e-mail address as an address on my Bluehost account.)
Here's how item #2 breaks Drupal: If you have a user attempting to send another user a message using the contact form, Drupal will populate the "FROM" field with the originating user's e-mail address. When this happens, Bluehost will send the message, but it will overwrite the e-mail address in the "FROM" field with "your-bluehost-login-name@yourbluehost-box-number.bluehost.com." (i.e. jsmith@box341.bluehost.com) This address will make no sense to the recipient and if replied to, will not reach the sender of the message!
What is desperately needed, is the ability to populate the "FROM" field with an "admin" or "no-reply" account, which the Bluehost user can configure in their account. THEN, add a "REPLY-TO" field which is populated with the sender's e-mail address. (Bluehost doesn't care what you put in the reply-to field.) That way, the "FROM" field is representative of the domain running the Drupal site, and the recipient can also reply to the sender.
I would love to write patch for this, but I lack the necessary knowledge of how to do this at the moment. While far from being an expert, I have done a fair amount of PHP programming, so perhaps I will be able to figure it out. I don't really have a choice. Either I figure it out, or I don't launch the site. I'll report back here if I am successful.
Comment #16
chriscjcj CreditAttribution: chriscjcj commentedI finally figured it out. The drupal module called "PHPMailer" effectively solves this issue. If you're hosted on Bluehost, here's what you need to do. Some of the steps are applicable to other hosting providers.
1. Host your mail somewhere other than bluehost. (i.e. Google apps.)
2. Go to mx_entry button in cpanel. Make sure that you remove the local mx record already there. Set the radio button to "Remote" rather than local.
3. Go to E-mail Accounts in cpanel. Create an e-mail address that you would like to display in the FROM field of drupal e-mails.
4. Ask Bluehost to put your domain on the remote domains list.
5. Install the "PHPMailer" module on your drupal site.
http://drupal.org/project/phpmailer
6. (Read the instructions and don't forget to install the PHPMailer Library, a separate download.)
http://sourceforge.net/projects/phpmailer/files/phpmailer%20for%20php5_6/
7. In the PHPMail module settings, do this:
- Check "Use PHPMailer to send e-mails"
- Primary SMTP server: localhost
- SMTP Port 25
- Use secure protocol: NO
- Leave SMTP Authentication blank
- Under Advanced SMTP settings, Enter a FROM name
- Under Advanced SMTP settings, CHECK "Always set "Reply-To" address"
- Under Advanced SMTP settings, Don't check "Keep connection alive"
This makes drupal set the FROM field to be an address that Bluehost sees as legitimate (the one you created in step 3.) The reply-to is set to the address of the drupal user who sent the message using the contact form. Even thought the FROM field won't be accurate, when replying to the message, it will go to the right place.
Comment #17
montesq CreditAttribution: montesq commentedI completely agree with Oceria's remarks and I think we must not give the chance to modify the "from" field whereas the e-mails are technically *always* sent by the application.
In the patch provided as attached, I've changed the signature of drupal_mail() to replace $from by $reply.
Then, the e-mails are *always* sent by the e-mail defined at the site level and if a module provides the parameter $reply, this value will be set in the section 'Reply to' of the e-mail.
Comment #18
montesq CreditAttribution: montesq commentedComment #19
Ela CreditAttribution: Ela commentedPatch in #17 works for me. Thank you montesq
Just one question... Would it be possible to set the site name as well in the emails?
Comment #20
mattconnolly CreditAttribution: mattconnolly commentedI just came across this problem with the Contact module sending email from the user's email, which of course doesn't work through the sites SMTP server.
The patch in #17 resolves the issue perfectly for me, where I now see messages from the site, but have a reply address to the user's email.
I can't believe this was spotted way back in 2007 and is still there!
I'd like to see this fix backported to drupal 6 and 7. In the meantime, I'll keep this patch handy for sure!
Comment #22
Konstantin Boyandin CreditAttribution: Konstantin Boyandin commentedCOntact form should never use user-provided From: address directly.
That will make mail validation fail, for whatever techniques used (SPF, DKIM etc).
This will also raise chances to mark the Drupal-running site to be marked as spam origin.
I had to use simple 'hack' that moves the From: address to the message body, and uses site-wide address instead. It was the only simple way to make SMTP services such as Amazon SES process Drupal system messages.
Comment #23
markabur CreditAttribution: markabur commentedA related problem is if you have multiple recipients in the sitewide contact form, they all get used as the confirmation message's From address. This technically violates RFC2822 since you MUST add a Sender field in that case, which Drupal doesn't do.
http://www.ietf.org/rfc/rfc2822.txt
Comment #24
Rj-dupe-1 CreditAttribution: Rj-dupe-1 commentedJust for future searchers that don't want to have to apply a patch, we've whipped up a small D6 module that sends all contact form email from the website's email address with Reply-To set to the from address. It's still in sandbox at the moment but we are using in production sites with Amazon SES. Our D7 sites aren't running on AWS at the moment but I'm sure a D7 version will follow at some point.
Comment #25
ExTexan CreditAttribution: ExTexan commentedI had the same problem as @markabur (#23) and am also using Bluehost. I modified the contact_mail_page_submit() function as shown below (sorry I wasn't able to create a patch file in the proper format). My reasoning is that the "site_mail" system variable should be used (if present) as stated on the Site Configuration page for Email Address...
In the unlikely event that the "site_mail" system variable doesn't exist, it falls back to the first email address in the list of "recipients"...
Comment #26
tobedeleted CreditAttribution: tobedeleted commentedMany thanks to Rj for making a solution available. I've got to say that not having this issue sorted out in core is totally crazy. I'm sure I'm not alone in struggling to find a solution to this issue, which frankly should have been addressed years ago, and looks to have been largely ignored, thereby limiting the usefulness of Drupal to many people.
Comment #27
ExTexan CreditAttribution: ExTexan commentedI see this has been bumped (yet again) to 8.x by Dave Reid in comment #7 with the explanation being "feature requests are closed in D7 now".
I don't see this as a "feature request". I see it as a bug. And if security fixes and other bug fixes can be applied to core (even after it has been "officially released"), then this should be able to be fixed in D7.
What do you say, community?... agree?... disagree?
Comment #28
jenlamptonI agree, it's a bug.
All the emails that are generated by Drupal core's contact module on various Drupal sites of mine are now landing in my Gmail inbox with a giant warning on them stating:
It's bad behavior for Drupal to cause these kinds of warnings, and since the solution appears to be really straightforward, we should just do it :) That siad, it probably still needs to be fixed in D8 first, then backported to D7.
So... attached is a patch against D8, and a re-roll of the D7 patch ready to be backported to D7 (named do-not-test)
Comment #30
naxoc CreditAttribution: naxoc commentedA test had been written for From-headers in the meantime, so the tests were failing. Here is a D8 patch that fixes the tests.
I am not entirely sure wether the From: header should still include the site name like "Drupal site <someemail@example.com>"? I removed it for now in the test.
Comment #31
jenlamptonNice work, thanks :)
Looks like the D7 patch is still good, unless we want to backport the new tests, too?
Comment #32
Dries CreditAttribution: Dries commentedThis looks good. Do we need to update any code in Drupal (other than the tests) to reflect this API change?
A quick grep seems to suggest there may be a couple of code paths to look at:
Comment #33
Dries CreditAttribution: Dries commentedChanging status because of #32. Feel free to set back to RTBC if this is a non-issue. Thanks.
Comment #34
naxoc CreditAttribution: naxoc commentedI took a look at the instances where an explicit sender is used. They all (I found two) using sender correctly I would say.
Comment #35
Dries CreditAttribution: Dries commented@nanox: thanks for checking. Good to know we checked all callers. Unfortunately, it seems like the patch no longer applies and that a re-roll may be required. Asking for a re-test.
Comment #36
Dries CreditAttribution: Dries commented#30: 111702-30.patch queued for re-testing.
Comment #38
naxoc CreditAttribution: naxoc commentedHere is a reroll.
Comment #40
naxoc CreditAttribution: naxoc commentedFixed the failing test.
Comment #41
javier.drupal.2012 CreditAttribution: javier.drupal.2012 commentedHi, could someone tell me how can I get this issue solved for D7?
Thanks!
Comment #42
naxoc CreditAttribution: naxoc commented@javier.drupal.2012 the patch needs to be committed to D8 first and then backported to D7 so it will take a little while.
But you can reroll the patch and apply it to your D7 installation while you wait :)
Comment #43
javier.drupal.2012 CreditAttribution: javier.drupal.2012 commentedHi @naxoc, Sorry, but I don't know how to get the patch for 7.x version. I'm waiting for the patch for D7 but I don't know how long it would take....Can you let me know whether there is any way to know when is going to be the patch backported?
I really don't know how to reroll the patch to apply it in my d7 version
Thanks heaps
Javier
Comment #44
dcam CreditAttribution: dcam commented#40: 111702-40.patch queued for re-testing.
Comment #45
mvcfor anyone on D6 or D7 who has this problem and doesn't want to use this core patch on their site, the module contact reply-to offers the same functionality.
Comment #46
rfayCrazy that this isn't solved after this many years :-)
Comment #47
mvcsetting back to RTBC because a) i tested the patch and b) the only thing wrong earlier was a failing test.
Comment #48
penyaskito#40: 111702-40.patch queued for re-testing.
Comment #50
mvctrying to fix the test case
Comment #51
penyaskitoBack to RTBC, tests are green and look right.
Comment #52
YesCT CreditAttribution: YesCT commentedThis issue was RTBC and passing tests on July 1, the beginning of API freeze.
Comment #53
alexpottI don't think we need this assignment anymore... can just use $site_mail instead of $from
I think this
if
is unnecessary now as we should always have a $site_mailAlso I think we need to update the watchdog message in
drupal_mail
too. The current code is:Maybe to something like what is below since the from is going to be unchanging for 99 use-cases out of 100...
Although in the case of multiple domains...
Might be more helpful... but then again you should get the uri on the watchdog message anyway...
Comment #54
mvcthanks for the thorough review, alexpott.
good points. i've fixed those now.
on the grounds that more debugging information is generally better than less, i've used your second suggestion.
Comment #55
mvcactually, let me try that again: i wasn't printing a useful watchdog message in the case where no reply-to value had been set. now i'm printing the words "not set".
Comment #56
penyaskito55: drupal-mail-fixed-from-111702-55.patch queued for re-testing.
Comment #57
penyaskitoComment #58
penyaskitoTestbot response didn't arrive, but the patch at #55 does not apply anymore.
Comment #59
mvcre-roll of #55.
@penyaskito: thanks for your interest; i was beginning to feel like this issue has been abandoned :)
Comment #60
mvcoops, forgot to change the status...
Comment #61
penyaskitoIf I'm right, we are losing here the ability of putting a beautiful human-from, only allowing the address.
I will like that emails sender are shown in clients like "Example Website", not info@example.org.
This would cause a regression of #209672: Use site name in From: header for system e-mails. As far as I know using that won't affect deliverability.
Comment #62
mvc@penyaskito: oops, you're right. here's a new patch that doesn't regress #209672: Use site name in From: header for system e-mails. i had missed that, but in my defense, this issue is a year older :)
let me know if you see any other reason not to set this to RTBC.
Comment #63
mvcgah, forgot to hide old files...
Comment #66
penyaskitoThis line should not change now.
I would maintain an assert for $mail['from'].
Adding one to reply-to is super!
Thanks to your insistence!
Comment #67
penyaskitoFixed #66 comments and also changed a comment that wasn't applicable anymore.
Renamed variable for readibility.
Comment #69
penyaskitoLooks like a testbot problem, the test pass locally.
Comment #70
penyaskitoComment #71
penyaskito67: 111702-replyto-67.patch queued for re-testing.
Comment #72
vijaycs85Can't believe this issue still around (was checking for a project in 2010)... So this won't cover the case where we want to set custom from field. e.g. want to send from no-reply@site.com and reply-to: sales@site.com where site default is contact@site.com.
One more thought is, why can't we make a generic $headers arg with default overwrite something like:
Comment #73
Sumeet.Pareek CreditAttribution: Sumeet.Pareek commentedI was trying to manually test the patch in #67 on simplytest.me but the following error would show up during installation - Configure site. The service definition "field.info" does not exist.
May be it is the patch or some D8 code outside the patch causing this?
Comment #74
Sumeet.Pareek CreditAttribution: Sumeet.Pareek commentedI am completely new to doing anything with/in the core, so please correct me if I am thinking on the wrong lines, but doesn't the latest patch in #67 prevent the FROM address from being anything other than the SITE_EMAIL? Wouldn't that be a very bad idea? Because there are use cases where the FROM address needs to be different at different times (eg: different newsletter categories when using the simplenews module).
Shouldn't we be passing both FROM and REPLY-TO addresses to drupal_mail() and use SITE_EMAIL as the default or the fall back.
Comment #75
penyaskito@Sumeet.Pareek Welcome to core development :)
Can you test again with http://simplytest.me/project/drupal/8.x?patch[]=https://drupal.org/files...
Now its working for me so could be an environmental random failure.
Simplenews could still override the From address if needed, just in their hook_mail_alter. And any contrib module could provide the D6/D7 behavior. We are just providing better default behavior, not disallowing at all doing whatever you need.
@vijaycs: Not sure if we want to change that. As said, we can override if needed, but this IMHO should be the responsibility of a mail-related contrib/custom module.
The scope is just improving the deliverability of Drupal mails for a default installation of Drupal 8, taking into account newcomers + shared hosts.
This was RTBC on July 1st, and it is almost 7 years since the original report. Let's move forward and try to get this in, or postpone or won't fix if there are enough reasons.
Comment #76
vijaycs85@penyaskito makes sense to me... we should get this in and worry about further improvements as follow up... +1 to RTBC
Comment #77
mvcpenyaskito's patch looks right to me. thanks!
Comment #78
penyaskito67: 111702-replyto-67.patch queued for re-testing.
Comment #80
andypost67: 111702-replyto-67.patch queued for re-testing.
Comment #81
andypostHEAD was broken
Comment #82
alexpottIt looks like we're doing more than just adding the
reply-to
header and setting thefrom
header to site mail. We're also adding theErrors-To
header which is deprecated according to https://commons.apache.org/proper/commons-email/userguide.htmlI think we should not be setting this header - especially since there is zero discussion of why this should be added on this issue. OH IT'S A REGRESSION :) ... searching of existing issues reveals #1397270: Errors-To header makes email sending via Amazon AWS SES impossible, #1197376: Remove Errors-To header for Amazon SES, and #1062616: Failed integration with Amazon's Simple Email Service (SES) - which actually removed the
Errors-to
! It'd be nice if we could add an assertion to ensure that theErrors-to
header is not set when usingdrupal_mail()
There are two emails sent by this test I think that with the first email sent we need to assert on both the
Reply-to
andFrom
headers. On the second we need to assert that theFrom
header is set correctly and the theReply-to
is not set. Actually does this bring up an issue with the patch - what happens if the $reply value is NULL indrupal_mail()
?Comment #83
penyaskitoWow, thanks for catching that!
The attached patch fixes it and adds assertions for verifying the
Errors-To
is not set.Also added requested assertions for From and Reply-to.
Because I was there, also added assertion messages to those.
If
$reply
value is not given indrupal_mail()
, we don't add theReply-to
header, so the mail client will response back to the sender, who is going to actually be the site email.Comment #84
andypostGreat! finally ready
Comment #85
alexpottCommitted 8e92d3b and pushed to 8.x. Thanks!
Comment #86
kim.pepperSuggested change notice:
drupal_mail() now uses 'reply-to:' instead of 'from:' header
In the fight against spam more and more ISP's require mail send though the php function to originate from the main domain. The contact form however uses the senders email as "from:" address.
This has been changed so that the 'from:' header is set to the site-wide email address, and the senders email is set in a 'reply-to:' header.
Drupal 7:
Drupal 8:
Comment #87
penyaskitoYAY! Big thanks everyone involved in reviewing this!
@kim.pepper: Looks good to me, I just created https://drupal.org/node/2164905.
I only appended that you can still modify the From: by implementing hook_mail_alter().
@all: Is the first Change notice I ever create and I'm not really aware of the process, so please review and mark this issue as Fixed.
If I did anything wrong please tell me for next time.
Comment #88
kim.pepperComment #89
kim.pepperComment #90
ExTexan CreditAttribution: ExTexan commentedOne thing I thought of when posting this issue for the Subscriptions module...
If I am understanding this patch correctly, for modules that have an override email address to use as the "from" address, the system default address will be used for "reply-to", is that correct?
If so, it will not be possible to do what I'm attempting to do in Subscriptions, which is to have it use "no-reply@mysite.com" instead of the system default "from" address. Sure, the user will see that it came from "no-reply" and probably won't even attempt to reply, but I'm getting out-of-office autoreplies - which (it seems) would go to the system default as a result of this patch.
Comment #91
mvcExTexan wrote:
no; with this the system default will always be used as the "from" address, and the override email address (if set) will be used as the "reply-to" address. (this is necessary so that SPF & other anti-spam measures which are becoming more common will still work.) so what you're planning to do in the subscriptions module will still work, since the out-of-office replies will go to no-reply@mysite.com or wherever you want.
Comment #93
jenlamptonLooks like this issue got closed accidentally... but some of us are still running D7 sites, so re-opening.
The patch from https://www.drupal.org/node/111702#comment-7000314 seems to still apply to 7.31, so re-uploading that here to kick off testing again.
Comment #94
jenlamptonlooks like the issue metadata didn't save, trying again.
Comment #96
tregismoreira CreditAttribution: tregismoreira commentedI'm using this simple patch and its works fine!
Comment #97
jenlamptongo test bot.
Comment #99
Les LimUpdated tests to expect "reply-to" instead of "from".
Comment #100
mikeytown2 CreditAttribution: mikeytown2 commentedBeen using patch #99 for some time in production. +1 from me.
Comment #101
jenlamptonRTBC from me as well.
Comment #104
Les LimRandom testbot fail.
Comment #106
kim.pepperComment #108
vijaycs85seems like testbot wasn't stable. It's green and cleanly applies to current HEAD
Comment #111
Les LimComment #112
Damien Tournoud CreditAttribution: Damien Tournoud commentedI don't understand the rationale behind this patch, which looks like a regression to me.
The original code looks perfectly correct:
Sender
andReturn-Path
) is set to the site address;From
) is set to the sender address.SPF / DKIM and friends act on the envelope, not on the message itself.
Could we get more details from people affected by this on what they are seeing that is not what they would expect?
For example, I cannot reproduce Jen issue described in #28. Emails sent to me via Drupal.org's contact form have all the headers that I expect and pass all the SPF checks in Gmail.
(Both the change notice and the original issue seem to misunderstand basic concepts about email.)
Comment #113
gorlov CreditAttribution: gorlov commentedIn response to #112, I am having a significant issue with this related to people using the contact form and who use a yahoo.com mail account. See the "bounce" message I am getting from gmail. I have full SPF and DKIM correctly configured on my mail server. Mail is being signed, but GMAIL sees the FROM address coming from YAHOO and marks it as spam.
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
@gmail.com
SMTP error from remote mail server after end of data:
host gmail-smtp-in.l.google.com [64.233.177.27]:
550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to domain's
550-5.7.1 DMARC policy. Please contact administrator of yahoo.com domain if
550-5.7.1 this was a legitimate mail. Please visit
550-5.7.1 http://support.google.com/mail/answer/2451690 to learn about DMARC
550 5.7.1 initiative. s43si6580166yho.201 - gsmtp
@gmail.com
SMTP error from remote mail server after end of data:
host gmail-smtp-in.l.google.com [64.233.177.27]:
550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to domain's
550-5.7.1 DMARC policy. Please contact administrator of yahoo.com domain if
550-5.7.1 this was a legitimate mail. Please visit
550-5.7.1 http://support.google.com/mail/answer/2451690 to learn about DMARC
550 5.7.1 initiative. s43si6580166yho.201 - gsmtp
------ This is a copy of the message, including all the headers. ------
Return-path: .com>
Received: from www-data by with local (Exim 4.80)
(envelope-from .com>)
id 1YPBV9-0005bc-5e; Sat, 21 Feb 2015 09:57:51 -0500
To: @gmail.com,@gmail.com
Subject: [other requests] Delete
X-PHP-Originating-Script: 33:system.mail.inc
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed; delsp=yes
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: admin@.com
From: @yahoo.com
Message-Id:
Date: Sat, 21 Feb 2015 09:57:51 -0500
=======================================
I have replace some real names with FQDN, my-mail, my2-mail, , site-domain-name.
Note that gmail rejects the mail because of Yahoo's DMARC policy that is strict, and doesn't let yahoo mail come from my server. And yes, it is reading the FROM addr, not the RETURN path or the SENDER, all of which are in sync with the site's DKIM and SPF records.
The YAHOO address is the user using the contact form. the two GMAIL accounts are the intended recipients of the contact form.
(using fully up-to-date DRUPAL 7.34, Debian Wheezy, Exim4, php54-fpm, apache2.2 on a large KVM VPS). I am a very experienced linux/BSD sysadmin and have been using drupal since 5 for many custom sites.
Hope that helps.
George
Comment #114
Damien Tournoud CreditAttribution: Damien Tournoud commented@gorlov: Ah, yes, DMARC broke email. And it is certainly a good reason for the change, but it's really the real reason is mentioned in this issue.
Let's move forward with this for 7.x.
Comment #115
mvcI've said this before in this issue queue, but for anyone who's new here, it's possible to work around this problem with the contrib module contact_reply_to until this change gets pushed to core.
Comment #118
dcam CreditAttribution: dcam commentedComment #119
Rajab Natshah CreditAttribution: Rajab Natshah commentedIs this issue fixed and pushed to Drupal 7.34 ?
I can see some of the code in the new Drupal mail.inc code. but not all changes.
Rewarded work :)
Comment #120
mvc@RajabNatshah this issue will be marked fixed when it is committed to the 7.x-dev branch, and will appear in the next stable release of Drupal 7 which comes out after that. You need to apply the patch manually until then.
Comment #121
Rajab Natshah CreditAttribution: Rajab Natshah commentedThanks Matt .
I do have the patch in a project, but I'm updating Drupal for that project. I will apply the patch then.
Rewarded time :)
Comment #122
David_Rothstein CreditAttribution: David_Rothstein commentedSo off the bat, this is missing some things from the Drupal 8 patch (the watchdog message and some tests). Also, the documentation of the new reply-to parameters is a little lacking.
But a bigger problem is that this is a very heavy-handed fix, especially for a stable release. As others pointed out above (e.g. #72 and #74) this will break various use cases. For example if your site mail is
webmaster@example.com
but you want to send something fromJudy Smith <president@example.com>
this will prevent that despite the domain being the same, and the display name will be removed from the email too. Plus, the Contact module becomes less user-friendly since without an indication somewhere in the "From" field of who actually sent it, it's hard for the recipient to tell from scanning their inbox - they'd basically have to open up each message. (I'd think we at least want the From to be something likesender@theirdomain.example.org via Site Name <webmaster@yoursite.example.com>
to avoid that problem?)All of this can be dealt with in hook_mail_alter(), but it's odd to require doing it there, and regardless I don't think we should be breaking existing code for Drupal 7.
The Webform module fixed this recently in a way that deals nicely with all the above considerations (see #2236237: Use Reply-To instead of From e-mail headers (Google/Yahoo/etc anti-spoofing policy marks e-mails as spam)) and this patch would break their solution. Shouldn't we try to implement something similar to what Webform did in Drupal core, and not change the API?
Comment #123
David_Rothstein CreditAttribution: David_Rothstein commentedLike Damien I am also a little confused about the history of the issue; this was committed to Drupal 8 in late 2013 but the serious problems with this didn't start until April 2014, I believe (when Yahoo changed their policy).
A lot of the earlier reports in this issue, while it's true the patch here would help with those kinds of situations, I think in most cases the root cause would be solved by adding an SPF record for your site to indicate which servers are authorized to send mail for your domain. For example, something like #28, where Gmail marks the message as "This message may not have been sent by:
someone@notmydomain.com
", what Gmail would like to do in that instance, I believe, is mark it as "someone@notmydomain.com
via yoursite.com" (that's what it does for mail sent via the drupal.org contact form). But it will only do that if it can verify via an SPF record that the mail was actually sent by yoursite.com.Comment #124
drupdan3 CreditAttribution: drupdan3 as a volunteer and commentedSo what should one use to fix this unbelievably backwards behavior of Drupal? The Webform module? The https://www.drupal.org/project/contact_reply_to module? I keep reading these bug reports, some of them are marked as "Closed", but there seems to be no definitive write-up on what to do.
Comment #125
Rajab Natshah CreditAttribution: Rajab Natshah commentedseems that issue #111702 still not pushed to the Drupal 3.37 or the dev .. Still we need to apply the patch
I will have a look at the contact_reply_to mdoule . https://www.drupal.org/project/contact_reply_to
I will keep following this issue until we have it in Drupal or have the right module to work around.
Comment #126
jenlamptonAttached is a re-work of the patch that was used for Webform. I personally object to providing users with a checkbox for "Do you want the bug fixed?" even if we default the answer to "Yes". However, it is late in the D7 cycle and I get that we don't want to break contrib modules that affect mail, so let's give this a go.
Comment #127
prateek_drupal1
down vote
There is one e-mail address that is used as From-address for all mails sent automatically by the system. By default, this is set to the same as the site administrator email. To change it, do the following:
Navigate to Administration » Configuration » System » Site Information.
Locate the field "E-mail address". Change it to whatever you want it to be.
Press Save configuration.
Note that will also be used as the From-adress for password resets and other generated mail.
If you require fine-tuning beyond this (to distinguish between different types of auto-generated email messages), you need to write a custom module. In that case, you may use hook_mail_alter to alter the From:-address, based upon some criteria (e.g. the subject field.
For avoidance of misunderstanding: The three steps listed above will not change the site administrator email id. To inspect and change the site administrator email id, click on the "My account" link while logged in as site admin, and then press the Edit tab to see it. The site administrator email id is by default the same email id as the From-address used for email sent by the site itself - but they do not need to be the same.
Summary is if you want to change the from and reply to address you can either change it from the above 3 steps mentioned which will be changed for all email which are going from system. But if you want to do it for some specific emails then you can do it from hook_mail_alter by writing own module.In this case you will have more power in terms of customizing each and every mail going from the system. Contact me if you need help in this or any calrifications.
Comment #128
prateek_drupal1
down vote
There is one e-mail address that is used as From-address for all mails sent automatically by the system. By default, this is set to the same as the site administrator email. To change it, do the following:
Navigate to Administration » Configuration » System » Site Information.
Locate the field "E-mail address". Change it to whatever you want it to be.
Press Save configuration.
Note that will also be used as the From-adress for password resets and other generated mail.
If you require fine-tuning beyond this (to distinguish between different types of auto-generated email messages), you need to write a custom module. In that case, you may use hook_mail_alter to alter the From:-address, based upon some criteria (e.g. the subject field.
For avoidance of misunderstanding: The three steps listed above will not change the site administrator email id. To inspect and change the site administrator email id, click on the "My account" link while logged in as site admin, and then press the Edit tab to see it. The site administrator email id is by default the same email id as the From-address used for email sent by the site itself - but they do not need to be the same.
Summary is if you want to change the from and reply to address you can either change it from the above 3 steps mentioned which will be changed for all email which are going from system. But if you want to do it for some specific emails then you can do it from hook_mail_alter by writing own module.In this case you will have more power in terms of customizing each and every mail going from the system. Contact me if you need help in this or any clarifications.
Comment #131
fonant CreditAttribution: fonant at Fonant Ltd commentedprateek_drupal: every Drupal site needs this if they are using the core Contact module, or any other module that sets the "From" address to be that supplied by a site visitor.
The reason is DMARC, which, against all RFC advice, looks at the From address for an email. So if you want Drupal to send emails where a reply will go to the visitor's email, you have to set the From address to belong to the site, and the Reply-To address to be the visitor's. This is what this patch does.
Without this patch, or a contrib module that does the same thing, mail from Drupal can get blocked by large email providers like Yahoo! that use DMARC.
For this reason Drupal 8 has this behaviour in core.
Comment #132
kim.pepperI created a Drupal 7 contrib project that simply swaps
From
email with site email, and uses senders email asReply-To
. https://www.drupal.org/project/reply_toComment #133
jenlamptonPatch in #126 still applies cleanly.
Comment #134
mvcthx, jenlampton. still works for me.
Comment #135
David_Rothstein CreditAttribution: David_Rothstein commentedI tested the patch and it didn't work for me:
"someone@example.com via Site Name <siteaddress@site.example.org>"
) never made it through in the email. I think that's because the code sets$message['from']
to the user-friendly value, but doesn't set$message['headers']['From']
to the same thing?I would also lean towards not having the checkbox, personally. One of the reasons for borrowing the Webform solution is that it bends over backwards to only change the behavior when necessary and to change it in a user-friendly way when it does... and if someone really doesn't want the new behavior for some reason I'm pretty sure they can still change it in hook_mail_alter().
Also, based on a quick code review:
Would be really good to get this issue fixed though! (And then there could be a followup issue to discuss forward-porting this solution to Drupal 8.)
Comment #136
bigfatguy CreditAttribution: bigfatguy commentedIn drupal-7.51 this is still not fixed? Why?
Comment #137
DamienMcKenna@bigfatguy: Please read the documentation on what the different issue status values mean: https://www.drupal.org/node/156119
Comment #140
wturrell CreditAttribution: wturrell commentedIs the intention to still back-port this to D7? Reason I ask: I became aware of the issue from using the contact forms on drupal.org to write to other people, and ticking the 'Send myself a copy' box. I adjusted my SPF record but obviously that doesn't help with DMARC, so I wonder if there's a significant amount of community mail being lost – to Gmail accounts for example – because people may not check spam folders.
Comment #141
AdamPS CreditAttribution: AdamPS at AlbanyWeb commentedNote that the comments link to two different modules that propose a solution to this problem whilst waiting for the D7 commit.
Comment #142
jenlamptonHere's a re-work of the patch in #126 without the setting since @David_Rothstein and I seem to be in agreement on that. The code in this new approach is more similar to the work we are doing over in https://github.com/backdrop/backdrop/pull/1823/files for Backdrop. Both still need tests.
Comment #143
scott.browne CreditAttribution: scott.browne commentedNoticing this recent issue popping up all over hundreds of websites I run. Core contact form is no longer usable and I have been porting them all to webform. Even webform you must use a valid email on that domain as the sender. Any solutions down the road for this?
Thanks
Comment #144
rajeevkSetting NR for test queue.
Comment #145
mlangenfeld CreditAttribution: mlangenfeld as a volunteer commentedPatch #142 works fine with Yahoo, but not with Gmail, since the 'My Name ' format is interpreted as multimple adresses. Here's the mailer-daemon message: gmail-smtp-in.l.google.com SMTP
Comment #146
David_Rothstein CreditAttribution: David_Rothstein commented@mlangenfeld you can put <code> tags around things like that so they don't get stripped out of your comment. I am copying it here now with that done so others can see the mailer-daemon message you're referring to:
This is very curious and it is not obvious why it would occur. Could you provide some more details (such as what the actual "From" header looked like in this message)?
Looking at the patch it is not obvious to me what would cause this, although I am curious what would happen if e.g. the site name has a
"
in it, since it looks like that could cause trouble with the From header.Comment #147
mlangenfeld CreditAttribution: mlangenfeld as a volunteer commented@David_Rothstein I'm sorry for the tag, this is my first post in the Drupal forum. The From line reads, as it should:
"[email of sender] via [Sitename]" <[standard site email]>
But the string has been exploded after 45 characters, so it reads:
<"[email of sender] via [part of Sitename]>, <[rest of Sitename]" <[standard site email]>>
which Gmail interpretes as 2 adresses. I have to say that our sitename is quite long (32 characters), but there are no
"
in it.Update: it seems the problem is already known in Drupal 8: #2863601: GMail rejects mail because long 'from' field is split by Drupal 8
Comment #148
David_Rothstein CreditAttribution: David_Rothstein as a volunteer commentedThanks - that is definitely helpful research! Sounds like the patch needs work then. Also, I confirmed that a site name with a single
"
in it breaks things in a similar way (not specific to Gmail) so it needs work for that too, although it's possible that's just a different (and less likely to encounter in reality) version of the same problem.The patch looks pretty good otherwise, but it still needs to pass the correct language into t(), and I'm adding the "Needs tests" tag since a couple people above mentioned that. Also there are a couple minor misspellings ("specefied" and "reformated").
Comment #149
mlangenfeld CreditAttribution: mlangenfeld as a volunteer commentedHere's my rework of the patch. I just added a mime encoding for
$headers['From']
. This seems to fix the issue as discussed in #2863601: GMail rejects mail because long 'from' field is split by Drupal 8 and proposed in #2717965 Site name is not UTF-8 encoded in email headers . It works fine on my production site.Comment #150
vinothkannan CreditAttribution: vinothkannan commentedComment #151
philsward CreditAttribution: philsward commentedInteresting this issue has been around for a decade "and" drupal.org doesn't even have DKIM or DMARC setup. At all.
Anything coming from security-news drupal.org is being sent from IP's that aren't even in the SPF records.
I recently got on the bandwagon for becoming DMARC compliant and within a week, was blown away at the number of emails being sent on behalf of my domains. Legit were on the order of maybe 15 - 25 per day on some low end domains but illegitimate? In the 100's, issued on my domains behalf from all over the world... I only have 2 outgoing email services, the webserver and G Suite and they're both US based servers. In otherwords, there's no reason I should have email being sent from servers in India, or China.
Here's an idea of where those security-news drupal.org emails are going:
Not only is the SPF failing, but neither DKIM nor DMARC are in place for Drupal.org.
Looking at the SPF record for drupal.org, it's only pointing to the mx records (smtp1-4.osuosl.org) and servers.mcsv.net, none of which contain the block for the server generating security-news on IP 140.211.10.49.
Considering Drupal.org isn't even compliant, I guess it shouldn't come as any surprise this issue for core is taking... a while?
Comment #152
jenlamptonPatch in #149 is also working for me. Thanks @mlangenfeld!
Comment #153
joseph.olstad@philsward makes some very good points. @jenlampton is on the ball too.
@jenlampton, can you please confirm if this patch still works even if you aren't supporting DMARC? I looked at the patch and it looks ok to me but I haven't tested with and without DMARC.
Comment #154
nnewton CreditAttribution: nnewton commented@philsward, I wanted to respond as I am re-taking over the d.o. infra maintenance currently and working through a backlog. DMARC/DKIM is on the agenda, but we currently use relays that don't support that (We will either be vacating them or supporting their upgrade). The SPF fail is an entirely different issue likely related to a recent migration of the lists server. I will be looking into and fixing that today or tomorrow morning.
-N
Comment #155
philsward CreditAttribution: philsward commented@nnewton thanks for the heads up. Super important stuff and at the time of writing my comment, I wasn't sure where to go to reach out to the folks necessary to get it done. My hope was to reach some attention from someone in this thread that might be privy to talk with the right folks. I later found the contact form for the "Drupal Association" which is where I should have gone in the first place, but wasn't aware of it.
Glad you're looking into it! I was floored at how "important" those items are for proper email setup when I went through the journey of setting it up on my end. I was shocked to see Drupal related emails going to spam for this very reason.
Sorry everyone for distracting from the original topic :-/ I was merely trying to be a squeaky wheel.
---
Glad to see this thread is moving forward though! It's been a long time coming!
Comment #156
ultrabob CreditAttribution: ultrabob commentedComment #157
benjifisherComment #158
th_tushar CreditAttribution: th_tushar as a volunteer commentedThe patch #149 looks good and fixes the issue.
Comment #159
kim.pepperPlease see https://www.drupal.org/core/issue-priority and ensure you provide valid reasoning of 'critical' according to that criteria before changing issue priority.
Comment #160
mvc@kim.pepper: I agree that this shouldn't have been bumped to "critical" but it's been flagged as "major" since @alexpott set it that way in 2013 and no-one who's looked at it since then has objected. Let's leave it there.
Comment #161
webservant316 CreditAttribution: webservant316 commentedWill this patch work with the 'smtp' module? I am not certain if 'smtp' bypasses all this logic or sits on top of it.
Corresponding ticket created at 'smtp' https://www.drupal.org/project/smtp/issues/2941007.
ANSWERING MY OWN QUESTION:
Yes this patch works great because the needed logic occurs before the 'smtp' hook or any hooks are applied.
Thanks for a great patch!
RTBC!
Comment #162
jenlamptonPatch in #149 applies cleanly to 7.58.
Comment #163
mike.davis CreditAttribution: mike.davis at Deeson commentedPatch #149 is working really well for me on 7.57 and now 7.58.
What is stopping this from being committed now?
Comment #164
Kisugi Ai CreditAttribution: Kisugi Ai commentedNot sure if its right place here
but i have a problem
since i have the patch 149 in
it will transmitted the user email in the Personal contact form
and this should not happend because
Comment #165
jenlamptonAh, that sounds like it needs work then. (changing status)
For those who are not using the per-user contact form, the patch in #149 still applies to 7.59.
Comment #166
pgrandeg CreditAttribution: pgrandeg as a volunteer commentedPatch #149 doesn't work for me in 7.59 and I solved using:
I hope it helps!
Comment #167
joseph.olstadBumping to 7.70. This didn't make it into 7.60.
Comment #168
ExTexan CreditAttribution: ExTexan commentedShouldn't that be 7.61?
Comment #169
webservant316 CreditAttribution: webservant316 commentedyes 7.61 please.
Comment #170
joseph.olstadThis is classified as a bug so yes it could be in the 7.61 release. The core maintainers will have the final say.
Comment #171
wturrell CreditAttribution: wturrell commentedI've just reread the recent issue history: besides the points in #164 and #166, we still haven't written any new tests (#135), and it'd probably also help to get an update from someone on the d.o infrastructure team.
Comment #172
tobybellwood CreditAttribution: tobybellwood at govCMS (Australian Government Department of Finance) for govCMS (Australian Government Department of Finance) commentedJust a quick word to the wise - we've been using #99 in production for a couple of years with no worries.
But recently a change in backend email providers has highlighted that this patch can create multiple reply-to headers (which AWS SES does not like...), when used in conjunction with the Webform versions (> 7.x-4.3) that contain the "webform_email_replyto" variable (discussion in https://www.drupal.org/node/1462192). Setting this variable to 0 is an effective workaround for now.
We'll test the same scenarios against the most recent patch and report back.
Comment #173
joseph.olstadComment #174
jenlamptonPatch in #149 still applies cleanly to 7.63.
Comment #175
klonosWould it help to say that this has been production-tested in Backdrop CMS since version 1.1.0 (2015?): https://github.com/backdrop/backdrop-issues/issues/305
Comment #176
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commented@klonos thanks for testimony.
this is one of the last patches should be committed in the Drupal 7 release before EOL
Comment #177
joseph.olstadWe need to backport the test updates committed 6 years ago to D8
git show 8e92d3b1f16d
see attachment
Comment #178
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedUpdated with a check the 'From/Reply_to header' in 'mail.test' - backported from D8
Comment #179
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedPlease check attached interdiff file
Comment #180
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedHere is a test, and the test-only patch is the interdiff.
Comment #182
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedTests that email headers for contact message are correctly set.
Comment #183
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedComment #184
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedTwo new checks are added in #183 based on #149 regarding test
backported from D8:
- A new check for 'From/Reply_to header' in 'mail.test'
- A new check for personal contact email headers in 'contact.test'
Please check interdiff file for details.
Comment #185
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedComment #186
imclean CreditAttribution: imclean at Digital Ink commentedI'm coming in pretty late here, but why is Drupal setting the
Return-Path
header? The recipient's mail server sets this to the value of the envelope sender. Drupal can influence what gets set in theMAIL FROM:
command but it probably shouldn't set the header directly.Comment #187
Kisugi Ai CreditAttribution: Kisugi Ai commentedhttps://dmarc.org/
its the keyword
some Mail provider use this.
and drupal use the email adress (Mail From) from the registered user to send emails to staff or anything else.
if Drupal setst the from mail to any addres the mail-demon will not send the mail cause the adress ist unknown.
Comment #188
imclean CreditAttribution: imclean at Digital Ink commentedWhen Drupal sends email via SMTP it sends the SMTP command
MAIL FROM: email@domain.com
. The SMTP server can then set theReturn-Path
header. If you set this header directly the SMTP server may at best ignore it and add its own or simply remove it.It should be possible to configure the sender (as opposed to
From
address) within Drupal but not theReturn-Path
header.Comment #189
fonant CreditAttribution: fonant at Fonant Ltd commented@imclean - you are confusing the Return-path: header (which is also known as the "envelope sender") with the normal Reply-To: header which can be set by the sender of the message.
This patch sets the Reply-To: header to be the email address any replies should be sent to, and the From: header to the email address configured for the website. This allows mail to pass DMARC checks which would otherwise fail if the From: address domain implements DMARC and the web server is not authorised by DMARC to send mail for that domain.
Comment #190
imclean CreditAttribution: imclean at Digital Ink commented@fonant I'm not confusing them, although I've probably brought up the problem with Drupal's handling of the "Return-Path" header in the wrong issue. This is the most thorough discussion I can find of mail headers but I probably should've started another issue.
On the topic at hand, as has been mentioned DMARC seems to have caused some problems. It probably should consider the "Return-Path" header or at least "Sender" or "Reply-To" instead of "From".
Comment #191
arvana CreditAttribution: arvana as a volunteer commentedTo pass DMARC on my server, I found that I had to set the message 'From' to an email address matching the originating domain, as well as ALL of the relevant headers:
And then set the
$headers['Reply-To']
to the original sender's email address.I realize that
Return-Path
is supposed to be set by the receiving server, however it seems that it is used by DMARC as well, at least in some configurations. The above is what worked for me.Comment #192
PROMESI got a new challenge: a malformed From header when using non-asci characters (like ä, é and so on) in the site_name:
SMTP error from remote mail server after end of data: 554 5.2.0 MXIN600 Message is not RFC 5322 compliant: 'From' header is missing or malformed ;id=YTdni2kWspxBdYTdnigqw5;sid=YTdni2kWspxBd;mta=mx7.tb;d=20191123;
My working solution for these common European characters is transliterating the site_name before mime_header_encode(). The code starts with 2 existing lines and end with 1 existing line:
I don't know if this will work as wel for other non-asci chars.
Comment #193
longwave@PROMES I think your issue is #1419874: "From" mail header incorrectly encoded if (a) has name and e-mail address and (b) name contains non-ASCII characters
Comment #194
PROMES@longwave: Thanks. I was following / using this issue for so long time I assumed it fits in this issue.
I will move my reply to issues/1419874.
Comment #195
webservant316 CreditAttribution: webservant316 commentedreapplying to Drupal 7.69.
Comment #196
ressa CreditAttribution: ressa at Ardea commentedUpdating target, so the issue is not forgotten.
Comment #197
PROMESThe current latest patch works a expected. But in non-English countries we have sometimes sitenames with special characters.
If module transliteration is installed I transliterate the sitename before this name is assembled in $from_reformatted.
Currently my Netbeans program refuses to make pach files so I changed the attached patchfile manually with my changes:
if ($matches) {
+ $sitename = variable_get('site_name', 'Drupal');
+ if (function_exists('transliteration_get')) {
+ $sitename = transliteration_get($sitename, '');
+ }
$from_reformatted = t('"!name via !site_name" ', array(
'!name' => empty($matches[1]) ? $matches[2] : $matches[1],
- '!site_name' => variable_get('site_name', 'Drupal'),
+ '!site_name' => $sitename,
'!site_mail' => $default_from,
));
Comment #198
Kisugi Ai CreditAttribution: Kisugi Ai commentedHello,
got still the issue with emails, see
https://www.drupal.org/project/drupal/issues/111702#comment-12567413
at the moment it's not DSGVO conform so i have to disable the contact forms
if a user sends a message from the users contact form so its the email address in reply-to from the user who sends the message and thats is wrong. in this case should the "no-reply" address used or remove reply-to adress.
Comment #199
mcdruidIs this still an issue now that https://www.drupal.org/node/3185877 is in D7?
Comment #200
Kisugi Ai CreditAttribution: Kisugi Ai commentedits irrelevant if sends the seite name or site email the reply-to field has the userser email who has send the contact to an other user.
btw. i cant test it without tis patch cause my email server uses DMARC
the pics shows the headers
Sender: info@japanisch-grund-und-intensivkurs.de
From: "------@outlook.de via Japanisch Grund- und Intensivkurs"
Reply-To: -----@outlook.de
should
Sender: info@japanisch-grund-und-intensivkurs.de
From: "<username> via Japanisch Grund- und Intensivkurs" <info@japanisch-grund-und-intensivkurs.de>
Reply-To: no-reply@japanisch-grund-und-intensivkurs.de
Comment #201
imclean CreditAttribution: imclean at Digital Ink commentedThe
From
header should contain the site email address. e.g.no-reply@sitedomain.com
. This is what DMARC uses.The
Reply-To
header is the email address of the user who filled out the form so they receive any replies.Comment #202
Kisugi Ai CreditAttribution: Kisugi Ai commentedokay DMARC make it so, but
Drupal self say
an this does not work
and is not DSGVO konform
caus the Reply-to has the email adress from the user who fillet out the form.
this is okay if you use the feedback or global contactform but user to user it is not right
Comment #203
imclean CreditAttribution: imclean at Digital Ink commentedI see. I don't use the personal contact form. How is that feature supposed to work if email is involved? You'd need to generate another email address for each user and also be set up to receive email.
Comment #204
Kisugi Ai CreditAttribution: Kisugi Ai commentedWell i have made a new pach includet this stuff
maybe you could review it
and sorry I am not firm in the test stuff
and I am not sure waht this match thing do may be we do not need it an can reformat the from thing with out
Comment #205
webservant316 CreditAttribution: webservant316 commentedThis patch
may need toMUST be updated for Drupal 7.77. Since the patch does not cleanly apply to D7.77 I thought maybe I no longer needed the patch. However, the reply fix from this patch is essential in my use case. My website was failing at emails all over the place.Hoping someone can update the patch for Drupal 7.77.
My solution for now, even though I am upgraded to Drupal 7.77 I still use the Drupal 7.76 version of /includes/mail.inc with this patch applied.
Comment #206
TR CreditAttribution: TR commented@webservant316: WHICH patch are you referring to?
This issue needs some serious cleanup, as there are still 13 patches listed in the issue summary, and with >200 comments it's essential that you be precise when you discuss things, to prevent misunderstanding.
As far as I can tell, the "best" patch we have so far is from @pandaski in #183, but that patch does not yet address the valid concerns raised by @Kisugi Ai in #200, #202, and #204.
The most recent patch, in #204, does not have an interdiff.txt so it is difficult to see what has been changed since the community-reviewed patch in #183. The concerns raised by @PROMES in #197 are not relevant to this issue - non-ASCII characters in an email header are dealt with through RFC 2822 encoding of the header. Transliteration is never the solution. If RFC 2822 encoding is not being done properly by this patch then that needs to be addressed. It wouldn't hurt to change the test case to use non-ASCII characters in the site name just to ensure it does work properly. (It should NOT be the responsibility of this test case to ensure mime_header_encode() and the new drupal_mail_format_display_name() work properly - we just want to ensure that we're USING these when needed.)
I retested #183 against core Drupal HEAD (see https://www.drupal.org/pift-ci-job/1936627) and this confirms that it no longer applies to the current version of core (because of #3098058: [D7] Use site name in From: header for system e-mails) and needs to be re-rolled.
Comment #207
webservant316 CreditAttribution: webservant316 commentedI have manually applied patch #204 to /includes/mail.inc successfully through Drupal 7.76. However, the update to Drupal 7.77 severely complicated even a manual application of any of the patches in this post. Sorry, for the vague answer. Point is, we agree that patch #183 or any of the patches above need to be refactored for Drupal 7.77.
Comment #208
Kisugi Ai CreditAttribution: Kisugi Ai commentedwell @wenservant316 the pach which i have made is from drupal 7.77 its includes all 7.77 stuff and the new stuff of hidden mails adress betwen contacts
@TR
i am not sure if its enoth
but here the interdiff
Comment #209
webservant316 CreditAttribution: webservant316 commented@Kisugi Ai can you post your patch here?
Comment #210
Kisugi Ai CreditAttribution: Kisugi Ai commentedAhm comment-13928917
[111702]-[204].patch
Comment #211
TR CreditAttribution: TR commentedThere are multiple problems with #204. Specifically:
While it's easy enough to re-roll #183 against 7.77, I'm not going to do that here because it seems the patch needs more than a re-roll - it perhaps needs to use the new drupal_mail_format_display_name() instead of a custom regex, and since this is the heart of the patch it needs some thought and time to re-do the patch properly. I don't have time for that at the moment.
I think any new patch ought to start with #183 and fix that for D7.77 and for the problems mentioned in #200, #202, and #204. There should also be a new test to validate the changes made for #200, #202, and #204.
Comment #213
Kisugi Ai CreditAttribution: Kisugi Ai commented@TR
Well we can remove the square brackets.
and I wrote that's I not firm in the test stuff
I am not a programmer like you, I am a hobbyist
i have no idea what or how the test codes stuff works
Comment #214
TR CreditAttribution: TR commented@pandaski You can't use short array syntax in D7. D7 needs to run on PHP 5.3.
Can you post an interdiff between your MR and a previous patch so we can see what you've changed?
Comment #215
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commented@TR thanks, I will work more in local environment then commit it later for review.
Comment #216
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedHi,
This is a re-roll #183 against 7.77
Please check interdiff_183_216.txt for the changes
It is ready for a review now, thanks folks
Comment #217
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedComment #218
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commentedHere is a quick brief of the changes before and after this patch #216 using site wide 'contact' - "Website feedback" as an example:
Set "site_mail" to "noreply@drupal-111702.local" - this is a good practise to protect the service email, and you can always have forms or contact page so that ppl can reach to your preferred communication channels.
Set "test1@example.com" as the test user's email address who submit the contact form
Before:
A copy to the test user (The test email address is test1@example.com):
A copy to the site manger:
After:
A copy to the test user (The test email address is test1@example.com):
A copy to the site manger:
Comment #219
webservant316 CreditAttribution: webservant316 commentedThis line references $message['from']watchdog('mail', 'Error sending e-mail (from %from to %to).', array('%from' => $message['from'], '%to' => $message['to']), WATCHDOG_ERROR);
However, $message['from'] is only conditionally assigned above.Scratch that.
Comment #220
webservant316 CreditAttribution: webservant316 commented#216 works for me.
Comment #221
webservant316 CreditAttribution: webservant316 commentedThe #216 patch should be changed to be more explicit on line 139 and 165...
'from' => !empty($from) ? $from : $default_from,
if (!empty($from) && $from != $default_from) {
Comment #222
webservant316 CreditAttribution: webservant316 commentedI am fighting with random failed emails ever since the upgrade to Drupal 7.77 and even still after applying patch #216. So I added a check point to monitor the value of $message['from'] and somehow it is unassigned at the watchdog error report. How is that even possible that it would be unassigned?
Doing further testing now and wondering if this line
$message['from'] = $headers['From'] = drupal_mail_format_display_name($display_name) . ' <' . $default_from. '>';
need to be changed to
Perhaps the $message['from'] format disallows the display name format.
Comment #223
wylbur CreditAttribution: wylbur as a volunteer commentedAdding new Drupal Target Release.
Comment #224
ressa CreditAttribution: ressa at Ardea commentedThanks @wylbur, I have added this issue to the meta issue #3192080: [meta] Priorities for 2021-04-07 release of Drupal 7 where the next Drupal 7 release will be coordinated from. Remaining issues will be transferred to next planned release (2021-06-02), so there shouldn't be any need for updating "drupal 7.xx targets" in individual issues:
From #3179845: [meta] Priorities for 2020-12-02 bugfix release of Drupal 7.76 / 7.77.
Comment #225
RoSk0Replying to #222: I don't think we should do what is suggested. Looking at latest RFC:
- means the values is a mailbox-list.
- list contains one or more mailbox
In short, display name can be there in the From header.
#216 looks good.
Comment #226
argiepiano CreditAttribution: argiepiano as a volunteer commentedPatch #216 WITH THE MODIFICATIONS outlined in #222 works for me. Otherwise it doesn't.
Comment #227
imclean CreditAttribution: imclean at Digital Ink commented#225, the question is, what else could
$message['from']
be used for? If it's the envelope sender (with SMTP) or -f using a local mail service then it should be the raw email address without the name.The
From:
header can contain a name as well. In short, if it doesn't break anything with Drupal 7 then I agree with #222.Comment #228
imclean CreditAttribution: imclean at Digital Ink commentedTo clarify,
$message['from']
is not the same as$headers['From']
. Although the email addresses can be the same.See also hook_mail().
Comment #229
mcdruidI don't think this should be at RTBC.
It sounds like we need a new patch, and also that there's some disagreement as to whether the latest proposed change is correct.
Comment #230
RoSk0I've done another round of reading and testing and it looks like I was wrong in #225.
This patch implements suggestion from #222 , thank you Jeff (@webservant316) and sorry, and I believe really is a correct way forward.
Removed target tag as we already missed it, but hopefully...
Comment #232
RoSk0Updated tests to match changed approach from #222:
- changed only variable names in
MailTestCase::testFromAndReplyToHeader()
- fixed email:from assertion in
ContactSitewideTestCase::testSendPersonalContactMessage()
and added email:headers:from assertionComment #233
drumm#3313951: Don't forge "from" mail in contact form but use "reply-to" functionality
Comment #234
BramDriesenLong time since I've seen D7 code :). Patch looks good, everything well documented with inline comment and the tests are very readable. All tests are passing on all php/mysql versions, so I believe setting it to RTBC is justified.
Comment #235
aitala CreditAttribution: aitala commentedHI,
Am I missing something? I've applied patch #232 but I am not seeing the 'Reply-To' headers in emails coming from the contact form.
Thanks,
Eric
Comment #236
aitala CreditAttribution: aitala commentedI found the issue but I'm not sure how to fix it...
The email headers it generates are 'incorrect', for example
Note that after the 'MIME-Version' line, the following six lines begin with a space. That _appears_ to be enough to cause issues. I think.
I used the Header tester at: https://mxtoolbox.com/Public/Tools/EmailHeaders.aspx and it did not find the 'Reply-To' field unless I removed the leading space... I'm just not sure how to fix it in the patch.
Thanks,
Eric
Comment #237
pandaski CreditAttribution: pandaski at govCMS (Australian Government Department of Finance) commented@aitala feels like an old bug.
https://bugs.php.net/bug.php?id=73507
Comment #238
aitala CreditAttribution: aitala commented@pandaski I have turned off mail.add_x_header in the php.ini and settings.php and that had no effect.
This is with PHP8.1 but when I switch it back to PHP7.4, things seem to work... (I just noticed this...)
ADD: https://stackoverflow.com/questions/70502065/php8-mail-function-adding-s...
and
https://www.drupal.org/project/drupal/issues/3270647 & https://www.drupal.org/project/drupal/issues/3319062
Eric
Comment #239
TR CreditAttribution: TR commented@aitala:
You've correctly identified the cause of the problem. That happens because of the line-endings issue in Drupal/PHP/Mail.
The RFC says that all line must end with \r\n.
Earlier versons of PHP and some earlier versions of some mail servers don't respect the RFC.
That hasn't been true for many years now, yet Drupal core and many mail servers still try to force line endings to be wrong in order to accommodate broken implementations.
IMO Drupal should be strictly adhering to the RFC. All headers and all line MUST end with \r\n. Using a different line ending always causes problems, and at this point in history we should reject the incorrectly-formatted emails that use only \n as line endings.
Comment #240
Vivek Panicker CreditAttribution: Vivek Panicker at Innoraft for Drupal India Association commentedHello everyone!
Can someone please look into this?
I am not able to contact the module maintainers to help with review/merge of MRs :(
Comment #241
fjgarlin CreditAttribution: fjgarlin as a volunteer and at Drupal Association commentedNote that drupal.org itself is affected by this issue.
Issue has been RTBC since October 2022. Is there anything else that needs to be done or that we can help with?
Comment #242
drummSince this is a backport, I wanted to check if there were any changes in modern Drupal to be aware of.
D8 commit: https://git.drupalcode.org/project/drupal/-/commit/8e92d3b
Current implementation: https://git.drupalcode.org/project/drupal/-/blob/11.x/core/lib/Drupal/Co...
The approach for D7 is different, the API explicitly accepts a reply-to instead of from address. So that does not apply for the current approach here since it is rewriting the from email address.
Comment #243
drummAnd I rebased the MR for GitLab CI testing.