Setting the "Return-Path" header doesn't follow RFC 5321

Thanks @cilefen. These days it's probably best to use an external mail delivery service such as Sendgrid or Mailgun which have generous free tiers for smaller websites.

But Drupal does need to cater for all possibilities.

This is the flow of the Return-Path header in Drupal's default implementation.

Drupal\Core\Mail\MailManager:

    // To prevent email from looking like spam, the addresses in the Sender and
    // Return-Path headers should have a domain authorized to use the
    // originating SMTP server.
    $headers['Sender'] = $headers['Return-Path'] = $site_mail;

This then gets handed off to Drupal\Core\Mail\Plugin\Mail\PhpMail:

    // If 'Return-Path' isn't already set in php.ini, we pass it separately
    // as an additional parameter instead of in the header.
    if (isset($message['headers']['Return-Path'])) {
      $return_path_set = strpos(ini_get('sendmail_path'), ' -f');
      if (!$return_path_set) {
        $message['Return-Path'] = $message['headers']['Return-Path'];
        unset($message['headers']['Return-Path']);
      }
    }

$additional_headers = isset($message['Return-Path']) && ($site_mail === $message['Return-Path'] || static::_isShellSafe($message['Return-Path'])) ? '-f' . $message['Return-Path'] : '';

If $message['headers']['Return-Path'] is set and $return_path_set evaluates to FALSE then then the "Return-Path" header value gets assigned to a separate parameter $message['Return-Path'] and the header "Return-Path" is removed. This is perfectly valid, just confusing as it shouldn't be called "Return-Path".

If the "Return-Path" is set both as a header and in php.ini then $message['headers']['Return-Path'] is not unset and $additional_headers is set to ''. This is a problem.

The first problem is setting the Return-Path header at all.

The second problem is the envelope sender stored in php.ini could be different to the Return-Path header. This means the header could be different to the envelope sender.

Ideally, Drupal would have no reference to Return-Path anywhere, except perhaps a general overview of how the envelope sender works. All variables which influence this could be called something different, such as Envelope-Sender.

For example, removing "Return-Path" and adding support for an optional parameter "envelope_sender". The comments could include how this can influence the "Return-Path" header.

Something like this would be needed to maintain current behaviour in MailManager->doMail():

$params['envelope_sender'] = $site_mail;

We're testing this in a contrib project. PHPMailer SMTP explicitly removes the Return-Path header. It also allows configuring the behaviour of the envelope sender (SMTP command MAIL FROM:).

There's no need to introduce a new parameter. From header is already seperate to the from address.

Addresses #12.

The next step would be to explicitly remove the Return-Path header. If someone wants to add it they can use a contrib module.

$message['from'] is always set.

Postmastery has a good explanation: About the Return-Path header.

Updated the IS to add some ramifications of having 2 Return-Path headers.

Thanks @sanduhrs for keeping this going and including a test.

+++ b/core/tests/Drupal/FunctionalTests/MailCaptureTest.php
@@ -54,6 +54,12 @@ public function testMailSend() {
+    $this->assertArrayNotHasKey('Return-Path', $email['headers']);

The header is case-insensitive so it might not be set to "Return-Path". Would this affect the test at all?

@alexpott, thanks for the feedback.

@imclean what is the bug here?

Setting the Return-path header may have been a design choice for Drupal, probably to get around problems with cheap hosting and non-compliant mail servers, but it isn't the correct behaviour.

True there is a "MAY" in the RFC, so mail servers can just ignore the extra header. I think there is confusion in the Drupal community about what the header is used for, how it should be formatted and where it should be set. Originally it was probably being used to help deliverability but I don't think it actually does.

This change will probably cause issues for Drupal mailers - and certainly test fails for some contrib modules.

Contrib modules which rely on the header being set by Drupal may need to be updated. There shouldn't be many though, it's it's only related to sending email, not constructing or adding attachments. See: https://www.iana.org/assignments/message-headers/message-headers.xhtml

That said, the proposed change here should only affect those sending email with Drupal core. Other modules which send email are free to do what they like.

Yes adhering to RFCs is a good idea but the mail RFCs are a bit notorious. If the header is being ignored by Sendgrid and Mailchimp then isn't the correct thing being done anyway.

At the very least we need a change record. But it would also be great if some analysis of the impact / necessity of this change from the POV of the contrib eco-system.

This is a good point. Which modules are setting the Return-Path header, or relying on it being set, and why? What does it achieve?

This could be a good opportunity to reduce confusion about what should be set where. Modules which construct emails, such as newsletter or MIME modules, might be adding it when they shouldn't be. It's possibly a legacy from earlier versions of the modules which the current maintainers aren't confident in removing. There's understandably a fair bit of caution and inertia when old modules have thousands of users.

As mentioned in #8, we're using the module PHPMailer SMTP, which explicitly unsets the Return-path header. We're using it with Mailgun, Sendgrid, Outlook/Office365 and soon Mailchimp. I'm sure it's been used with other mail services.

The Symfony Mailer module also doesn't set the Return-Path header.

@Berdir #3165762-11: Add symfony/mailer into core:

One thing to check would be the Return-Path header. I saw that I ended up with two return-path e-mails.

This is what's happening now.

@alexpott,

@imclean what is the bug here?

There is another problem. In my comment in #5 I tried to explain it.

The second problem is the envelope sender stored in php.ini could be different to the Return-Path header. This means the header could be different to the envelope sender.

This happens when both the sending module sets $message['headers']['Return-Path'] and there is a different php.ini value in sendmail_path. It can result in 2 different return-path headers.

This change will probably cause issues for Drupal mailers - and certainly test fails for some contrib modules.

This is Core behaviour so I'm not sure how a contrib module would test it. A module which generates and sends email would check its own input and output, but by this stage the message has been handed off to PhpMail to send the email.

To approach this another way, why is Drupal setting the Return-Path header, against the recommendation of the RFCs?

Set to Needs review to hopefully encourage further discussion.