Problem/Motivation

#3165762: Add symfony/mailer into core introduced the system.mail mailer_dsn config as a way to configure a system wide symfony mailer DSN. The form of this string is an URI (plus some additional syntax sugar for special transports).

The DSN as a single string is convenient for apps which are configured through environment variables and container params (as in case of symfony). However, a more structured form would be more convenient in Drupal. Especially considering that at some point there will be GUIs for transport configuration. Implementing GUI forms for different transport factories will less complicated with structured config instead. Also specifying the values in settings.php is less intimidating when the config is structured vs. a raw URI string.

Steps to reproduce

Proposed resolution

Instead of that (current implementation):

$config['system.mail']['mailer_dsn'] = 'smtp://user%40some-domain.org:%5EC%3E6P2EdL.b%3C%3F9jBXgOu8%2AA%3E@smtp.example.com:25'

Allow people to specify the DSN like this:

$config['system.mail']['mailer_dsn'] = [
  'scheme' => 'smtp',
  'host' => 'smtp.example.com',
  'port' => 25,
  'user' => 'user@some-domain.org'
  'password' => '^C>6P2EdL.b<?9jBXgOu8*A>'
];

Note that this has one caveat: The mailer DSN string allows for advanced use cases. Namely configuring load balancing over multiple transports. This configuration isn't possible using a structured DSN. Contrib or custom modules could step in and cover those more exotic configuration.

Remaining tasks

User interface changes

API changes

Data model changes

Convert system.mail mailer_dsn from a string to a map.

Release notes snippet

The system.mail mailer_dsn format has changed in 10.2.0-beta2. This feature was new in 10.2.0-beta1; sites that configured this in beta will need to reconfigure their settings.

Issue fork drupal-3399645

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

znerol created an issue. See original summary.

znerol’s picture

Status: Active » Needs review
znerol’s picture

Title: Use structured DSN insteaf of URI in system.mail mailer_dsn » Use structured DSN instead of URI in system.mail mailer_dsn
smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs change record

Love the idea, think we will need a change record for it.

znerol’s picture

I propose to just link and update the change record (if this gets in before the 10.2 release). Basically we'd need to update the examples:

In order to configure the DSN, use the following commands:

  • For the default sendmail transport:
    drush config:set system.mail mailer_dsn.schema sendmail
    drush config:set system.mail mailer_dsn.host default
  • For mailhog on localhost:
    drush config:set system.mail mailer_dsn.schema smtp
    drush config:set system.mail mailer_dsn.host localhost
    drush config:set system.mail mailer_dsn.port 1025
  • For authenticated SMTP:
    drush config:set system.mail mailer_dsn.schema smtp
    drush config:set system.mail mailer_dsn.host smtp.example.com
    drush config:set system.mail mailer_dsn.user some-user
    drush config:set system.mail mailer_dsn.password some-pass
znerol’s picture

Status: Needs work » Needs review
Issue tags: -Needs change record
znerol’s picture

Issue summary: View changes

Added a caveat to the issue summary: The structured DSN doesn't cover a rather exotic use case.

imclean’s picture

Load balancing could be supported by providing multiple DSN configs, although it might complicate things a bit.

$config['system.mail']['mailer_dsn'][] = [
  'scheme' => 'smtp',
  'host' => 'smtp.example.com',
  'port' => 25,
  'user' => 'user@some-domain.org'
  'password' => '^C>6P2EdL.b<?9jBXgOu8*A>'
];
$config['system.mail']['mailer_dsn'][] = [
  'scheme' => 'smtp',
  'host' => 'smtp.anotherexample.com',
  'port' => 465,
  'user' => 'anotheruser@some-domain.org'
  'password' => '^C>6P2EdL.b<?9jBXgOu8*A>'
];

or

$config['system.mail']['mailer_dsn']['default'] = [ 
  'scheme' => 'smtp',
  'host' => 'smtp.example.com',
  'port' => 25,
  'user' => 'user@some-domain.org'
  'password' => '^C>6P2EdL.b<?9jBXgOu8*A>'
];
$config['system.mail']['mailer_dsn']['second'] = [
  'scheme' => 'smtp',
  'host' => 'smtp.anotherexample.com',
  'port' => 465,
  'user' => 'anotheruser@some-domain.org'
  'password' => '^C>6P2EdL.b<?9jBXgOu8*A>'
];
znerol’s picture

Load balancing isn't possible with the current mail system in core and it wasn't a goal in the original issue #3165762: Add symfony/mailer into core. I really think core shouldn't try to implement it. It is an advanced use case and for that kind of things we have contrib.

adamps’s picture

> It is an advanced use case and for that kind of things we have contrib.
I agree. It's part of my plan for DSM, but it hasn't been implemented yet😃.

adamps’s picture

I think it's a very good idea, and it's similar to what we already do in DSM.

I see you have matched the names of each parameter to the DSN class, which does make sense. FYI In DSM we used 'pass' (to match the examples in https://symfony.com/doc/current/mailer.html) and 'query' (because they are query parameters).

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Rebased so I could run the test-only feature

There was 1 failure:
1) Drupal\Tests\system\Functional\Update\MailDsnSettingsUpdateTest::testSystemPostUpdateMailerDsnSettings
'sendmail://default' does not match expected type "array".
/builds/issue/drupal-3399645/vendor/phpunit/phpunit/src/Framework/Constraint/Equality/IsEqual.php:94
/builds/issue/drupal-3399645/core/modules/system/tests/src/Functional/Update/MailDsnSettingsUpdateTest.php:41
/builds/issue/drupal-3399645/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
FAILURES!
Tests: 1, Assertions: 179, Failures: 1.

If we add to the existing CR think it should be noted with

$config['system.mail']['mailer_dsn'] = [
'scheme' => 'smtp',
'host' => 'smtp.example.com',
'port' => 25,
'user' => 'user@some-domain.org'
'password' => '^C>6P2EdL.b<?9jBXgOu8*A>'
];

is possible and the problem with load balancing.

rest looks fine.

longwave’s picture

I do wonder if we should just move to the structure in #9, allowing multiple DSNs but only supporting 'default' in core, similar to the way we allow multiple database definitions but in practice only the 'default' is used except for special cases. Wouldn't this then make life in contrib easier? Otherwise I think you will have to override this entirely and provide multiple configs elsewhere, and the system.mail one becomes redundant?

longwave’s picture

Status: Reviewed & tested by the community » Needs review
znerol’s picture

Wouldn't this then make life in contrib easier? Otherwise I think you will have to override this entirely and provide multiple configs elsewhere, and the system.mail one becomes redundant?

As per @berdir over in #3379794:

the most likely use case for a contrib UI is going to be config entity/plugin based, so even if there is only one, we don't want to read it from raw config, but the config entity API.

.

Mailer transport config is also slightly more involved than the database since transports can be nested. E.g., the Transports transport inspects the X-Transport header of a message and dispatches it to other transports accordingly.

In order to effectively support multi transport configuration in core we'd have to implement a tree-like config structure. A simple key-value model (as with database definitions) wouldn't be sufficient.

longwave’s picture

Status: Needs review » Reviewed & tested by the community

Sending this back to RTBC. I think this is a valid improvement that we should get in before 10.2.0 otherwise we risk having to provide some form of BC.

borisson_’s picture

I was wondering if this did not mean we lost schema validation but it is the opposite, we are gaining a lot of validation here. I wonder if we should add the same regex we added in #3379102: Add validation constraint to `type: label` + `type: text`: disallow control characters, to disallow control characters?

znerol’s picture

@borisson_ true. Do you think this could go into a follow-up or is that a requirement for an initial iteration?

borisson_’s picture

I think that can be a follow-up.

znerol’s picture

Thanks to the fact that the config schema matches exactly the parameter names of the Dsn constructor, we actually can unpack the config array directly into the constructor parameters.

longwave’s picture

Status: Reviewed & tested by the community » Needs work

Discussed with @catch. We would like to land this in 10.2.0 to avoid adding backward compatibility later, but because we already released beta1 there is a chance some users will now have the old structure in their config.

As we cannot delete post update hooks, we propose making the old post update hook a no-op and overwriting the config with the new structure in a new post update hook. This will cover users upgrading from beta1 or earlier to the next release; we don't expect anyone to have started using this feature yet so we can safely overwrite the config as long as we do it before 10.2.0.

znerol’s picture

Status: Needs work » Needs review
alexpott’s picture

Status: Needs review » Reviewed & tested by the community

The approach to updates from beta1 to 2 make sense.

I guess we could add something to the release notes about needing to reconfigure it.

  • longwave committed 749e1f03 on 10.2.x
    Issue #3399645 by znerol, smustgrave: Use structured DSN instead of URI...

  • longwave committed 0b7c2a60 on 11.x
    Issue #3399645 by znerol, smustgrave: Use structured DSN instead of URI...
longwave’s picture

Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs change record updates

Committed and pushed 0b7c2a60a5 to 11.x and 749e1f030f to 10.2.x. Thanks!

Tagging for change record updates, as the examples there need to be updated now.

longwave’s picture

Issue tags: +10.2.0 release notes

Also tagging for release notes as we can add a line for beta-to-beta upgraders, just in case anyone started looking at this.

znerol’s picture

Status: Needs work » Needs review
Issue tags: -Needs change record updates

Setting status to NR for the updated change record.

longwave’s picture

Issue summary: View changes
Status: Needs review » Fixed

The updated change record looks good to me, thank you @znerol. Added a release note snippet as well.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.