Introduce email plugins

I had a first go to understand, and found it not so easy😃. I feel that for a module to send an email should not be so complicated. It seems much simpler in DSM+. Could you help me compare them? Either we could make this simpler, or list clear reasons why it is more complex.

1) These lines seem to happen a lot. I feel it should be just one line to send a mail. The underlying layer can handle the separate build/render/send.

try {
        $email = $this->emailRenderer->render($definition, ['items' => $items], $langcode);
        $this->mailer->send($email->to($recipient));
        $sent++;
      }
      catch (TransportExceptionInterface | PluginException $e) {
        Error::logException($this->logger, $e);
    }

2) I'm confused by inconsistencies between the modules, they seem far from following a pattern. Some have a builder some not. I feel that one class is enough and it can have separate functions for the parts. All modules can have the same 2 functions.

I general, I like the idea of declaring emails with a plugin like this. The YAML syntax is very straightforward and consistent with other plugins, and for more dynamic use cases, developers can use a deriver class. Having the emails formally declared opens up use cases that are difficult to deal with now. In Easy Email, I had to make my own plugin for emails and then declare the core emails myself in order to make them overridable. I think discoverability will be nice for other use cases like ECA, as well.

A couple of specific changes I propose we consider for the plugin definition.

1. Option for template instead of controller: I like the _controller option. It makes rendering an email very similar to rendering a route. But I think many emails could probably be rendered using only a template that would use the provided params as variables. In such cases, the controller is a boilerplate creation of a render array specifying the template. I propose that we allow developers to skip that boilerplate by adding a key to the plugin specification to specify a template instead of a controller. So the developer can choose one or the other. And if they choose neither, we can default to a template that is derived from the email's machine name. The implementation for this could be that we always call a controller, but there's a default controller that we write that renders a template from the email specification if no other controller is specified.

2. Typed data for parameters: In Easy Email, my specification includes data types for the parameters. This allows me to match up the parameters with fields in the email entities, such as entity references for users or Commerce orders. I don't think we should require developers to provide the types for their parameters, but it would be nice to support it if we are able to do so.

3. Consider whether we really should support closures: It's a neat idea, but I'm struggling to figure out the use case where it really helps. Yes, it can be nice to have the definition of the rendering function in the same place as the call site, but it seems like if we encourage this type of behavior, we make it harder for people to override the rendering of emails. In the case where we make everyone who declares an email provide a controller or use a default one that renders a template, that specification will be alterable by any developer who wants to provide their own controller instead. How would that work with a closure? The closure is being created at the same time as the call to send the email is being made, so it seems like it you won't be able to alter it ahead of time. You might be able to catch it before it's rendered, but that's an email by email alter rather than just altering the email specification for all emails of this type. So, I think we should think this one through a little more. Personally, I think a cleaner specification with clearer options for overriding is more important, but I'm open to changing my mind if someone can provide some concrete use cases where the closure is superior.

Regarding the rest of the code in the MR and related commits:

I feel that for a module to send an email should not be so complicated.

I agree with this sentiment. I think sending an email should be roughly as simple as it is now, which only requires calling a single method with module, key, params, etc. The way it's currently implemented here requires the developer to know more about how the email system works than they probably should have to know. I think if we don't support closures, this should be possible.

Next, I think that your code demonstrates the need for the Drupal adaptation layer that has been a controversial topic in our previous discussions. Here are a few key snippets:

<?php
$params += $definition->getDefaults();
    $context = [
      'id' => $definition->id(),
      'module' => $definition->getModule(),
      'key' => $definition->getKey(),
      'langcode' => $langcode,
    ];
    $this->moduleHandler->alter('email_params', $params, $context);
?>

<?php
$context = [
          'id' => $definition->id(),
          'module' => $definition->getModule(),
          'key' => $definition->getKey(),
          'langcode' => $langcode,
          'email' => $email,
        ] + ($build['#props'] ?? []);
        unset($build['#props']);
        $this->moduleHandler->alter('email_build', $build, $context);
?>

<?php
$context['metadata'] = BubbleableMetadata::createFromRenderArray($build);
        unset($context['email']);
        $this->moduleHandler->alter('email_post_render', $email, $context);
?>

These are alter hooks where you're passing around multiple objects. The specification for what is in the params and context changes a bit per call, but it's clear that passing around the params alone or the Symfony email object alone is not enough. If I'm developer, I have to understand which different types of objects I'm going to get in each of these alters, and I have to know what is expected to be in context at each point, as it changes. The Drupal adaptation layer would provide a single object that we could pass around that would encapsulate the context, params, and potentially the final Symfony email. It would simplify the API that developers have to interact with at each point.

Moreover, I think we should consider using the adaptation layer object in place of the initial params array. For example, most email call site code in the current API looks something like this:

<?php
$params = [
  'user' => $some_account,
  'order' => $commerce_order,
  // ...
];
$this->mailManager->mail('commerce_order', 'order_receipt', $to, $langcode, $params, $actually_send);
?>

The call site code in your commits is a bit different, but much of the internal code is passing around these same variables. With the adaptation layer, I think we could get to something like this:

<?php
$email = $this->mailManager->new('commerce_order.order_receipt')
  ->setTo($to)
  ->addParam('user', $some_account)
  ->addParam('order', $commerce_order)
  ->setLangcode($langcode)
  ->setSend($actually_send);
$this->mailManager->mail($email);
?>

I think the latter is easier to read and more discoverable, as the context variables will have specific methods that can be autocompleted in IDEs, found by AI code agents, etc. And now we have one object we can pass around internally, though alter hooks or event subscribers, until we finally get to the point of actually sending the email, where we convert it into a Symfony Email. (We potentially could also have a Symfony Email embedded it that is being updated by the parent object all the time so there's no conversion process at the end. I don't have a strong opinion about that.)

Those are my initial thoughts on the MR and other commits. I think it's a good start on layers 5 and 6, as currently defined in the proposed resolution in #3534136: Requirements and strategy for Core Mailer, but I think it would benefit from us discussing layers 3 and 4 in more detail, which is where my comments above are focused. But overall, I'm feeling optimistic. After seeing this work and talking to @adamps last week, I feel like I'm starting to see the outlines of a workable solution. I'm looking forward to discussing this in more detail at our next meeting.

Looking at this code example:

<?php
$email = $this->emailManager->getEmailBuilder()
         ->id('contact.' . $key_prefix . '_mail')
         ->param('message', $message)
         ->langcode($recipientLangcode)
         ->build() # returns a Symfony Email

$this->mailer->send($email->to(...$recipients)->replyTo($message->getSenderMail()));
?>

This code suggests that mail ID, parameters, and langcode will not be available at the point the email is being sent. This will complicate the process of anyone attempting to alter the sending of the email, such as rerouting or logging it.

The routing.yml file is flexible and it can map to many "things" - not only to a controller but also a form, an entity list, and entity form, an entity view, and maybe more. In the case of emails, I feel that we only have "thing" and so we don't need the flexibility. There is another pattern which is more common/familiar and simpler: plug-in plus attribute. We could make an @EmailBuilder plug-in and put the metadata (currently in the yaml file) on the attribute.

The current controller design only covers the email body. To build an email we also need to cover the subject, recipient and perhaps more. These must also be set within the switched environment because they are language dependent, and can affect render context. I feel we should let go of the idea of returning a render array (even though it does give a pleasing similarity with the routing file) and instead the controller should act on the Drupal EmailInterface object, typically in the build phase, but could also be init or post-render. This gives the entire implementation in a single class which is easily swappable.

I didn't read everything, neither comments nor code, but I do have a few thoughts:

* Route controllers are not plugins and I honestly think that the current way routes are defined is really not great DX. Between defaults, options and other keys they can have, I still have to look up examples every time I do something non-trivial (types for example). There is a reason we tried to move to attributes for route definitions instead of a yaml file for years. I think we shouldn't try to reuse terminology like defaults and controller for this. IMHO, it makes it harder to use, not easier.
* This uses a plugin manager but they aren't really plugins, this is quite confusing. I think plugins are pretty much expected to have a class property in the definition. There can be a default for it when for example using yaml discovery, such as local tasks (see LocalTaskManager) but it's still there. Plugins are then expected to be a class implementing an interface, not just a callback.
* I'm not sure if plugins or something else is better, but I think we should pick a side, either they are plugins or not. If they're not, we shouldn't use a PluginManager.
* On one side, plugins seem like the default choice but if all we need is a callback and especially when that is hard to put on an interface due to dynamic arguments (although I think that's still up for discussion) then that's possibly indeed not the best fit. Plugins are good when you need to instantiate them, have multiple methods and state/configuration. I don't really see that here. For discovery, OOP hooks *might* be an inspiration, not sure (discovery of attributes on methods/classes in a specific namespace, service-autoregistration).
* I do think that figuring out the API at the call site is more important than discovery/definition of the thing that builds the mail and I think it should influence how those things work, not the other way round. Also, to add what what I said in the call we had, I think we didn't fully understand each other yet. In my opinion, what I have in mind might make it *easier* for something like easy mail, not harder. More on that later. The call site API is important and should be generic, but that doesn't mean that it should require as little code as possible to invoke.

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

@znerol, I've read through the latest MR, and it's pretty complex. I'm having some trouble following it from reading the code alone. I think we have a meeting scheduled later this week, so perhaps you could give us a walkthrough then of the classes and how you've broken up the responsibilities between the EmailBuilder, EmailRenderer, EmailManager, Email plugin instance class, and modulename.emails.yml definitions. I can understand the code class by class, line by line, but what I feel like I'm missing is the high-level overview of how things fit together and why.

Also, I think it would be helpful for review to have a minimal example module that sends a "Hello {{ user }}" HTML email. I see you have some examples in your test module, but I think those implementations could be more complex than the typical email because you're checking edge cases. An example of what we expect most developers to do would be useful for reviewing the DX of the new API.

Looking forward to discussing this further at our next meeting.

I'm attempting to test this, but I can't seem to get even a minimal example working. Which branch should I be using? I'm currently trying with 3539651-email-content-yaml-plugin.

Here's what I've got. In a module called test_rig:

test_rig.email.yml

test_rig.test_notify:
  label: 'Test Rig: Notify'
  module: 'test_rig'
  key: 'test_notify'

Then, I've got a controller I can call to trigger an email as a test.

<?php

declare(strict_types=1);

namespace Drupal\test_rig\Controller;

use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Mailer\DefaultEmail;

/**
 * Returns responses for Test Rig routes.
 */
final class TestRigController extends ControllerBase {
  public function __construct(
    protected readonly \Drupal\Core\Mailer\EmailFactoryInterface $emailFactory,
  ) {}

  public function testNotify(): array {
    $email = $this->emailFactory->create('test_rig.test_notify');

    $body = [];
    $body[] = $this->t('This is a test email from Test Rig module.');

    $subject = $this->t('Test Email from Test Rig');
    $params = DefaultEmail::createParams($subject, $body);

    $email->to('wayne@zengenuity.com')
      ->langcode('en')
      ->params($params);

    $result = $email->send();
    
    $build['content'] = [
      '#type' => 'item',
      '#markup' => $this->t('It works!'),
    ];

    return $build;
  }

}

When I run this, I get:

Symfony\Component\Mime\Exception\LogicException: An email must have a "From" or a "Sender" header. in Symfony\Component\Mime\Message->ensureValidity() (line 132 of /userdata/dev/sites/core-dev/vendor/symfony/mime/Message.php).

So, it's not picking up the site mail as the default sender. But then, I also tried to add it to the $email object, and it's not clear how to do that. I don't see any methods on that object that would allow me to add headers. The only examples in the test code are for adding headers to the Symfony Email object in an alter.

I have created a simple YAML-based implementation in branch 3539651-email-yaml-plugin-zengenuity.

This implementation may not include all the functionality of @znerol's version, but it's very simple and easy to understand. I'm posting it so that we can discuss the code differences and requirements at a meeting in the near future.

The implementation requires that emails be declared in a YAML file in your module. Beyond that, you have three ways you can implement the content of the emails.

1. At the call site, you can specify the subject, body, and recipient to send a simple email with the default email plugin.
2. You can specify twig templates in your YAML declaration that will be used for the HTML and/or text bodies of the email. These twig templates can use parameters, which are mapped to twig variables for use in the templates.
3. You can specify a custom class to use in place of the default plugin class. When using a custom class, you can override the getHeaders(), htmlBody(), and textBody() methods to provide programmatically generated values.

You can also combine these approaches. For example, you could have a custom class and also use declared twig templates for the body of the email.

The system defines four alter hooks:

hook_email_info_alter(&$email_info)
hook_email_pre_build_alter($email)
hook_email_pre_render_alter(&$body, $context)
hook_email_pre_send_alter($email, $symfony_email)

Note that $email in the hooks is an email plugin object. You can access the Symfony Email object in the last hook. It's also possible to call the EmailManager::build() and EmailManager::send() steps separately if you want to manipulate the Symfony Email object before sending. This is not normally needed, though, and you can call send(), which will call build() internally.

I'm attaching a module to this comment that demonstrates the various approaches for sending emails.

I have updated the 3539651-email-yaml-plugin-zengenuity branch with the changes discussed at our last meeting. These changes are:

1. Removed ability to set body text directly from the call site.
2. Changed subject() method parameter to type \Stringable.
3. Changed template variable for body to use #params as the variable name.
4. Removed the ability to pass a built Symfony Email object to the send method
5. Removed hook_email_post_render()
6. Added ability to define an email plugin using an attribute on a plugin class rather than YAML. (YAML also works, following the pattern from layout plugins.)

In the 3539651-email-yaml-plugin-zengenuity-with-core-emails branch, I also took a first pass at converting a core email to use the new system. Right now, only the update notification email is converted. I chose that one because the text is very dynamic, and I wanted something that needed to use a plugin class. It's implemented using just an attribute-defined plugin class, into which I ported the existing code for generating that email body. No YAML file needed, and minimal changes at the call site. The call site is pretty complex, so perhaps more refactoring would be useful, but I skipped that as out of scope for now.

Based on our discussion from the last call, I've updated the 3539651-email-yaml-plugin-zengenuity and 3539651-email-yaml-plugin-zengenuity-with-core-emails branches to allow headers to be altered in hook_email_pre_render(). Making this change also allowed me to provide a way to allow altering of the Symfony Email class, which was something @znerol's branch originally had marked as a to-do.

Could you open a MR(s) on the active branch(es) so it's easier to review and discuss the code?

I've updated my branches to make some changes that @znerol and I discussed on Slack. These changes are:

Added the ability to set the subject and recipients of the email from the plugin class. There are two new methods for this:

<?php
public function getDefaultRecipients(): ?array;
public function getDefaultSubject(): \Stringable|string|null;
?>

The return value from getDefaultRecipients() is expected to be a nested array like this:

<?php
[
  'to' => ['to@example.com'],
  'cc' => ['cc@example.com'],
  'bcc' => ['bcc@example.com'],
]
?>

You're not required to have all three recipient type keys in there, but if you have them, and if the corresponding recipient type has not been set at the call site, these will apply.

It's similar with getDefaultSubject(). If no subject is set at the call site, then the value from getDefaultSubject() will apply.

Once I had these in place, I realized we could also allow people to set the subject and recipients statically in YAML declarations, so I implemented that. The structure of the declaration is the same as expected from the methods above.

The defaults are applied in EmailManager before the headers are evaluated using a preBuild() method in the default email plugin class.

With these changes, it's now possible to send the simplest of emails with just a YAML declaration and template file. The call site in this case can be as simple as:

<?php
$email = $this->emailManager->createInstance('my_module.my_email');
$this->emailManager->send($email);
?>

In this case, I would expect subject and recipients to be declared in the YAML file, but you still have the option to set the subject, recipients, and parameters at the call site as needed.

Emails declared by email plugin classes can override:

<?php
public function htmlBody(): RenderableInterface|array|null;
public function getDefaultSubject(): \Stringable|string|null;
public function getDefaultRecipients(): ?array;
?>

And then the call site could be as simple as above, with the option to set subject, recipients, and parameters at the call site if needed.

@znerol I worked on the issues we discussed this week. Here's what I've done

1. I added back the plugin manager service definition and made it autowirable.

2. I added getters and setters for params on the email plugin default class and updated the interface.

3. I reworked the template declaration to be more like layout discovery. I'm not sure I've done this 100% properly, but it does seem to work. Now, in a module, you can declare your email body templates in modulename.emails.yml like this:

email_tester.from_template:
  label: 'Twig template email'
  description: 'Email with twig template'
  module: 'email_tester'
  key: 'from_template'
  html_body_template: 'testers/email-tester-html-body'
  text_body_template: 'email-tester-text-body'

And you don't need to declare them with hook_theme(). Instead, you put them in an emails subfolder of your module, and the plugin manager will look for them there and automatically declare them. The theme declaration requires the email object, and there's an initial preprocess that will pull the params and the langcode out of that for use in the twig template. So, the template you declare in the above can have something like this inside:

<p>Hello {{ params.name }},</p>

If you don't declare html_body_template or text_body_template in your emails.yml file, the plugin manager will attempt to find a template in your module at emails/email-body-html-[PLUGIN_ID_WITH_DASH_REPLACEMENT].html.twig. So, it's possible now to simply declare the email in the YAML file with no template declarations, and then create emails/email-body-html-XXX.html.twig without anything else.

Agreed that we should either work with a base template and suggestions, which don't need to exist but are a bit tedious to provide by modules, or make it optional.

It's also not quite clear to me what exactly the HTML templates especially would contain. On top of my head, I have a few thoughts:

* I think a common use case is to have a common wrapper/HTML for all mails, so maybe that should be there by default and we should have two templates, the content and a wrapper?
* It might all be quite different once HTML mails are a built-in feature, but right now, it's pretty common that you need to alter in templates and alter the content (such as converting it to a format that handles basic formatting within a HTML mail)
* even though core currently doesn't actually support HTML mails, in practice, e-mails are quite often built as HTML (for example contact module) and then the HTML is converted to plain text. The result tends to be rather ugly, but it's still more manageable than the opposite, "upcasting" text to HTML. Right now body and text are completely separate. Are plugins expected to provide both? Do we have a primary format that we convert to the other as a fallback? how would a text template even look? how is it translated?
* That said, defining in detail how HTML mails are actually going to work and how far core goes to support that seems way out of scope for this issue and we should probably handle as little as possible of that in this initial issue?

And yes, we really need some tests/example conversions here, it's still all very abstract and hard to imagine how this works in practice.

Based on our last meeting, I made the following changes to the 3539651-email-yaml-plugin-zengenuity and 3539651-email-yaml-plugin-zengenuity-with-core-emails branches.

Removed All Autodiscovery of Templates: Modules using the html_body_template and text_body_template parameters in the email type declaration must also declare whatever template they are using in hook_theme().

Ensured that emails can be declared without a template: I think this may have been possible before, but I have confirmed that if you use a class to declare an email type, you don't need to provide a template. I also confirmed that you can use either an HTML or a text template, and you aren't required to use both. If you provide HTML but not text, a plain text version gets automatically generated by Symfony Mailer, but if you provide just a text template, you get a plain text only email.

Provided a Default email type in Mailer module: With this type, it's now possible to send emails solely from the call site, without declaring an email type. This might also be useful for triggering emails from tools like ECA.

Allowed for the key to be automatically derived from the email plugin ID: We still have an open question as to whether module and key are useful parameters for email types. But in the meantime, it's now possible to leave them out of your declaration, and they will be automatically derived.

With the changes, the minimal declaration for an HTML templated email looks like this:

MYMODULE.emails.yml:

MYMODULE.templated_email:
  label: 'Twig template based email'
  html_body_template: 'MYMODULE_html_body'

In this case, you'd also need to declare MYMODULE_html_body in hook_theme.

All other configuration in the YAML file is optional. Here is an example with all available parameters declared except class:

MYMODULE.from_template:
  label: 'Twig template based email'
  description: 'Email with twig template'
  module: 'email_tester'
  key: 'from_template'
  html_body_template: 'MYMODULE_html_body'
  text_body_template: 'email_tester_text_body'
  default_subject: 'Default subject from template email'
  default_recipients:
    to:
      - 'someone@example.com'
      - 'other@example.com'
    cc:
      - 'cc@example.com'
    bcc:
      - 'bcc@example.com'

You can declare an email type class either in YAML or by using an attribute on the class. In YAML, this looks like:

MYMODULE.with_class:
  label: 'Email with custom class'
  class: '\Drupal\MYMODULE\CustomEmail'

As an attribute it looks like this:

<?php
namespace Drupal\MYMODULE\Plugin\Email;

#[Email(
  id: 'MYMODULE.from_attribute',
  label: new TranslatableMarkup('Email From Attribute'),
  description: new TranslatableMarkup('Email configured from attribute'),
)]
class EmailFromAttribute extends EmailPluginDefault implements EmailPluginInterface {
}
?>

When sending an email declared by any of the above, the call site code will look like this:

<?php
$email_manager = \Drupal::service('plugin.manager.email');
$email = $email_manager->createInstance('MYMODULE.from_template');
$email->setParams([
  'name' => 'John Doe',
  'order_total' => '$123.45',
]);
$email_manager->send($email);
?>

You can also set subject, recipients, and langcode at the call site using the subject(), to(), and langcode() methods.

With the default email you can specify the body as a parameter:

<?php
$email_manager = \Drupal::service('plugin.manager.email');
$email = $email_manager->createInstance('mailer.default');
$email->subject('Test from call site')
  ->sender('from@example.com')
  ->to('to@example.com')
  ->langcode('en')
  ->setParam(
    'html', [
      '#type' => 'markup',
      '#markup' => '<p>This is a test email generated at the call site.</p>',
      ]
  );
 $email_manager->send($email);
?>

It's also possible to use a text parameter for a text body in the mailer.default email type.

Starting to work on this issue, first problem for me was to test the mailer. While the provided Tester-module was a good start, there were some errors, because the component evolved.

So I created a sandbox project for the tester, hoping it might be easier to follow the changes of the mailer-core