Problem/Motivation

There's no Drupal 8 activity yet with Mailhandler and the project page contains a similar statement.

In the past months we have been very active with Drupal 8 development and decoupled many problems and released (or started to do so) many modules.
Inmail is a project that has been started as a base system to process incoming mail, including bounces.

Proposed resolution

Consider building mailhandler for Drupal 8 on top of inmail.

We can fetch mails from IMAP or through a drush command piped from an MTA. Adding more plugins for POP3 or other sources is easy.
When processing mail, inmail triggers analysers (plugins) that can classify a message, enrich a result object and then handlers (plugins) are triggered.

Inmail itself offers already multiple analysers for VERP and other bounce analysers. And we have handlers that trigger mailmute to put the address on hold (stop sending further mails).

We also natively support MIME. And in case storing mails is of importance we have a collect handler that passes the mails into an entity.

I would expect that mail handler could easily provide an analyzer and handler (or multiple) to implement all its fancy processing functionality.
This would allow mailhandler to drop quite some complexity and thus less work with porting to Drupal 8.

Remaining tasks

Discuss, decide, implement. ;-)

User interface changes

API changes

Comments

giorgio79’s picture

giorgio79 commented

These analyzers and handlers could be configured as Rules :) Also see #2135717: Provide an email content type mimicking all the email fields, such as from, to, subject, body

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

@gregario79:
We need and have a solution that works NOW. Rules is far from ready.

I consider using rules a good idea for wiring some higher level things together.
Our analyzers provide components that are a good fit for code.
We could have a handler (and/or analyser) that passes things into rules.

Finally, if we do anything entity related, it would be an new entity type and not a content type. We currently use the Collect module to pass mails into persistency. Note that this is also related to maillog that could be extended to log incoming mails too.

danepowell’s picture

danepowell commented

Hey, maintainer here. I'm happy to consider this, let me investigate more and get back to you.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

Awesome. We are also open for futher architectural suggestions to make our solution more robust / matching all kinds of applications.

I'd be happy to discuss collaboration also at dev days if you're here. We have a proposed session: Inmail: Inbound email processing in D8

danepowell’s picture

danepowell commented

It sounds like Inmail doesn't have any external dependencies, which is good if true. I assume you're going to add POP/IMAP support, but ideally this would be pluggable- many people have requested support for other protocols like MAPI, and for PHP-based mail libraries (instead of the C-based PHP IMAP extension that Mailhandler uses).

Looking at Inmail's architecture, it seems like it could be a replacement for feeds, since it has separate pluggable fetching and processing stages. I've mused about using Rules instead of Feeds to create content in #2001018: Plans for 8.x / transition from Feeds to Rules. So I wonder if the ideal approach is to use Inmail to handle low-level mail processing, Rules to handle content creation, and Mailhandler would be the "special sauce" that ties it all together.

Do you know of any other similar mail-handling projects? There used to be Mail API, but I think it's defunct now.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

We have pluggable deliverers and IMAP is an example implementation we provide. A second passive deliverer is a drush command to be fired from MTA.
Sure. We are happy to receive submissions for more fetchers. Test coverage required. ;-)

And yeah, we also want to create an entry point to hand over to rules.

We did quite some research and consolidated everything we knew of and spent a lot of time and love to properly abstract the things.
There is also:
https://www.drupal.org/project/bounce_handler

Unexpectedly we needed to write quite a few things on our own since we didn't find any good PHP library that did the relevant base work on its own.
Yet clean and well separated, but not overly abstracted.

Hope this helps.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

I have proposed a project idea for Google Summer of Code 2015 to port this module based on inmail.
https://groups.drupal.org/node/455978#project13

Let's see if someone applies. Would be great if the maintainers join mentoring the project.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

Anyone at DDD montpellier?
http://montpellier2015.drupaldays.org

How about a BOF to discuss the mail processing topic in general.

danepowell’s picture

danepowell commented

@miro_dietiker, I was just contacted by Mehul (therealssj, https://www.drupal.org/u/therealssj) who also seems to be interested in a D8 port for GSOC, you should coordinate.

Like I told him, if you work on the port in Github or a sandbox project and send me the link to review once it's semi-stable, I'm happy to make one or both of you maintainer.

danepowell’s picture

danepowell commented

mbovan (https://www.drupal.org/u/mbovan) has also expressed interested in a GSOC port.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

Yeah i reviewed their proposal and discussed the direction.
It seems many things would just end up bring inmail processing plugins.

Also i have seen that most issues in mailhandler are related to mail fetching.
Some features in mailhandler would directly be added as an inmail core feature. The Inmail UI for the fetchers / deliverers is too minimalistic now. It doesn't display a queue / mailbox status info with size and can not reliably batch process all items from UI.
And we should review all inmail implementation to make sure it is robust and covers all the known problems from mailhandler.

But would be awesome to make this happen as proposed.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

This project is happening as part of Google summer of Code, performed by Milos Bovan.
Mentors are Primsi and me.

I proposed to start with a sandbox. But it would be good if we can push the project into official repo as soon as we have reached some minimum milestone.
From there on we could work with regular review workflows and also get feedback of the primary maintainer(s).

giorgio79’s picture

giorgio79 commented

Somewhat related, https://www.drupal.org/project/message_private plans to use this Symfony bundle https://github.com/FriendsOfSymfony/FOSMessage

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

Not sure how much it is related. Inmail / MailHandler is about incoming mails.
The libraries you mention seem to be sending only.
If you are interested about sending frameworks, you might want to look at Courier too.

Receiving is a completely different problem.
Also inmail is no generic messaging. It focusses on processing mail with all its challenges.

giorgio79’s picture

giorgio79 commented

That lib "provides a data structure" that could be useful, and I assume it is similar to the Zend Mail class that can be used by both incoming or outgoing mail. For outgoing http://framework.zend.com/manual/1.12/en/zend.mail.read.html

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

One of the values Inmail offers is extensive Interface definitions and parsing mails including multipart mails and providing OO access to the mail parts. It already removes all this complexity, if you work with it.
We checked the situation and thought about using some of the libraries, but none was satisfying.

Thus, for now we stay with our interfaces and data structures and first iterate on completing UI and providing more handlers.
I'm open to consider other libraries again, but only if we have a larger set of well test covered cases and the library proofs that the majority of things is simplifying.

mbovan’s picture

mbovan commented
Issue tags: +gsoc2016

Hello! Regarding #12, I've created a roadmap based on my proposal document.

The project code updates will happen on Mailhandler D8 Sandbox.

Timeline

(by weeks/phases)

[#W1][#W2] Proof of concept

(23/05/2016 - 08/06/2016)

Creating a new D8 version of Mailhandler. Info, module files, configuration, schema, install hooks. Create a minimal proof of concept. This will include creating new plugins for Inmail - a body-email handler that will be able to create a new node by filling the body field of a node from email body. This period could require work on Inmail issues in order to get it into the state where it can be used well to integrate with Mailhandler.

[#W3] Authentication via "from" email

(08/06/2016 - 15/06/2016)

Support for authentication via “from” email address. This will be done by providing a new Analyzer plugin for Inmail. We will provide basic authentication by only checking “from” header of an email. Beside this option, there will be more secure authentication option where an email has digitally signed and verified. However, that option will be optional. We will provide a new field on User entity (public key). By processing incoming emails we will extract the email address from the “from” header field and try to match it with an existing Drupal user. In case there is a user with this email and “require crypto sign” option was selected we will try to verify the the signature with the corresponding public key. This will be done by relying on GnuPGP PHP extension.

[#W4] Create a node by sending an email

(08/06/2016 - 15/06/2016)

Support node creation from an incoming email. A new Inmail analyzer plugin for nodes should analyze the content of an incoming email message and create a node in case identified user (sender) has enough permissions to do it. A node created via email will be logged. Extend proof-of-concept plugin built in first weeks. Introduce an Inmail MailFooter plugin that should process the body of an email. Footer plugin should support limited number of footer variations. The basic cases that are going to be supported are including a regex match for "-- \n" (RFC3676) cutoff line and "On ... wrote" part which is de-facto standard within Gmail. MailFooter plugin will separate the email into real mail body and footer parts. From there, we can remove unnecessary signature/footer and the rest with a body field of a node. Subject of an email will be used as a node title. Subject must be provided in: “[node][node_type] Node title” format in order to map the node type we creating a node for.

[#W5] Validate parsed content

(15/06/2016 - 22/06/2016)

Implement validation for parsed content after node analyzer. Use node entity validation, validate node type, fix bugs, refactor the code and prepare the project for mid term submission.

[#W6] Mid term submission

(20/06/2016 - 27/06/2016)

Final preparation of the project for mid-term submission. At this point, Mailhandler port should provide a D8 module that integrates with Inmail, can authenticate a user via “from” email address. A recognized authenticated user will be able to create valid nodes via email.

[#W7] Test coverage

(06/07/2016 - 13/07/2016)

In this week I will work on adding/extending SimpleTest/Kernel/Unit test coverage for MailFooter, node create and authentication plugins.

[#W8] Create comments by sending an email

(13/07/2016 - 20/07/2016)

Add support to post comments via email. We will process incoming emails and look for node references in subject of an email ([comment][#node_id]). Based on the “from” email authentication we will post a comment, in case a sender (user) has required permissions. Log comment attempts made via email. Implement a new Inmail plugin to support creation of basic Comment entities.

[#W9] Demo module

(20/07/2016 - 27/07/2016)

Create a demo module to present all the implemented features. We will create a mailhandler_demo module that integrates mailhandler, inmail and displays the preconfigured system with demo content.

[#W10] UI/UX/DX improvements

(27/07/2016 - 03/08/2016)

UI/UX improvements (User-interface changes, validation, user experience/Drupal experience and usability improvements). I will work on improving mailhandler user-interface and analyzer settings.

[#W10] Bug fixes

(03/08/2016 - 10/08/2016)

Refactoring, bug and test fixes (Optimization of the code, fixing bugs discovered using demo module, test fixes)

[#W11] Documentation

(10/08/2016 - 17/08/2016)

Writing documentation. ReadMe, Drupal.org module page updates, API documentation, improving code comments.

[#W12][#W13] Wrap up

(17/08/2016 - 23/08/2016)

Wrap up and Final submission,

mbovan’s picture

mbovan commented
Related issues: +#2731519: Roadmap - Port Mailhandler to Drupal 8

Attaching a related issue #2731519: Roadmap - Port Mailhandler to Drupal 8