It was suggested that we separate the mail backend from the front-end. We might want to include the mail backend in core as includes/mail.inc. Then, other modules like massmailer, subscriptions, notify, pm (private messages), project (project issues), contact, etc. could reuse this component. The mail.inc file would implement some kind of mail queue functionality, and modules just add a mail to the mail queue using a simple mail API. In future, the mail backend could deal with bounces and report back proper status codes. Moreover, mail.inc would be pluggable so it could be replaced by a more powerful one, or one that is build specifically for the underlying mail transport (SMTP server).

Furthermore, this would solve issues with how many mails get send within a specified interval. Like, when more than one module sends out e-mails it is hard to enforce limitations/restrictions imposed by the hosting company.

I'm posting this here so you can keep this in mind when working on simplenews, and to solicit for feedback. Chances are we'll setup an IRC meeting to discuss this further. Details to follow.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Amazon’s picture

A good place to start would be to look at existing Mail API's

Pear Mail: http://pear.php.net/package/Mail
Pear Mail tutorial: http://www.zend.com/pear/tutorials/Mail.php#Heading2 <--probably old
Java Mail API: http://java.sun.com/products/javamail/
We should also support the US SPAM act(not the right name) that has some authenication requirements for sending mass mail.

If anybody would like to volunteer to study these APIs please let me know.

walkah’s picture

um. +1 and then some. I think this is good... and some good references from Amazon. I'll throw in one other :

http://phpmailer.sf.net/ - has great mime handling (i've run into issues with PEAR::Mail's mime handilng in the past, but that was around 2 years ago, and things may have gotten better since then).

Oh - and you don't mention it - but support for multipart / mime messages would be lovely :)

I'd be interested in / happy to sit in on any IRC discussion as well.

robertDouglass’s picture

phpmailer++

I've used it and like how easy it is to send both text and html mail. Did I say html mail? You bet! It's time that we had the option.

cyberchucktx’s picture

I'm definitely interested.

Hopefully by adding a post to this I'll get notified on new postings (?)

If not I may miss the IRC ..

I've been doing a lot of work with safe_mode PHPLIST (have posted to
the drupal site somewhee, can dig out the reference if anyone is
interested).

Charlie in TX

Dries’s picture

It was suggested that mail.inc also provides a HTML to text services. Lots of modules (newsletter, notify, project) try converting HTML to text. Having a reusable function makes sense, and will lead to better conversion routines. XSLT anyone?

Owen Barton’s picture

+1 for phpmailer

I found it very easy to use - and most importantly for me - it supports SMTP authentication. As many ISP's will not let you send mail without using SMTP-auth nowadays it may be worth this feature is incorporated into the mail.inc API.

- Grug

Robert Castelo’s picture

For about a year now I've been working on an email newsletter and announcement module. There's a version in my sandbox commited about 2 months ago which works fairly well. Dan Robinson and Varr Willis at CivicActions, Moshe Weitzman plus a few others have been testing it, and I've had lots of positive feedback and feature and bug reports from them.

A few months ago a realised that there's not much point in putting so much effort into creating all this functionality, only for it to be locked away in my module. Better to

Better to create discreet component modules which provide a particular service, such as bounced email handling, but which can be used by any other module that also needs that service.

For the last two months I've been working to split functionality into component services modules and make these services available to all modules.

One of the biggest challenges was to make these component modules independent of each other. The only area where I haven't managed this is some of the database calls - but thanks to a chat with chx I realised that db_rewrite_sql could be used to handle this very nicely.

This is what I have:

  • bounced_email.module - process bounced emails
  • html2text.module - convert HTML to plain text equivalent (e.g. list item becomes "* item")
  • identity_hash.module - manages full and partial loggin based on hash which can be used in email links
  • publication.module - defines and packages content of publication, which could be any kind of publication
  • schedule.module - defines and manages schedules, e.g. email sending schedule
  • subscribed.module - manages subscriptions to publications
  • templates.module - manage and define templates

What's great is that the component modules are not limited to email, they could be used to quickly create RSS newsletters, PDF newsletters, text message newsletters, or even personalised/filtered website sections.

I haven't made the new component modules available anywhere yet, but I'll be happy to upload them to contrib and let others get involved.

What would be the best way to commit them - as a single directory? or each component module in it's own directory?

vwX’s picture

Status: Active » Needs review
FileSize
3.71 KB

This is a very basic mail queue system I would like to contribute. There are some changes to other modules that will have to be made such as changing user.module to use this instead of directly calling the php mail function and moving mime_header_encoding to mail.inc.

killes@www.drop.org’s picture

Project: Simplenews » Drupal core
Version: 7.x-1.x-dev » x.y.z
Component: Code » base system

Note: I am moving this to the main project.

VwX: What I see looks very nice for a start. Some polishing is certainly needed, but I'd like to try this. If I only had the time...

killes@www.drop.org’s picture

Ok, I've been doing some clean-up on this one and made contact.module use it. The module and the .inc file work.

I've put the files into my sandbox for now. I plan to work on them over the weekend (unless Dries tells me they are too late for 4.7).

http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/killes/mail/

But I could use some input:

- the sql file defines a column to store the module that generated the mail. Do we want this? If yes, we need to fix it.

- Do we need a throttle for only sending x mails per cron run?

- can somebody try the files with php5?

- what else do we need?

Please give input, links and what not.

Dries’s picture

Some of my comments might be false. If so, sorry. I only looked at it for 5 minutes:

  • I wonder why we need per-module settings. I can't think of concrete use cases.
  • For consistency, mailid needs to be mid.
  • I don't see an option to send mails immediately?
  • I don't see any "messages per hour" accounting?
  • I don't understand why mail.inc should be node system aware?
  • mailid = '%s' should be mailid = %d.
  • Incorrect use of t(): don't translate user input.
  • The coding style needs work; inconsistent variable names, inconsistent spacing, etc.
  • Some PHPDoc is rather cryptic: Updates table mail queue address table sets sent flag to 1 for a queued msg..
killes@www.drop.org’s picture

I've fixed a number of things. needs testing.

killes@www.drop.org’s picture

Ok, changed and tested.

Some more changes, taken some concepts from massmailer.module.
Included hacked copies of simplenews and contact module for testing purposes.

We need to discuss if this should be a module or a part od system.module.

depending on the outcome we need to put the functions in mail.module and mail.inc into different files.

Testers welcome.

IamPter’s picture

Dries,

Are you talking about something like I request here http://drupal.org/node/28079 ?

m3avrck’s picture

Well first off, just wanted to say I'm currently evaluating this and been sending regular feedback to killes for improvements. So far so good on PHP5 machines :-)

But in terms of where should this code go, I think mail.module should be eliminated and those functions should be put in mail.inc which *should* be seperate from common.inc. This way, as Drupal grows and more and more modules depend upon mail functions, they will be in their unqiue spot which can easily be patched and extended instead of rummaging through the common.inc mess ;-)

As a result, the mail.module settings should be moved to Admin > Settings with the mail setup there, since the basic Drupal "request new password" etc framework requires mail functionality and having these settings there makes the most sense, usability wise.

I think with this setup the code could be organized in the cleanest and most straightforward method and would be an elegant refinement in mail handling for 4.7... no new features, just cleaned up and seperate for other modules to make use of, instead of duplicating code and time. This also has the benefit of bringing in more patches to the Drupal core and how it handles mail and making it rock solid.

Just my 2 cents.

vwX’s picture

Did some testing on Killes changes. Queued ~1000 addresses for one email. When sending started receiving this message :

warning: mail(): Could not execute mail delivery program '/usr/sbin/sendmail -t -i ' in /home/webuser/mars/www/drupal/modules/mail.module on line 189.

at around message 258.

Looking further seems sendmail got hammered. This is a RH6.2 box 550MHz processor with 256MB memory and sendmail configured to use a smarthost qmail server. I can share the script to test if anybody wants to help. Will try a bit more after I do some tuning.

Now about the changes.

I think that the mail_queue_addresses table key structure needs to be changed. Remove Aid and just use address. Also, I've always used a primary key structure of parent table primary key + one or more additional keys for a child table. Originally I had something like this, but it is now one primary key and an index. I don't think this is correct and I've been doing database work for 15 odd years.

Also, I agree with what m3avrck says. Orginally when I submitted the code I had most of the functionality in mail.inc and the module was primarily for administrative functions. I also functionality to view what was in the queue and force the sending of messages in the queue. That has since been removed and I think it should be re-added.

noid’s picture

Sorry guys for interrupting your discussion, but just wanted to make sure that batch processing ( that is, to help guys like me whose host limits the sending of emails to say, 200 per hour) is still not implemented in Simplenews? From my shallow understanding this is what you are working on right now? Raised this issue which Dries marked as duplicate here: http://drupal.org/node/28311 .

Just wanted to make sure that I didn't miss the release of this feature. Thanks in advance! :)

killes@www.drop.org’s picture

Assigned: Unassigned » killes@www.drop.org

@vwx: How did you send the mails? direcly or through the queue? can you try to queue the messages and use the throttle setting?

Also, I believe we do neeed aid. There might be the same address twice in the same batch of mails.

How the code should be split up depends on Dries, I guess. Personally I'd put the module functions into system.module and the rest into a small .inc that is lazily loaded like the .incs for xmlrpc. I don't think we need a way to look at the queue in core. Do you eve look at sendmail's mailqueue? I don't.

@noid: this issue is not about simplenews, but we do have a throttle setting.

vwX’s picture

>@vwx: How did ...

Via the queue. This is on a 2.2 linux kernel so it will probably perform better on 2.6. I turned up the number of open files allowed and am now running into the 240 second processing time limit. Hitting that limit left one zombied sendmail process which restarting Apache cleared. Will do some testing on a 2.6 kernel soon.

>Also, I believe we do neeed aid...

Replace aid with address and make the primary key structure address,mid. This will prevent duplicates.

>How the code should be split up depends on Dries, I guess. Personally I'd put the module functions into system.module and the rest into a small .inc that is lazily loaded like the .incs for xmlrpc. I don't think we need a way to look at the queue in core. Do you eve look at sendmail's mailqueue? I don't.

If you run a heavily used mail server you do. Plus, drupal admins should be able to see what is going on without resorting to a sql cli. This is one of the reasons I orginally included which module queued the message so that in the future priority could be given to particular modules.

m3avrck’s picture

vwX, I already recommend to killes to make the key(mid,address), however, i later changed that to only aid. reason being, there are instances where the same address will be part of the same mid, for example, in the To: and CC: fields, so it doesn't make sense to have mid,address as the key. aid seems to be sufficient after thinking about it.

noid’s picture

Hi killes,

Assumed I had to post here because Dries pointed me to this thread (Dries marked my thread -- http://drupal.org/node/28311 -- a duplicate of this thread). I guess I have to get back to the first thread then. :)

killes@www.drop.org’s picture

@vwx, ok, so it is up to the user to tune the throttle setting to his computer's capabilities. AS I see it the worst thign that coul dhappen is that a mail gets send twice. Email loss should not happen. can you confirm this?
Also, I think that the queue should not discard duplicates, the aid needs to stay.

I removed the queue monitor because it looked like something Dries wouldn't like in core and that could be added through a contrib module. Less is more.

vwX’s picture

@m3avrck, Why would you want duplicate addresses on an email. To:, CC: and BCC: are just syntatical sugar to generate RCPT TO: lines in the smtp conversation anyway. Injecting duplicates is just bad form.

@Killes, Didn't see any email lost when the timeout was hit, just a hung sendmail process and non-of the sent emails got their status updated. I imagine that this could be self tuning by every now and then updating the status on what has been sent instead of batching it to the very end (which wasn't reached). I think it would be bad if we just let this hammer a shared server environment by requiring the user to do the tuning, might get some isp admins upset and ban Drupal from their hosting. Wouldn't want that to happen.

killes@www.drop.org’s picture

@VwX: I agree with you, this need to be made more secure.

As I see it there are two options:

1) Limit the throttle choices to a lower maximum that we think will be possible even on shared hosting (25 mails?). People knowing what they are doing could still override this in settings.php.

2) Break up the sending and marking as sent process into smaller chunks, say 25 mails each. (too high?)

killes@www.drop.org’s picture

Ok, I've made two small changes to mail.inc:

1) stop sending mail if an error occurs. This will take care of not overloading sendmail, I hope.

2) update the "mailed" status every 25 mails. this should help if Drupal times out during mailing.

Dries’s picture

Why not update the mailed status every mail? Wouldn't that reduce the error rate?

Dries’s picture

1. I don't understand the $can_send logic in mail_queue_send(). Why don't we use the database to keep track of how many mails we sent in the last hour?

2. Why do we have helper functions like _mail_queue_clean() when they are only called from one location (here, mail_queue_clean()).

3. We can probably remove this helper function:

function _mail_queue_send($mail) {
  return mail_mailengine('send as is', $mail);
}

4. The mail engine hook is a bit ... weird. I think I prefer the image toolkit approach.

5. mail_send() doesn't do any accounting?

6. Why are mail_send() and mail_queue() part of mail.module, and not part of mail.inc? Why does mail.module exist to begin with?

7. You can win an award with this:

    default: // method not implemented.
      return FALSE;
      break;

or this

      return $status;
      break;

;-)

killes@www.drop.org’s picture

Dries, thanks for reviewing.

Yes, updating evey send mail would decrease the error rate to may 1. I've changed it this way (I still think we could do with less queries).

1) Using the database would mean to keep some stats about mail sending in another db table. I prefer the current logic. It first computers the number of hours passed since the last sending and then multiplies by the allowed hourly amount of mails. i've shuffeld the code a bit to make that clearer.

2) removed

3) removed

4) You prefer scanning through directories to search for matching files? I don't like it. Maybe explain a bit.

5) mail_send doesn't do accounting. it is is just a replacement for user_mail(). What accounting would you like it to do?

6) mail.module does exist because I don't know if you woud like the implementation of the hook that comes with Drupal to reside in its own module or say in system.module. it is also easier for testing. Where the individual functione belong needs discussing. mail_send is a replacement for user_mail() and should probably be available all the time. mail_queue() is part of an implementation of the mailengine hook and should be in a module or a module's .inc file.

7) the default test is actually intentional. I could then test for the existence of a specific tcase inside the switch. Maybe I need a more specific return value. :)

Dries’s picture

mail_send doesn't do accounting. it is is just a replacement for user_mail(). What accounting would you like it to do?

I'd think that mail_send should only send the e-mail now if the mail quota hasn't been exceeded?

I think mail.inc should also have a html2text function to convert HTML content to something that can be used for sending plain-text e-mails. Ideally that function would be called xml2text and use XML technology that is reliable. In the mean time, simeplenews's conversion regex could do the job. This code is probably duplicated in the project module, the various subscription modules, and what not.

Robert Castelo’s picture

There is a library/component module in contrib which I wrote for this specific purpose. It converts HTML to it's plain text equivalent, by equivalent I mean for example * is added to list items.

It also annotates linked text and collects the links for inclusion at the end of the text.
Leaves HTML which is between < code > < /code >.
Wrap text option.

http://drupal.org/project/html2txt

This code is probably duplicated in the project module, the various subscription modules, and what not.

Yep, that's why I converted this into a library/component, now they can all leverage this code.

killes@www.drop.org’s picture

The accounting is only applied to the queued mail, not individually sent mail. I think that password reminders and similar mails should still at least attempt to get sent without interference of some quota. It is up to the calling module to log an error or issue an error message. it could still queue the mail when direct sending fails and the quota would apply then.

I've copied the utf8html2txt functions from simplenews.module. the module itself borrowed them from html2txt. :p

shouchen’s picture

Will the new mail backend support embedded images as discussed here? http://drupal.org/node/28908

killes@www.drop.org’s picture

We will probably include functions that will allow you to do this.

killes@www.drop.org’s picture

the html2txt conversion includes strtoupper and ucfirst in a number of places. I'd like to remove those. Objections?

Stefan Nagtegaal’s picture

Well, I'm not sure but isn't it better to replace them by their drupal functions drupal_ucfirst() and drupal_strtoupper()??
(See unicode.inc)

killes@www.drop.org’s picture

Yes, we should do that in case we decide to keep them. I don't like to mess with the author's chosen way of displaying content, though.

Stefan Nagtegaal’s picture

Indeed, I think it's better to get rid of them...

Robert Castelo’s picture

Presenting plain text as a heading requires conventions like all uppercase or first letter of each word uppercase, or underline, or underline and overline.

Idealy these should all be optional and the module that calls mail.inc could specify how headings are represented, based on admin preferences - with mail.inc doing the actual conversion.

killes@www.drop.org’s picture

The formatting function is not called from anywhere inside the module, it is just provided as a library function. So any module can chose to not call that particular function.

I'd rather underline a heading than changing its words in any way.

killes@www.drop.org’s picture

Just a short update: I committed a version that allows mime message composition. That means that you can send attachements. See mail-utf8.php for example.

killes@www.drop.org’s picture

sending alternative html mail works now, see mail-alernative.php script.

killes@www.drop.org’s picture

We should investigate how we shoudl deal with files that are included from html pages that get attached. Maybe a mail_extract_file_links function?

killes@www.drop.org’s picture

FileSize
17.06 KB

Here is a patch against Drupal core.

killes@www.drop.org’s picture

FileSize
18.67 KB

Here is the mail.inc file that goes into /includes

killes@www.drop.org’s picture

the mysql file is this the mail_queue.mysql file in my sandbox. I will roll a patch that includes database definitions later.

killes@www.drop.org’s picture

FileSize
18.61 KB

The mail.inc had a leftover include line.

killes@www.drop.org’s picture

FileSize
18.61 KB

heres the same with a .txt extension to get around the silly code colouring. Can we switch that off?

scroogie’s picture

Without reading the whole thread, i just wanted to tell you that there are PEAR projects for

- a general Mail Class
- a class for MIME encoding and decoding, including inline Images and attachments
- a Class for a general Mail Queue

which you could easily use. I dont know if you even care about it, but i thought maybe it can save you some time.

killes@www.drop.org’s picture

FileSize
17.06 KB

The patch had two typos.

killes@www.drop.org’s picture

scroogie: thanks, this is known to me. However, we try to stay away from PEAR for a couple of reasons. One being that it usually does too much. However, with the approach taken in my patch, you can easily write a mail backend which uses these classes.

killes@www.drop.org’s picture

The patch has been tested with both user.module (password reminders) and contact module (both private and group contacts). I am fairly confident that it does not break anything, but adds a lot of functionality which can be exploited by contributed modules.

killes@www.drop.org’s picture

FileSize
1.31 KB

Here is a (rather crude) patch to simplenews.module that makes it use the new system.

killes@www.drop.org’s picture

FileSize
2.38 KB

I hadn't noticed that simplenews hasnt been converted to forms API yet.

Here is a patch to contact module that will make it send messages through the queue if you use the "group contact" form.

To use the patch get a clean copy of contact module from cvs and apply it. It should probably not be applied to core.

Stefan Nagtegaal’s picture

Very nice patch Gerhard!
I did test this together with Gerhard a couple of hours ago and couldn't find anymore things than I reported to him immediatly..

I think it's to soon to set this 'ready to commit' after 1 review/test, but this will bring drupal once again to a higher level!

Really great job Gerhard!

thank you..

killes@www.drop.org’s picture

FileSize
18.3 KB

testing finds bugs, bugs lead to simplifications... New mail.inc.

Gerhard Killesreiter’s picture

FileSize
5.04 KB

Here is the database part of the patch.

killes@www.drop.org’s picture

To help with testing:

You can test the basic functionality as follows:

1) apply the patch from

http://drupal.org/node/28604#comment-52129

2) copy the file from

http://drupal.org/node/28604#comment-52151

to includes/mail.inc

3) apply the patch from
http://drupal.org/files/issues/mail_db.patch

and run update.php

Your Drupal install does now send mails through the new mail API. You can make some settings on admin/settings/mail and send mails through user.module and contact.module.

There is by default no "massmailing" functionality in Drupal core, To test it you can proceed as follows:

4) remove the patched contact module

5) get a clean copy

6) apply the patch from

http://drupal.org/node/28604#comment-52140

7) set up a group mail form at

admin/contact

Now any mail you send through the group contact form at /contact will be send through the mail queue. To get the mails actually sent you will need to trigger cron.php.

18:53 < killes> 2) rm contact.module
18:53 < killes> 3) get new copy
18:53 < killes> 4) apply 18:53 < killes> http://drupal.org/files/issues/contact_0.patch <== that patc

sepeck’s picture

running update.php after patching I get
Fatal error: Incorrect table definition; there can be only one auto column and it must be defined as a key query: ALTER TABLE contact DROP PRIMARY KEY in D:\Inetpub\kali\includes\database.mysql.inc on line 108

I shall try again later, I may have followed the instructions incorrectly

Sergio Beristain’s picture

Title: Separate mail backend » Throttle - yet allow immediate send

For more info see: http://drupal.org/node/37958.
(to discuss on the issue use the forum)

Sergio Beristain’s picture

Sergio Beristain’s picture

Title: Throttle - yet allow immediate send » Separate mail backend
killes@www.drop.org’s picture

FileSize
19.82 KB

Here is an updated patch for the modules and the database update.

moshe weitzman’s picture

so whats happenned with this patch? is send.module a replacement for this? this was a priority at one time.

Gerhard Killesreiter’s picture

This is somewhat on the back burner since Dries didn't want it for 4.7. I will revive it after 4.8 development opens up.

robertDouglass’s picture

Well, 4.8 is now opened up and I'm trying to look into the future so that I can start using whatever code might make it into 4.8. As I see it, either this pach or the mimemail module are the likely candidates. Have sentiments about Gerhard's work changed any in the last months?

By the way, this functionality is one of the most critically lacking areas in 4.7 and I really look forward to resolution on the issue. Thanks Gerhard for the great work so far.

robertDouglass’s picture

FileSize
18.44 KB

I've re-rolled the patch against HEAD. I've not done any testing yet.... starting that now.

robertDouglass’s picture

@Gerhard: I' curious as to why the following is part of the system_mailengine and not a helper function in mail.inc:

      // We make sure the body contains the correct CRLF sequence
      $message['body'] = str_replace("\r\r\n", "\r\n", str_replace("\n", "\r\n", $message['body']));

It isn't much code, but it seems like it will get copied repeatedly for every mailengine hook that gets written. It also seems like making the CRLF a variable that can get set in settings.php, for example, would be better than burying the "Note: if you are having problems..." in code comments.

killes@www.drop.org’s picture

Robert: I agree with both ideas.

Dries’s picture

* Maybe 'send as is' should be called 'raw'?

* Why is 'single' and 'multiple' fundamentally different? I'm not sure that complexity should be exposed. I can't think of a good reason and would suggest that the underlying mail engine implementation hide that? In such model there would only be a 'send raw' and 'send formated' or something. If multiple recipients are provided, the underlying implementation figures out how to deal with that.

* The 'single mail' solution doesn't seem to do any accounting, yet there seems to be a throttle setting? Shouldn't we pipe all mails through mail_queue() so there is a single point of accounting?

* For the mail_queue and accounting functionality to work, you need cron.php. Cron is not a requirement at the moment. Tricky problem.

* I think it is easier to define function _mail_define_search() and mail_define_replace() as a single array:

   array('pattern' => 'replacement',
            'pattern' => 'replacement');

That would significantly improve readability.

* url should be URL, html should be HTML, uri should be URI, etc.

* Some of the helper functions are poorly documented. There help read 'helper function for' without any explanation as what they do.

robertDouglass’s picture

I think there is a bug in _mail_file. Because $new_file is pushed into the array without a key, the files accumulate over the lifetime of a request if more than one mail is sent (so that the second mail gets its files plus the first mail's files).

function _mail_file($file = NULL, $type = '') {
  static $files = array();
  [...]
  $files[] = $new_file;
  [...]
  if ($file) {
    return $file;
  }
  else {
    return $files;
  }
}
robertDouglass’s picture

Status: Needs review » Needs work

I had 720 unsent mails in the queue and the throttle was set on 100. I ran cron and all 720 mails got sent.

Allie Micka’s picture

I see no compelling reason to include the mailqueue stuff in core, when it can easily be added to this code as a mailengine backend in contrib. Most sites don't need this functionality at all, so leaving it out by default will be lightweight, less to support, and more flexible.

mimemail has the same roots as this module, and a lot of work has been done to fix bugs, improve the API, fix mail headers, and clean up HTML handling. If there are no objections, I'll roll a separate patch based on that work.

robertDouglass’s picture

Allie, I'll gladly test your patch =)

Note that I think your _mimemail_file function is prone to the same error as the one I mentioned above wrt static caching. This fixes it... though you'll probably think of something more elegant:

  if ($file) {
    return $file;
  }
  // reset the results
  $result = $files;
  $files = array();
  
  return $result;
moshe weitzman’s picture

Lets get this one in, folks. Any progress, Allie? If not, we will proceed with whats been posted here. Maybe Robert or killes have made some progress or fix the stated bug in it?

Sergio Beristain’s picture

Hi guys,
Another problem is that mail() uses automatically the localhost, without allowing Drupal administrator to chose a different server.
I wonder if it would be possible to add the functionality of customising the outgoing.server and account in which the mails are sent?
This can be done by simply ini_set("SMTP_SERVER") and : SMTP_USERNAME, SMTP_PASSWORD, SMTP_PORT.

This is important for some of us who need to send mails from different outgoing mail servers.

Gábor Hojtsy’s picture

Note that due to user_mail() changed to drupal_mail(), with some similar changes suggested here and then some more, this patch needs to be rerolled.

Also note that it is completely possible to implement this mail queue system in contrib, since Drupal still provides the possibility to set a wrapper to get called instead of the PHP mail() function. And that wrapper can queue, handle quotas and such.

sun’s picture

Version: x.y.z » 6.x-dev

Subscribing. +1 for having this in core.

smk-ka’s picture

Lucky me! I've just found this virgin thread and would like to share some thoughts on this issue, before the damage is done ;)

Anyway, the case: I've recently hacked the buddylist module to allow (read: prefer) sending its requests via privatemessage, instead of emails, which seems the preferred way for community sites. So, my instant thought then was: couldn't this be useful for a number of modules?

I think of an API with open ends on both sides, input as well as output. It would receive a message, process it (queueing, etc) and pass it on to one (of multiple) delivery layers, which actually send the mesage.

In practice, this would mean that we'd need the receiving layer to support not only one delivery engine (modules), but a number of them instead (think of SMS gateways etc.). While there could be global preference which delivery module to prefer (weights), each module that uses this API still needs to have control over the way a specific message is actually sent (some messages simply *must* be sent via email, think of legal issues for example). In other cases, the calling module trusts in the global preference, or even better, each module would have its own preference setting about which delivery engine to be used.

What do you think about such a "double layer" solution?
--
Stefan Kudwien
www.unleashedmind.com

moshe weitzman’s picture

subscription.module (http://drupal.org/project/subscription) has abstractions like what you describe and IMO its logic is very hard to follow. At some point, your abstractions start hurting more than helping. Just something to be aware about. Anyway, this dual layer you discuss is not directly relevant here. We are specifically talking about queueing email. Lets maintain focus.

merlinofchaos’s picture

Subscribing. We need something like this in Drupal 6; the lack of decent mail handling in Drupal is a weakness we need to address.

Owen Barton’s picture

Mimemail has been updated for D5, so I think is still well worth a look for turning into a core mail.inc

psi-borg’s picture

i havent seen any working updates for bounce handling - many users here may not have php_imap compiled into php and require a socket class to connect to the bouncer box. i've got a good working socket class that i use for another php mailing program that can be used for, say, simplenews bounce handling.

my main problem is i'm a complete noob to drupal and don't know how to even get a bounce_count field in the simplenews_subscriptions table to render from a call to simplenews.module ...the only solution i have is a hack and it's ugly (but it works).

if someone wants to make a dummy's api for simplenews.module - i'll hook the socket class in.

dualdiesel’s picture

Title: Separate mail backend » Separate mail backend w/ PHPlist?

I'm looking for a solution for my band's site. I'd lke to have a free standing email subscriptions list on pages like myspace, sonicbids and other sites we use (indepentent from our drupal site) The most important part of this mail subscriber would be that we can collect the subscriber's zip code. This will allow us to mail subscribers in specific cities when we have events in their area.

PHPlist has the ability to do this, but there doesn't seem to be any solution for integrating it in a drupal 5.0 site. Massmailer was the way to go in eariler versions of drupal, but i'd like to move forward. If anyone knows of any solutions to look into, please share.

Also if anyone else is interested in this soultion and is willing to chip in please message me.

NOTE TO DEVELOPERS: There are at least 2 other people interested in this project and willing to chip in. Please contact me if you are interested in providing this solution and we will work out specifics.

ciao
-Mec
mj[at]meczilla[dot]com

Owen Barton’s picture

Title: Separate mail backend w/ PHPlist? » Separate mail backend

Please don't change the title :)

This ticket is not about integration - just providing hooks for contrib modules and perhaps a very simple queue system in core.

If you need this integration now check out CiviCRM + CiviMail - it does just what you describe (although requires some knowledge to set up)

killes@www.drop.org’s picture

I am still interested in working on this if we reach a consensus on what shoudl eb in core and what not. I tend to believe that html->txt conversion should not.

Allie Micka’s picture

The mimemail module that resulted from all of this basically does two things:

1) accepts text or rendered HTML and generates a valid, themeable, multipart/alternative message. It gives users and administrators the ability to set messages to plain text only. Yes, it generates plain text for this.

2) Delivers messages through any administratively-selected module that implements hook_mailengine. This module could be a queueing engine, a "don't deliver this message right now" engine, anything. By default, it uses its own, which simply calls mail().

There's additional functionality cited in this issue for implementing a whole queuing system. This should not be in core, but rather implemented in a separate module.

I have mixed feelings about putting #1 in core. On one hand, it's potential bloat. Let me just say up front, I don't want any part of the "html or plaintext" holy war.

It would be cool and simple to just throw fully-rendered nodes at drupal_mail() instead of having each module under the sun try to do the html-to-text conversion on its own. You can also control plaintext preference-setting and themeing for everything from one spot.

More importantly, If modules never have the option to send HTML to drupal_mail, they must first convert to plaintext on their own. And then, if a mime mail module decides to format it as HTML, it re-converts it (which, naturally, won't look right after the double-translation).

Now, having #2 in core is a no-brainer. And it would get rid of that frightening smtp_library apparition. I'll begin this part, and leave #1 open for discussion.

Soooo... talk amongst yourselves!

Allie Micka’s picture

The mimemail module that resulted from all of this basically does two things:

1) accepts text or rendered HTML and generates a valid, themeable, multipart/alternative message. It gives users and administrators the ability to set messages to plain text only. Yes, it generates plain text for this.

2) Delivers messages through any administratively-selected module that implements hook_mailengine. This module could be a queueing engine, a "don't deliver this message right now" engine, anything. By default, it uses its own, which simply calls mail().

There's additional functionality cited in this issue for implementing a whole queuing system. This should not be in core, but rather implemented in a separate module.

I have mixed feelings about putting #1 in core. On one hand, it's potential bloat. Let me just say up front, I don't want any part of the "html or plaintext" holy war.

It would be cool and simple to just throw fully-rendered nodes at drupal_mail() instead of having each module under the sun try to do the html-to-text conversion on its own. You can also control plaintext preference-setting and themeing for everything from one spot.

More importantly, If modules never have the option to send HTML to drupal_mail, they must first convert to plaintext on their own. And then, if a mime mail module decides to format it as HTML, it re-converts it (which, naturally, won't look right after the double-translation).

Now, having #2 in core is a no-brainer. And it would get rid of that frightening smtp_library apparition. I'll begin this part, and leave #1 open for discussion.

Soooo... talk amongst yourselves!

Allie Micka’s picture

The mimemail module that resulted from all of this basically does two things:

1) accepts text or rendered HTML and generates a valid, themeable, multipart/alternative message. It gives users and administrators the ability to set messages to plain text only. Yes, it generates plain text for this.

2) Delivers messages through any administratively-selected module that implements hook_mailengine. This module could be a queueing engine, a "don't deliver this message right now" engine, anything. By default, it uses its own, which simply calls mail().

There's additional functionality cited in this issue for implementing a whole queuing system. This should not be in core, but rather implemented in a separate module.

I have mixed feelings about putting #1 in core. On one hand, it's potential bloat. Let me just say up front, I don't want any part of the "html or plaintext" holy war.

It would be cool and simple to just throw fully-rendered nodes at drupal_mail() instead of having each module under the sun try to do the html-to-text conversion on its own. You can also control plaintext preference-setting and themeing for everything from one spot.

More importantly, If modules never have the option to send HTML to drupal_mail, they must first convert to plaintext on their own. And then, if a mime mail module decides to format it as HTML, it re-converts it (which, naturally, won't look right after the double-translation).

Now, having #2 in core is a no-brainer. And it would get rid of that frightening smtp_library apparition. I'll begin this part, and leave #1 open for discussion.

Soooo... talk amongst yourselves!

Allie Micka’s picture

Assigned: killes@www.drop.org » Allie Micka
Status: Needs work » Needs review
FileSize
8.68 KB

I'll try to keep my posts to 1 this time ;)

This patch implements hook_mailengine instead of the smtp_library call. This permits us to implement contrib backends that handling queuing and digests, or prevent delivery entirely (e.g. for testing)

The drupal_mail() API has not changed, except for the fact that you can use user objects or arrays for the from/to address.

The API for hook_mail_alter has changed. It's now passing an array ( #mail_id, #to, #from, #subject, #body, #headers) instead of named parameters. This is a byproduct of #110888 , and also permits a contrib backend to set additional array elements in $message so that it can access them at delivery time.

moshe weitzman’s picture

Status: Needs review » Needs work

This looks small and clean. Quite analogous to the dblog patch.

Alas, one large hunk fails. Please please reroll.

m3avrck’s picture

Subscribing, hopefully we can get this into D6 since right now you have to jump through hoops to properly send HTML encoded emails...

drewish’s picture

subscribing. i'd really like to see some HTML -> plain text functionality rolled into this too...

drewish’s picture

Status: Needs work » Needs review
FileSize
7.16 KB

I fixed a few small bugs and pulled out Allie Micka's drupal_mail_address() function which changed the API and, IMHO, complicated things.

I still think we should still add in a mail.inc file that has functions for converting HTML to plain text. I'll work on that when I get some time.

moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
7.83 KB

i tested this and mail was sent as expected. this is a small but important change, like pluggagle watchdog and pluggable cache.

rerolled to remove offset. no other changes.

moshe weitzman’s picture

FileSize
7.35 KB

put one emoty line back in.

Dries’s picture

Status: Reviewed & tested by the community » Needs work

Now we have the form API, the mail engine settings hook might no longer be needed -- we can use FAPI for that (and do much more). I'd like to discuss this some more because there might be room for a small but subtle simplification. :)

moshe weitzman’s picture

Thanks for the review ... I don't yet undestand how fapi helps us here. Please elaborate.

Owen Barton’s picture

@moshe: what I believe Dries is getting at (and correct me if I am wrong here!) is that engines should be able to use hook_form_alter to create (if necessary) and add themselves to $form['mail_engine'].

If this is correct I am not sure I fully agree - I think a 'pull' model is preferable here, since contrib modules (especially ones that send lots of mail!) might want to present the user a mail engine setting for that module only. This would be difficult if we rely on the mail engines altering a specific form.

moshe weitzman’s picture

@Dries - is that what you meant. Seems like you might have meant something more substantial than that . not sure.

moshe weitzman’s picture

Status: Needs work » Needs review

OK, I have simplified as per Dries' fine comment. The admin pref page and the whole mailengine hook are simplified. No more operations for name, description, or settings. those are provided via form_alter by a mailengine module.

moshe weitzman’s picture

FileSize
6.65 KB

with patch

moshe weitzman’s picture

Status: Needs review » Needs work

actually, we could reduce my patch further and just not show any admin pref by default, and let mailengine modules do a variable_set upon install so they are the new mailengine. no UI needed.

once we do that though, we are basically at the functionality we have now with swappable SMTP libraries so i'm questionning the whole need for this patch. anyway, maybe someone else can carry the torch from here.

catch’s picture

Version: 6.x-dev » 7.x-dev

Bumping to 7.x, but looks like a potential won't fix.

preventingchaos’s picture

Subscribing. I'm interested in what will happen with this and how it will affect the future of the SMTP Auth module and my upcoming Mail API module that I've been working on.

einsicht’s picture

I'd like this feature to be in Drupal 7. Many companies want to be able to send out HTML formatted emails as newsletters. If nothing else, it doesn't have to be too fancy, just support rich text (bold, underline, etc), lists, headings, p's and div's, and colors (text+background).

I haven't tried myself, but I hear it's hard to turn webpage HTML into emails that render well in all clients (web and non-web based). If anyone is still working on this, maybe I could help. I'm okay with PHP, but don't know much about email formatting & co.

Allie Micka’s picture

Hi einsicht,

You can already send newsletters through a variety of contributed modules ( e.g. News, SimpleNews, and about 99 others ). You can also format multipart, themed, mime-encoded messages using the Mime Mail (many of the newsletter tools rely on that module).

Mime Mail can, based on sitewide and/or user preferences, convert any outgoing message into themed HTML format. I suspect this is a job best left to contrib, so things are OK on that front. However, unless you short-circuit drupal_mail, the resulting process is HTML -> Plaintext -> HTML, which is less than ideal because you lose the originally-intended formatting. If there's a way to address that problems, this ticket should stick to mailengines and Mime should be left to contrib.

RobLoach’s picture

I liked where Moshe was going with this for a complete Drupal mail system revamp. The mail system as it is right now is pretty dirty. You have to set the smtp_library variable when a module is installed, and then the smtp library variable file implements the drupal_mail_wrapper function. In my opinion, it's a pretty hacked solution and hook_mailengine is problematically a nicer solution. What happens when a user enables two different modules that implement the SMTP_library variable? We run into conflicting modules.... Exposing a user interface for this, as well as hook_mailengine will fix it.

RobLoach’s picture

Status: Needs work » Needs review
FileSize
7.39 KB

This is an update to what Moshe uploaded at #101 for HEAD. I also added $op == 'info' so that the admin/settings/mail form is cleaner......

/**
 * Implementation of hook_mailengine().
 * 
 * Delivers messages for drupal_mail() using PHP's mail(). More information in the
 * <a href="http://php.net/manual/en/function.mail.php">PHP function reference
 * for mail()</a>. 
 */
function system_mailengine($op, $message = NULL) {
  switch ($op) {
    case 'info':
      return t('Mail');
    break;
    case 'send':
      if (is_array($message)) {
        $mimeheaders = array();
        foreach ($message['headers'] as $name => $value) {
          $mimeheaders[] = $name . ': ' . mime_header_encode($value);
        }
        return mail(
          $message['to'],
          mime_header_encode($message['subject']),
          // Note: e-mail uses CRLF for line-endings, but PHP's API requires LF.
          // They will appear correctly in the actual e-mail that is sent.
          str_replace("\r", '', $message['body']),
          // For headers, PHP's API suggests that we use CRLF normally,
          // but some MTAs incorrecly replace LF with CRLF. See #234403.
          join("\n", $mimeheaders)
        );
      }
    break;
  }
}

Thoughts?

RobLoach’s picture

Status: Needs review » Closed (duplicate)

#331180: fix pluggable smtp/mail framework does this in a much cleaner way.

zolipe_shree’s picture

Title: Separate mail backend » Newsletter Bounce back mail log
Version: 7.x-dev » 5.7
Priority: Normal » Critical
Status: Closed (duplicate) » Closed (works as designed)

Kindly help me to view the log of the bounced mails which are going from newsletter module . I use drupal 5.7 - Simple News letter module .

sun’s picture

Title: Newsletter Bounce back mail log » Separate mail backend
Version: 5.7 » 7.x-dev
Priority: Critical » Normal
Status: Closed (works as designed) » Closed (duplicate)

Please don't hi-jack issues.