I have site with lots of subscribers to the newsletter, and when sending I get this error:

*Fatal error*: Allowed memory size of 33554432 bytes exhausted (tried to allocate 40 bytes)

I'm pretty sure this happening in simplenews_send_node. I notice that it selects everything from simplenews_subscriptions, then inserts one at a time into simplenews_mail_spool. It seems to me this could be done in one database query with an INSERT ... SELECT ...

Is there any reason it's not done that way? Interested in a patch if I can make that work?

Thanks.

CommentFileSizeAuthor
#3 simplenews_942238_spooladd.patch2.18 KBmiro_dietiker

Comments

miro_dietiker’s picture

This is in 6.x-2.x on my radar. I already added notes in code+issues that the current implementation doesn't scale fine and the code is pretty silly.

I'd be very happy if you address this. Note that the spool is currently nod cleanly designed and needs further adaptions.
If you would like to work on its cleanup, let me know.
(currently if a node is right before sending by cron and gets edited and resaved, the spool is being rebuilt - on every save. This is pretty critical for "late edits" that usually happens when admin are in panic - and they're in panic quite often with newsletters...)
The transition into "scheduled / waiting to be sent" is a one way transition which needs representing correctly fixed implementation.

So this would in addition fix some well known issue.

Thank you. :-)

Dave Cohen’s picture

Category: support » bug
Status: Active » Needs review

Here's the code I'm testing. Works on my development server. Haven't yet tested on the live server with thousands of subscribers. Would like you to review before I do that.


Index: simplenews.module                                                                                         
===================================================================
--- simplenews.module   (revision 2820)                                                                          
+++ simplenews.module   (working copy)                                                                           
@@ -1415,11 +1415,11 @@
                                                                                                                 
                                                                                                                 
     if (empty($accounts)) {                                                                                     
-      // No accounts specified. Get email address of all accounts subscribed to this newsletter.                
-      $result = db_query('SELECT s.mail FROM {simplenews_subscriptions} s INNER JOIN {simplenews_snid_tid} t ON s.snid = t.snid WHERE s.activated = %d AND t.tid = %d', 1, $node_data['tid']);                                  
-      while ($account = db_fetch_object($result)) {                                                             
-        $mails[] = array('mail' => $account->mail);                                                             
-      }                                                                                                         
+      // No accounts specified.  Write all active subscriber addresses to mail spool.                           
+      db_query("INSERT INTO {simplenews_mail_spool} (mail, nid, vid, tid, status, timestamp) SELECT s.mail, %d, %d, t.tid, %d, %d FROM {simplenews_subscriptions} s INNER JOIN {simplenews_snid_tid} t ON s.snid = t.snid WHERE s.activated = 1 AND t.tid = %d",                                                                                
+        $node->nid, $node->vid,                                                                                 
+        SIMPLENEWS_SPOOL_PENDING, time(),                                                                       
+        $node->simplenews['tid']);                                                                              
     }                                                                                                           
     else {                                                                                                      
       // Get email address of specified accounts.                                                               
miro_dietiker’s picture

Title: Fatal error Allowed memory size exhausted when sending to list » Directly populate spool in database, avoid php memory limit exhaust
Version: 6.x-1.3 » 6.x-2.x-dev
Status: Needs review » Fixed
StatusFileSize
new2.18 KB

Great, thank you!
Do you know which mysql versions support this INSERT ... SELECT?
I found elder mysql seems to support it...
http://dev.mysql.com/doc/refman/4.1/en/insert-select.html

(How about postgres?) :-) ;-)

If there's some limitation we might need the previous way as a workaround if e.g. the query fails somehow...

Note that we'll also introduce a scaling issue (INSERT ... SELECT could execute very long in huge lists) -- but this will come much later after the PHP issue.

In general this seems to work fine.
Attached the patch for 6.x-2.x which is already applied. I've needed to add a new condition to exclude unsubscribed records. Also i needed to store the amount of subscribers which is needed in 6.2.
Note that this will only go into the 6--2 feature release and won't be backported to 6.x-1.x. Please help test the new release and make it stable. If there are some new insights please reopen.

Dave Cohen’s picture

I can't answer the postgres question. Not sure, but I don't think mysql created this syntax.

So, I have one site currenly using simplenews, and I don't want to upgrade to the 2.x branch unless the upgrade path is seemless and it is reliable. How far along is 2.x and when do you recommend people upgrade?

The code you added for t.status does not need to be in the 1.x branch, correct? My database does not have that column.

Thanks, -Dave

miro_dietiker’s picture

Version: 6.x-2.x-dev » 7.x-1.x-dev
Status: Fixed » Patch (to be ported)

Dave, we're already using it in production.

The upgrade path has been written and tested and many features got an overhaul to implement them more fundamentally. I think 2.x is much cleaner now.

There are some critical blockers left, but we get very few help in fighting them.
Also as stated above - we're also looking for some parts to rewrite (a little) - or they will remain in the current (better than 1.x, but not so perfect) state.

:-) thanks for your support.

This patch needs port (or check). Please switch back to 6.x if some work here needed.

miro_dietiker’s picture

we're working on this.

miro_dietiker’s picture

Status: Patch (to be ported) » Fixed

D7 port submitted by DrupOn. Fixed.

DrupOn’s picture

Status: Fixed » Closed (fixed)

Replaced the iteration through every single record with the proposed INSERT ... SELECT ... statement in D7 version of simplenews.