I have site with lots of subscribers to the newsletter, and when sending I get this error:
*Fatal error*: Allowed memory size of 33554432 bytes exhausted (tried to allocate 40 bytes)
I'm pretty sure this happening in simplenews_send_node. I notice that it selects everything from simplenews_subscriptions, then inserts one at a time into simplenews_mail_spool. It seems to me this could be done in one database query with an INSERT ... SELECT ...
Is there any reason it's not done that way? Interested in a patch if I can make that work?
Thanks.
Comments
Comment #1
miro_dietikerThis is in 6.x-2.x on my radar. I already added notes in code+issues that the current implementation doesn't scale fine and the code is pretty silly.
I'd be very happy if you address this. Note that the spool is currently nod cleanly designed and needs further adaptions.
If you would like to work on its cleanup, let me know.
(currently if a node is right before sending by cron and gets edited and resaved, the spool is being rebuilt - on every save. This is pretty critical for "late edits" that usually happens when admin are in panic - and they're in panic quite often with newsletters...)
The transition into "scheduled / waiting to be sent" is a one way transition which needs representing correctly fixed implementation.
So this would in addition fix some well known issue.
Thank you. :-)
Comment #2
Dave Cohen commentedHere's the code I'm testing. Works on my development server. Haven't yet tested on the live server with thousands of subscribers. Would like you to review before I do that.
Comment #3
miro_dietikerGreat, thank you!
Do you know which mysql versions support this INSERT ... SELECT?
I found elder mysql seems to support it...
http://dev.mysql.com/doc/refman/4.1/en/insert-select.html
(How about postgres?) :-) ;-)
If there's some limitation we might need the previous way as a workaround if e.g. the query fails somehow...
Note that we'll also introduce a scaling issue (INSERT ... SELECT could execute very long in huge lists) -- but this will come much later after the PHP issue.
In general this seems to work fine.
Attached the patch for 6.x-2.x which is already applied. I've needed to add a new condition to exclude unsubscribed records. Also i needed to store the amount of subscribers which is needed in 6.2.
Note that this will only go into the 6--2 feature release and won't be backported to 6.x-1.x. Please help test the new release and make it stable. If there are some new insights please reopen.
Comment #4
Dave Cohen commentedI can't answer the postgres question. Not sure, but I don't think mysql created this syntax.
So, I have one site currenly using simplenews, and I don't want to upgrade to the 2.x branch unless the upgrade path is seemless and it is reliable. How far along is 2.x and when do you recommend people upgrade?
The code you added for t.status does not need to be in the 1.x branch, correct? My database does not have that column.
Thanks, -Dave
Comment #5
miro_dietikerDave, we're already using it in production.
The upgrade path has been written and tested and many features got an overhaul to implement them more fundamentally. I think 2.x is much cleaner now.
There are some critical blockers left, but we get very few help in fighting them.
Also as stated above - we're also looking for some parts to rewrite (a little) - or they will remain in the current (better than 1.x, but not so perfect) state.
:-) thanks for your support.
This patch needs port (or check). Please switch back to 6.x if some work here needed.
Comment #6
miro_dietikerwe're working on this.
Comment #7
miro_dietikerD7 port submitted by DrupOn. Fixed.
Comment #8
DrupOn commentedReplaced the iteration through every single record with the proposed INSERT ... SELECT ... statement in D7 version of simplenews.