mailhandler fails to process batch (error batch.inc line 294)

I am finding the same. Seems to be caused by attachments on version 1.10. Going back to 1.8. It's good to know I am not the only one stuck. The retrieve just retrieves the one email then goes to a blank screen at:
myurl.org/batch?id=24&op=finished

Type php
Date Monday, February 15, 2010 - 18:17
Location /batch?id=24&op=finished
Referrer /batch?op=start&id=24
Message Invalid argument supplied for foreach() in public_html/staff/includes/batch.inc on line 294.
Severity error

(I'm in the same boat; back at 1.8)

Could anyone having this problem outline the steps to get it to repeat (including how you have mailsave setup) and I'll try to debug it this week? Thanks

Regarding 1.10 problem
My mailsave is just simply to attach files, no mobile options
As to mailbox options:
Using IMAP, 143, /novalidate-cert
Security:
Disabled
Send error replies:
Enabled
Delete messages

By the way:
I forgot where to turn on attachments so that mailsave could save them. The upload option was enabled under optional modules, permissions were OK for mailsave and upload. Finally found it under content-type options. Each content type must also have attachments enabled. In case anyone else is unable to get the mailsave attachments working.

For me:

1) I set up both mailsave and mailhandler with the most recent versions;

2) Then I tried to map an attachment to an incoming message to a CCK imagefield.

and BLAM.

=====

Dropping down to 1.8 eliminated the batch problem, but the imagefield problem remained. So I currently have 1.8 working with my below cobbled-together fix to mailhandler which for some reason works.

http://drupal.org/node/714402

Which re-adds a newly-missing image processing function among other things to the mailhandler-to-cck-imagefield module.

If I recall, I think I actually got the error without processing the attachments, but I can't be sure.

Hope this helps a little... (?)

Thank you for the additional information. The attachment problems are likely related to this bug http://drupal.org/node/718344 which I just committed a patch for to dev http://drupal.org/cvs?commit=330842 ... be aware the fix may not be available in the dev version until several hours from now but you could try patching your 6.10 version of mailhandler to see if it works.

I will check it out soon. Does it work for you?

Hi I've just installed Mailhandler and Mailsave and I'm having the same issue . Although the retrieve of the email attachment never worked for me. After sending an email attachment with an image, I am able to retrieve and the then page goes blank and I get a similar error:
Type php
Date Wednesday, February 24, 2010 - 07:22
User Me
Location http://www.mysite.com/batch?id=19&op=finished
Referrer http://www.mysite.com/batch?op=start&id=19
Message Invalid argument supplied for foreach() in /home/mysitedatabasename/public_html/includes/batch.inc on line 294.
Severity error

I have 6.x-1.3 version of Mailhandler installed (just did this 2 days ago) and Mailsave imagefield was giving me a problem so I have the mailsave imagefield patch from http://drupal.org/node/315381 installed as well.

Any ideas for me?

Which version of mailsave do you have?

Not sure if this is relevant, but it might help:

http://drupal.org/node/714402

Thanks for checking in I actually installed the version of Mailsave that is patched and reworked from http://drupal.org/node/714402. Does that make a difference? I was having problems with the image field ( see my prior query here http://drupal.org/node/723658 ) and this patch fixed it and I thought I was home free then got this new batch error.

I am getting many of these warnings, so I think it is a problem in mailhandler only:

# warning: reset() [function.reset]: Passed variable is not an array or object in /home/drupal/drupal-6.16/includes/batch.inc on line 184.
# warning: array_shift() [function.array-shift]: The argument should be an array in /home/drupal/drupal-6.16/includes/batch.inc on line 196.

I am only using mailhandler, I don't have mailsave installed, nor any of the image modules (yet). Note this is my first time using mailhandler, I've never actually had it successfully work.
This happened when I clicked on the retrieve, after receiving a successful test connection for the IMAP configuration.
It got stuck at retrieving 2 of 3 messages. None of the messages have images in them.

I'm using:
Drupal 6.16
Mailhandler 6.x-1.10
I'm connecting to a GoogleApps based Gmail account.

I checked for content and no posts were found.

I ran into errors when either the letter content was blank, or the subject blank... OR where the mime-encoded mail did not have a plain-text alternative. Basically, the module chokes whenever there isn't something to process, or so it seems. (See my issue with either this or mailsave re: not processing emails from Yahoo mail.

I'm seeing the same errors when trying to retrieve messages from a Gmail account. Tried both 6.x-1.10 and the dev version (as at Mar 8 2010). This does seem to be a showstopper for this module.
I'll try to go to the 1.8 version and see if that works.

Hi...
I've debugged tonight latest dev version (2010-Feb-20) a bit. Here is my "research" result. Maybe it can be helpful for somebody. I was trying write "mailsave_to_filefield" module which retrieves files from attachments and add them to cck fields. I was playing with pdf files.

--

The reason of white screen in line 294 of batch.inc exists because in function batch.inc/_batch_page() {...} there is call $batch = unserialize($data). And in our case the $data variable may be sometimes unserializable. :(

This variable $data is taken from {batch} table. See batch.inc file.
$data = db_result(db_query("SELECT batch FROM {batch} WHERE bid = %d AND token = '%s'", $_REQUEST['id'], drupal_get_token($_REQUEST['id']))))

This call of unserialize() sometimes can fail. If it fails then $batch variable become boolean value equal to "false".

And this is why foreach ($batch['sets'] from line 294 reports error. It can not use boolean variable as an array.

---
OK. But when/why this unserialize() can fail?

Here I'm not so certain. Sorry.

Function mailhandler_get_parts()/mailhandler.retrieve.inc creates some $part->data member.
This member is retrieved from from attachment and then directly serialized to {batch} array (to the "batch" field). This happens in case of APPLICATION/PDF mime. I don't know if this happens also in case of images.

But anyway our attachment may contains unrealizable binary characters.
It may be also bigger than capacity of "batch" field which is "longtext". According to my installation it accepts 64kB only.

For tests I've added near the end of mailhandler_get_parts() following line
$part->data = base64_encode($part->data);
In this case variable can be serialized() and then unserialized() properly. Problem with batch.inc error no longer manifests.

--
What next?
At first I don't know if storing potentially big attachments in {batch} table is OK. Probably storing them on disc as temporary files is sufficient. I'll try to solve this problem that way in following days but I can not promise.

Kuling, Thanks for your hard work on this; have you managed to solve the problem?

Hello...
Yes. I made some fix. This is some solution which works for me. Please tell me if it helps you.
As you read yesterday the main problem is because we want update "batch" field in {batch} table with some unserializable data taken from $part->data. ($part->data contains attachment content).

I have implemented following solution to avoid this batch 294.

a)
File: mailhandler.retrieve.inc
Function: mailhandler_get_parts()
Every time when retrieved attachment is relatively big or contains binary content I don't put this attachment in $part->data. It will not work. I create temporary file instead. Then I add information about this file to the {files} table. And instead of putting big content of attachment to $part->data I just add $part->fid which contains ID of file. Then I remove $part->data at all (to keep things simple).

b)
File: mailsave.module
Function: _mailsave_use_mailhandler_attachments
This change works in opposite direction. Every time when somebody wants retrieve content of stored attachment I made something opposite to previous work.
When I see that $mimepart has $fid member then I know that it refers to file. I just load file from disc and put it content to $mimepart->data.

As you may see this whole change shall be completely transparent to all modules. "batch" field in {batch} table no longer holds large potentially unserializable content but just small ID of file.

--
If you want test my changes then please apply attached patches to following code.
http://ftp.drupal.org/files/projects/mailsave-6.x-1.3.tar.gz (2008-Aug-20)
http://ftp.drupal.org/files/projects/mailhandler-6.x-1.x-dev.tar.gz (2010-Feb-20)

Please use also following patch.
http://drupal.org/node/735488#comment-2688664

Remark that this patch is probably insecure. I'm still learning and don't know all Drupal security rules. Patch is not tested very well because of lack of time so I ask about your feedback. Use on your own risc.
I was able to process 1.8MB PDF after change (gmail/IMAP/windows). So enjoy.

Thanks! Could you just upload the rolled-up module rather than the patch files? That might be a better way to go about this... (I'm on a mac, so patching is also a pain ;) )

For the people who tried going back to 1.8 can you confirm everything does indeed work fine on 1.8? Knowing that 1.8 works fine would nearly confirm this is a regression. I will not have time until over the weekend to further debug this, but if someone in the meantime could trace this back to a regression and roll a related patch if fixable I'll do what I can to get it committed.

Could you please clarify which version of mailsave to use with 1.8?

@mig5 thanks for confirming about 1.8. I debugged this, and @Kuling was right about where the failure is. This is a regression. In 1.8 and below mimeparts was stuck onto the $node object and never passed through during batch processing. This was possible because in 1.8 and below mailhandler was more node-centric. In > 1.8 I further abstracted some of the node-centricity. Part of this required passing $mimeparts through the batch API. However, because of this issue http://drupal.org/node/718344 I did not see the breakdown in batch api. Batch shoves the whole result set into {batch} table, which is a longtext. If there are attachments, it looks like it breaks down trying to push in the serialized binary data.

I see two options:

* Something like @kuling's approach, where all files are stored temporarily
* Shut off progressive mode of the batch API.

I believe shutting off progressive mode of the batch api will make using the batch api pointless since the process will not be broken down into multiple requests and therefore the operation could time out. However, I also wonder how dependent people are on this batch API functionality.

The patch to turn off progressive mode is attached. I've not tested it with mailsave, but I have tested it with mailhandler alone, and it seems to work.

In conclusion - I'd like to hear from users whether they're really dependent on the ajax batch retrieval. To me, this functionality does not seem critical since I'd imagine a final web application would be dependent on automating the retrieval through cron, and that this batch ajax retrieval is probably only really used when people are building/testing their site. I'd love to get rid of this batch functionality entirely and just have the same button "retrieve" on admin/content/mailhandler but do it using the same method as cron. Then, focus in the future on making the non-batch retrieval smarter and more efficient. If batch api functionality is removed, this patch would be much larger in order to remove all the code devoted to that. People can try the non-progressive mode patch to simulate what it would be like, user-end, w/o the batch api.

Thoughts?

Same issue here. I agree that the batch retrieve is a nice-to-have.

@tmetzger - as in 'nice to have but not necessary' ? If I remove the batch API support there would still be a retrieval button for each mailbox... you just would not see how many messages have been retrieved until it's done retrieving. I've been thinking about this more. If PHP's max execution time is set to 30 seconds and you're using batch API to retrieve messages with large attachments, if any one message takes longer than 30 seconds to retrieve then I believe it will fail/time out. When you compare batch retrieval to cron retrieval, batch will be able to retrieve more/all messages, as long as no single message retrieval takes longer than the max execution time. I can see where batch is useful/necessary when running update.php, but in the case of mailhandler I really think making sure cron/automated retrieval is first priority and I don't like the complexity that batch support adds to the module.

In any case, what I'd probably do is apply the patch in #22 as to not affect much code, and then in the 2.x branch I'd actually remove the batch support and related code. Anyone else have thoughts on this?

Good point about the execution time. I can see that being an issue in some cases. Would it not also be an issue under cron, since cron also runs PHP?

Frankly I'm having trouble getting the thing to create nodes. When I use a taxonomy name it's failing on these lines:

        array_walk($data[1], 'mailhandler_term_map');
        $node->taxonomy = array_merge($node->taxonomy, $data[1]);
 

because $data[1] isn't an array for some reason. Giving me these warnings:

* warning: array_walk() [function.array-walk]: The argument should be an array in /Users/tmetzger/Documents/Groupanizer2009/htdocs/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 288.
* warning: array_merge() [function.array-merge]: Argument #2 is not an array in /Users/tmetzger/Documents/Groupanizer2009/htdocs/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 289.

And after that it doesn't put it into the forum properly.

Any clues?

@tmetzger Thanks for your comment. Yes, the same issue would apply for running via cron. My thinking is the time going to support batch (there have been issues w/ it since I started maintaining in July... there are essentially two different stacks to deal with: cron, and batch.) is killing time which should go toward making base functionality of the module work well, and at very little gain, from my perspective (zero gain, IMHO). Less is more.

About your taxonomy error, please open a separate issue, but check the commands documentation first if you've not already http://drupal.org/node/38943

I'm a dork - I read the documentation again and finally realized that the little square brackets are necessary. HA!

I folllowed the #17 and #22 patches but I could have the node created by not the attachments. Any followup on this patch working or not? Or some instruction? I don't have any error but the attachments just disappear.

I have it working; however, I would add this warning:

The batch problem seems to cause cron to never finish; so it continues to run. I only noticed this when I suddenly could not send mails from my site. The reason was that there was a limit to the number of processes an account could have open, and I had 20 cron processes running, so the mail process wouldn't go.

This is therefore critical to have fixed, as it can cause errors throughout the site if cron starts going a little bonkers.

I attach herewith my working versions (not sure which patches have been applied to both, but I think it's the most recent version of Mailhandler with the batch fix applied, and a version of Mailsave here http://drupal.org/node/714402 (I think).

In any case, I attach my pair of modules running on d6.16 install. I wish I could be more detailed about everything I have done to them, but I have lost track a little. I do hope they are a help to someone.

double post. sorry!

WHOOPS!

IGNORE THE ABOVE FILES. They still hang the cron process.

I think the attachment below works.

One thing: It might be best NOT to set a image size limit on your imagefield, as it could be the resize that causes cron to hiccup. Perhaps better to do your image-processing later via imagecache.

In any case, I think it very important that this gets handled; I wish I were better at this stuff, but I am going to have to leave it in IanWard's capable hands. Would love to know when this showstopping bug gets squashed.

acb,
Thanks for creating these new versions of mailhandler and mailsave. I installed them and they do seem to be working better when I manually retrieve mail from my mailbox. I initally got this error: about not being able to find the mailhandler_mailbox_load function on line 410, but clearing my caches fixed that problem.

I just sent a message to my mailbox, and clicked on Retrieve, and got the following:

warning: Invalid argument supplied for foreach() in /home/drupal/drupal-6.16/sites/all/modules/mailcomment/mailcomment.module on line 669.

1 message retrieved for test@xyz.org.

FYI, I checked my mailbox and got this error, which I believe is valid:

The email you sent to test@xyz.org was rejected because there was a validation error.

In order for emails to be accepted by XYZ:
- They must be sent in reply to a valid notification email.
- The reply must be done from the same email address the notification was sent to. 

Thank you for the updated merged files. I can confirmed the problem and the solution of "mailhandler_mailbox_load function" on #32. However, I could successfully retrieve the mail but the attached file still missing. I have no any error, the file just disappered. The same as I reported at #28。

So, what should I look to find the problem?

===> Updated form #33

Both #32 and #28 settings are all working, I didn't set the mailsave attachment rights on for the icoming mail author. Once set, they all working now.

The only problem I had is the filename's problem. Double-bytes characters are truncked as ____#.png. I will search for possilbe solutions. Because I used the upload in node editing the name shown Chinese ok.

If the patch in #22 works, that's what I'd like to go with. Here's what you can do to test:

* Download the latest release of mailhandler http://drupal.org/node/712596 (Do not use an older version when doing this testing)
* Apply the patch from #22 http://drupal.org/files/issues/mailhandler-batchapi-files-713656-22.patch ... if you do not know how to apply the patch, you can just do it by hand... it's a matter of adding those three lines in the spot shown in the patch file.
* Test using mailsave ( I am not sure what the best version to use is in this case, but, starting with an unpatched version would be ideal).

Thanks

Have you tried the transliteration module which helps those pesky filenames? I think I have it running on my server.

I still think something needs to be done with mailsave and large attachments hanging processes on a server, but that's probably not an issue for this thread anymore. We just want the damn thing working.

To #35, I tried the 1.10 and the patch. No luck the attachment error: "Invalid argument supplied for foreach() 在檔案 /url/...l/modules/mailsave/mailsave.module line 307"

To: #36, I had "transliteration module" before but it uses phonic translation for the filename. I may help for Latin base language system, user can guest what's it. But for Chinese, users still prefer the understandable filename.

Ok, I just digged into some Chinese Drupal community and try the following modification to mailsave.module (not sure to other language base, using with care). But my site is ok so for.

Change the line in function of "_mailsave_sanitise_filename($filename)" (the end of mailsave.module)
from "$filename = mb_decode_mimeheader($filename);"
to "$filename = imap_utf8($filename);"
... remark out the rest of code till last command "return $filename;"

That's what I got, hope this work for others.

BTW: this is all based on the Code at #31

I got around to testing mailsave myself. Everything works fine. Here's what I did.

* I applied the patch from #22 to mailhandler 6.1.10 http://drupal.org/node/712596
* I installed mailsave, development version: cvs checkout -r DRUPAL-6--1 -d mailsave contributions/modules/mailsave (it does not look like there's a dev release and the last release was from 2008 so i went with the dev version checkout from CVS
* I enabled mailsave, followed INSTALL.txt.
* I sent an email with a .png attached, retrieved via the UI on admin/content/mailhandler, and everything works fine. The image is attached to the node, no errors.
* I did the same as above with a text file, works fine.
* I did the same as above with two images and a text file, works fine.

If the patch in #22 does not work, and you're using the latest mailsave release, maybe there are bug fixes which are only in the dev version of mailsave.

If others can follow what I did and confirm everything works, as well as provide detail on what does not work, that will help to get something committed and released. Thanks.

Hi Ian (and thanks for your committed work to this),

To see if problems still exist, I would suggest a few options:

1) Send a Large (7mb or so) image.

If that works, try

2) Let an automatic cron retrieval do the processing, not the retrieve via the UI (this is where a number of bugs have popped up, esp. when doing so with large files).

If that works, try

3) Set up your imagefield (as many do) to accept images of only a certain size (1024x1024 is where mine is set), so the resize process has to happen during node creation after the image-retrieval, and use an automatic cron run to retrieve the email. This, to me, seems the most typical workflow for an image, and the one that times out/hangs cron and causes the most problems.

Anyway, that's my 2 cents; hope it helps.

If you can get #3 to work, we may just have a winner.

Mailhandler now processes batch email retrieval, but when used as part of Media Mover I get error

warning: imap_num_msg(): supplied argument is not a valid imap resource in /sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 338.

Btw, Mailsave isn't an option, since it's submodule "save to cck imagefield" isn't compatible with D6.16

UPDATE: I figured and solved what was the reason for the error. Two functions are missing from 1.x.dev that were present in 1.8:
mailhandler_authenticate()
mailhandler_process_message()

Had to copy-paste them into the latest mailhandler 1.x.dev

I'll update MediaMover maintainer.

FYI the only reason the maihsave to CCK Imagefield doesn't seem compatible is that it's info file is screwed up. There is a post over at mailsave somewhere detailing the 2-line fix.

Thanks for the hint, but I think I'll stick with MediaMover. Is there really big difference betwen MM and MailSave?

Does mediaMover grab from emails? I don't know. All I know is that both are stuck in beta

Yes, it grabs data from mail very well, however you have to play with "harvesting" options. It can save image attachment to a new node or save to a CCK field. Works very good in fact.

Hate to hijack this thread, as it is dealing with a critical bug in mailhandler, but does media mover deal with the node body as well, or just map the file over to a CCK field?

----

Back to the crucial question, though: Ian, did you manage to get my scenarios in #39 to work?

Subscribing. Got the same problem.

Works well with mailcomment, but fails with mmb on a fresh install.

Thanks

Same error in batch.inc line 294 with Mailhandler 6.x-1.10 and Drupal 6.14, but i don't use poormancron and mailsave.
The patch #22 solve the problem for me.

I haven't managed to populate body field with my email message body content. Still looking into it.

(skolesnyk :.... and therein lies the difference between the two!)

ok. Back on topic!

Perhaps we could get someone who has a working copy of Mailhandler AND Mailsave on D6.16 that retrieve content and messages successfully on cron (not on manual retrieve) to post their working modules?

Is this the case for anyone?

@acb I have not had time to test http://drupal.org/node/713656#comment-2756970 ... maybe this weekend but no promises. If you know a certain part in #39 fails please elaborate, or anyone else. Testers wanted. What's going to get this fixed is following #38 AND #39, but not posting working versions of modules because those versions could be from revisions to which reverting is not practical. If it's easier to post modules then the versions should be as stated in #38. I did notice there are issues w/ the dev release of mailsave that would prevent it from ever working w/ the latest version of imagefield, so if someone knows what version of mailsave to test against (clearly not latest stable and clearly not latest dev) and can list the version and list of patches necessary that would be helpful. Thanks

Ian,

I am happy to help out, but have prob's with patches (as I am on a mac and don't have dev-tools installed).

In any case, I found this helped clear up my batch errors:

      foreach (module_implements('file_insert') as $module) {
        $function = $module .'_file_insert';

//CHANGED  $function((object) $file);

       $function($file);
      }

in Mailsave_to_imagefield.module (see my comment here: http://drupal.org/node/690930)

I now have working pair of modules, if you want to see them, though not necessarily patches that you could apply easily.

Hi Ian,

The patch from #22 solved my issue.

Cheers.

@acb thanks for the info. @halcyonCorsair thanks for confirming. I'll try to test the rest of the cases in http://drupal.org/node/713656#comment-2756970 and commit this weekend.

As ever, thank you, Sir Ian. If you can confirm that #39 case 3 above is solved for you, then I think we have this licked.

Only other problem is when there is no content in the letter being sent other than the image, which also causes the pair of modules to choke (see my comment #1 on this thread: http://drupal.org/node/732728 )

Keep up the good work! A grateful world of mailhandlers will thank you.

The patch in #22 works for me as well.

One additional problem that I found is that 'mailhandler_node_process_message' is not receiving the mimeparts. This prevents any handler code from processing attachments. The fix is simple. The call needs to be modified to:

line 258:
  mailhandler_node_process_message($message['header'], $message['origbody'], $message['mailbox'], $message['mimeparts']);

and the function signature:
line 288:
function mailhandler_node_process_message($header, $origbody, $mailbox, $mimeparts) {

This will allow the assignment in line 300 ($node->mimeparts = $mimeparts) to work properly.

@swood:
That change is already in CVS, but unreleased. Nice spotting though. :)

I committed #22. Instead of closing I am deferring this issue because the fix is a stop gap. Batch support should be removed or should be refactored to handle cases where the data cannot actually be stored in the batch table while processing. As for mailsave, @acb the code you post in http://drupal.org/node/732728 I cannot find in the repos for mailsave, like "imagefield_check_directory($widget_image_path);" ... that module is going to need some work still but should be further discussed on that queue. It looks like there are functions from the Drupal 5 version of imagefield called in the mailsave module which are no longer supported in the imagefield d6 version.

Thanks as ever Sir Ian.

Are you thinking of a different post? on http://drupal.org/node/732728 I don't post code, just the problem!

Perhaps this one?

http://drupal.org/node/690930

@acb oops, yes, I meant on http://drupal.org/node/690930#comment-2790930

OK, I will keep my eyes over on that side of the fence for any progress. Is there anything else you need me to look into when I have a moment?

Thanks as ever, Ian.

Subscribing..

I understand the hesitancy to call this "fixed", but I think it would be worthwhile to drop a new stable release now that a "fix" has been committed. This is a huge regression and show-stopping bug for many people that additionally impacts other modules (Mail Comment for instance is completely unusable as it depends on Mailhandler > 6.x-1.10).

@Dane Powell - You're right, mailhandler 1.x needs a new release. I'm shooting for a release this weekend / early next week and will be committing patches in the queue beforehand. There are a few issues that still need patches if anyone has time to work on those:

* #743572: Body content when body field is omitted
* #778198: CSS inclusion far too broad
* #765080: Patch for mailhandler_get_fromaddress() when $from[0]->personal isn't set

I was facing the same issue. Whenever my mail had an attachment, the retreive would show a white screen on urls like this
http://myurl.com/batch?id=5&op=finished
If the mail did not have any attachments, things went well, and my node was created perfectly.
Looking at this thread i downloaded the latest release, and now i dont have the white screen of death, but now my nodes are made without any body. Nothing comes in the body of the node ireespective of wether the email had an attachment or not.
Please help.

bump, why postponed?

Sorry, 6.x-1.x is no longer a supported release. Please try upgrading to 6.x-2.x and reopen if still an issue.