Bad message number on retrieval

subscribing

I'm not sure whether it's the module's fault or PHP's fault.
If you can further debug the problem and come up with a solution, I'd appreciate it.
One thing to try: manually erase all your mailbox messages (through direct access to it), and retry fetching them with mailhanlder, with your above config. See if it gives you immediate remedy.

I have a little more information about this.

The module retrieves messages one by one and for each message it opens and closes an IMAP connection. Once the connection is closed the messages marked for deletion are deleted and the count of messages drops by one.

What tobbe_s guessed is correct. There are two ways of doing this.

One is to rewrite the fetching to reuse the same IMAP connection. This sounds more efficient to me.
The second is to use the existing strategy and use message_uid which is a required header and unique instead of the message number which is fickle.

I will try this and post a patch.

This patch implements the strategy of using imap_uid instead of the imap_msgno as was being done earlier. This is the right way of uniquely identifying the messages given that the message numbers will keep changing when the messages are deleted and connection reopened for every message.

Cheers
Anoop

Should this be applied against the latest dev? I just applied it to the Dec25 version and it went fine. I was having the same exact problem. Still doing a bit more testing but so far this has fixed the issue. And I am changing this to critical because we should be able to fetch more than one email at a time without error:)

R,
Coby

I did not test against the dev version but if it works fine then it should be fine :-). If you can read the source and confirm, perfect :-)

I m able to fetch multiple mails now, but I need to test more with mail2web. Not sure if this may change things that mail2web needs to track threading etc. Quick look at the code made me think it would be OK, but my initial test posted a comment against the wrong thread. Not sure if this is an artifact from other things.

It looks like it was an artifact from other issues. I am conducting a private beta with some clients on Mailhandler/Mail2web and have made a few changes at the same time as applying this patch. Not very scientific I know:)

This looks to be working fine for me on latest D6 dev as well. Thanks.

R,
Coby

I have this error in the most recent non-dev Release DRUPAL-6--1-5.

Looking forward to that "March bug fix release" i guess :D

I am using this patch on three sites now so changing status. Thanks much for the patch.

Patch works for me (for over a month now), the code makes sense. So I'll second the "reviewed & tested" status.

If you can confirm the patch doesn't influence any existing messages/content, I'll include it in the coming release (already in CVS for now)

Committed to 6.x-1.6 release

I am getting the same error, and I am using the 6.x1.6 release of mailhandler. Help?

@kddailey: Please post more details - Drupal version, PHP version, mailhandler version (verify), and the error message itself.

I can confirm I am still getting this error using the latest version of mailhandler, while using mail2web.

PHP 5.2.8
Mailhandler 6.x-1.7

In watchdog, I see the messages:

"Could not fetch structure for message number X" where X is the message number, like 2, 3, 4, etc. I sent 4 test emails at once time, ran cron, and two got through, while the other two did not.

The on-screen errors I get are:

    * warning: imap_header() [function.imap-header]: Bad message number in /home/ianward/public_html/dev/drupal6/modules/mailhandler/mailhandler.retrieve.inc on line 624.
    * warning: imap_fetchstructure() [function.imap-fetchstructure]: Bad message number in /home/ianward/public_html/dev/drupal6/modules/mailhandler/mailhandler.retrieve.inc on line 58.
    * warning: imap_fetchstructure() [function.imap-fetchstructure]: Bad message number in /home/ianward/public_html/dev/drupal6/modules/mailhandler/mailhandler.retrieve.inc on line 58.
    * warning: imap_fetchstructure() [function.imap-fetchstructure]: Bad message number in /home/ianward/public_html/dev/drupal6/modules/mailhandler/mailhandler.retrieve.inc on line 119.

I have mailhandler set to delete messages after retrieval. I'm using a gmail account for the mailhandler box and the connection test works fine.

More information: the problem occurs even when I do not have the mailhandler mailbox set to delete the messages after retrieval. Additionally, it appears that every other message is getting processed. For example, I sent 8 messages at once, then ran cron to fetch them. Messages 2, 4, 6, and 8 were posted to the site. I got 5 sets of the 4 error messages posted in my previous comment (#17).

I'll do my best to check that over the weekend, as it seems critical.
Can you run another test of 11 messages, and confirm the error occurs on non-zero pair message numbers (2,4,6,8...)?

Great. I ran the test with 11 messages. In this case, messages 1, 3, 5, 7, 9, and 11 were posted to the site. Before I ran this most recent test, I ran cron a few times to empty out the messages that were still waiting to be fetched. It's possible on my first test w/ the 8 messages that there were still messages from other testing that were waiting to be processed, which might explain why it's the odd numbers in one case and even in the other.

I accidentally reverted the version and status on last comment... switching back.

I am having the error referenced in the original ticket.
I am running:

Drupal 6.11
PHP Version 4.4.9
Mailhandler: 6.x-1.6

I am using POP3 but the error messages always reference imap.

If there is only one message, no problems.
When I sent 4 test messages, it confirms receiving 2 and sends the following errors:

# warning: imap_header(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 623.
# warning: imap_fetchstructure(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 58.
# warning: imap_fetchstructure(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 58.
# warning: imap_fetchstructure(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 119.
# warning: imap_header(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 623.
# warning: imap_fetchstructure(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 58.
# warning: imap_fetchstructure(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 58.
# warning: imap_fetchstructure(): Bad message number in /hsphere/local/home/medical/eco-voice.org/sites/all/modules/mailhandler/mailhandler.retrieve.inc on line 119.

Thanks.

I wrapped my head around this and discovered a number of things:

* It seems you cannot check whether a message has been read or not in POP. Is this really true? If so, and we want to support this, we'll have to implement some counter. Otherwise, help text should be added on the mailbox settings page saying you cannot leave messages on the server if you're using POP. There's a comment in mailhandler.retrieve.inc that says: "Flagging messages as unseen only works for IMAP so for POP mail to use limited downloads messages must be deleted after retrieval to prevent repeated downloading of the same messages on each run." I did some testing, and this seems to be the case.

* imap_uid does not work for POP. See the docs page http://us3.php.net/manual/en/function.imap-uid.php (look for note about POP3) and see http://bugs.php.net/bug.php?id=10850 which is the report that lead to getting this fact documented.

* Multiple imap streams are being opened and closed, one for each message. I do not know the overhead of this, but it seems unnecessary in any case.

* If you're using Gmail, you cannot delete messages with imap_delete. I can confirm they're not deleted, and there's an explanation here http://www.electrictoolbox.com/delete-message-php-imap-gmail/ but I did not test that. Maybe down the line this is something else to support.

* The bad message number error was happening because of a race condition. The array of messages would be passed in, and since there's no UID's for POP, this was a numbered list. The first message would open a stream, get processed, the message would be marked for deletion, the stream closed, and thus the message expunged. Then, the next message would be passed in, which should be #2, but now there's only 1 message left on the server because the other one was already deleted (assuming there were two messages on the server). This made be go to the single-stream approach.

Attached is a patch. I tried to make it smaller, but the problem was further complicated because of support for the batch API as well as the regular cron-time processing. The patch changes a number of things. A summary:

* Only open the imap stream w/in mailhandler_retrieve_message() if being called by the batch API. It did not seem possible to open one stream for the batch API, and pass it in in the operations array.

* Check whether we're passing in a stream to mailhandler_retrieve_message() and if not, open one (for batch) and then check whether one was passed in and close it if so.

* For cron processing, mailhandler_cron_retrieve() now opens and closes a single imap stream to the server instead of one for every message. This is what fixes the problem encountered in this thread in regards to the bad message number. I assume the people still having problems are those who are using POP.

I tried to test thoroughly, with IMAP and POP, with and without message deletion, as well as using the batch API. This could certainly use more testing.

@Ian Ward - I can give a feedback on one of the questions you had posted

Multiple imap streams are opened because it is executed as a batch job. This becomes relevant when you have a large number of messages.

@Anoopjohn - the retrieval of messages via the UI still uses the batch API and opens a stream per-message. The cron retrieval no longer opens a stream per message. Even before this patch, the cron retrieval did not use the batch API, but it did open a new stream per message. The mailhandler_max_retrieval ceiling is still respected during cron runs to avoid time outs. All in all, I think the same protections against timeouts are still in place.

@Ian: Many thanks for your efforts.

I applied the patch to the dev version, to facilitate the testing. Please test thoroughly, to hunt any additional issues, concerning this bug.

I only fixed one thing - you may have replaced two lines by accident:

-        array($mailbox, $message_number),
+        array($result = FALSE, $mailbox, $message_number),

and later on:

-function mailhandler_retrieve_message($mailbox, $i, &$context) {
+function mailhandler_retrieve_message(&$result, $mailbox, $i, &$context) {

/$result = FALSE/ is probably more suitable on the later change.

Thanks Zohar. That was indeed an accident... I think that:

array($result, $mailbox, $message_number),

should be:

array(FALSE, $mailbox, $message_number),

since $result at that point is an uninitialized variable, using FALSE is what I meant to do there.

Or initialize it to FALSE beforehand, otherwise it'll always be FALSE when it gets to mailhandler_retrieve_message, wouldn't it?

Or initialize it to FALSE beforehand

-- Yes, that would work too. Yeah, if it's initialized to false it will be clearer what's going on there as opposed to leaving uninitialized (even though it would still be false, in PHP).

Applied.

I use IMAP $mailbox = '{imap.gmail.com:993/ssl/novalidate-cert}INBOX';

When I fetch gmail list using IMAP in php in my site, it shows/fetches ok. But some problems are occurred like "Warning: imap_body() [function.imap-body]: Bad message number in /home/mibubble/mibubble.itcslive.com/gmail/myinboxDetails.php on line 15" during fetching the body of gmail.

And another interesting issue occures that some gmailid's body is fetched nicely, and some gmailid's body has not been showed. Instead of this orror has been occured like "Warning: imap_body() [function.imap-body]: Bad message number in /home/mibubble/mibubble.itcslive.com/gmail/myinboxDetails.php on line 15"

If any one knows about this thing and knows what is the solution for this? pls reply me in my email id like ankan@itcslive.com or ankanbhadra.developer@yahoo.com