'Node' filter rejecting forwarded emails

Have tested the above scenario with both: IMAP and POP3.

Same effect, no luck.

Reference old Issue: #1192516: Why 'Forwarded Messages' Fail Authentication?

We use MS Exchange Server 2010 with SP1

Dear Dane,

I understand from the last Issue that my scenario was not reproducible at your mailing environment, which adds a difficulty to trace down the real problem factor.

However, I might hack the module code to insert a few var_dump($variable) statements, that shall reveal what is going under the hood, to help spot the failure point.

If I would do that, what variables would you direct me to diagnose?

Shall print for a you a nicely formatted array representation:

function print_array($aArray) {
 echo '<pre>';
 print_r($aArray);
 echo '</pre>';
}

Thanks and Regards
Bakr

Hoping this narrow down will help, I have added the following line below:

drupal_set_message("Parser Thinks mail is empty!");

  /**
   * Implementation of FeedsParser::parse().
   */
 
public function parse(FeedsImportBatch $batch, FeedsSource $source)
{
    $fetched  = $batch->getRaw();
    $mailbox  = $fetched['mailbox'];
    $messages = $fetched['messages'];
    
    
    
    if (!empty($messages)) {
        foreach ($messages as $mid => &$message) {
            $this->authenticate($message, $mailbox);
            
            if ($message['authenticated_uid'] == 0) {
                //  " User was not authenticated");
                module_invoke_all('mailhandler_auth', 'auth_failed', $message);
                $source_config = $source->getConfigFor($this);
                
                switch ($source_config['if_auth_fails']) {
                    case 'remove':
                        if ($class = mailhandler_plugin_load_class('mailhandler', $mailbox->settings['retrieve'], 'retrieve', 'handler')) {
                            $class->purge_message($mailbox, $message);
                        }
                    case 'retry':
                        throw new Exception('User could not be authenticated. Please check your Mailhandler authentication plugin settings.');
                    case 'unpublish':
                        if ($class = mailhandler_plugin_load_class('mailhandler', $mailbox->settings['retrieve'], 'retrieve', 'handler')) {
                            $class->purge_message($mailbox, $message);
                        }
                        $message['status'] = 0;
                        break;
                    case 'ignore':
                        if ($class = mailhandler_plugin_load_class('mailhandler', $mailbox->settings['retrieve'], 'retrieve', 'handler')) {
                            $class->purge_message($mailbox, $message);
                        }
                        $this->commands($message, $source);
                        break;
                }
            } else {
                if ($class = mailhandler_plugin_load_class('mailhandler', $mailbox->settings['retrieve'], 'retrieve', 'handler')) {
                    $class->purge_message($mailbox, $message);
                }
                $this->commands($message, $source);
                
            }
        }
        $batch->setItems($messages);
        
    } else {
        if (isset($fetched['new'])) {
            drupal_set_message('No new messages.');
        }
		else
		{
++++			drupal_set_message("Parser Thinks Mail is Empty!");
		}
    }
}

Keep in mind, that currently I have to (legitamte) mail messages waiting to be handled by mailhandler, in the it.helpdesk mailbox.

I can send the php header dump

Watchdog returned empty variable strings.

Alternativaly,
I have produced my own header printout, below you will find the headers of two messages,
- the first one, was sent directly to the mailhanlder.
- the other one(was forwarded), still resides in the mail box, and gets unoticed by mailhanlder.

The ration here is to give your wisdom a glimpse into what is happening, such you might catch the malformed spotty area

==================================================================
// THE FOLLOWING IS NORMAL MESSAGE HEADER DUMP -- NO ISSUE WITH IT :-)
==================================================================

stdClass Object
(
    [date] => Fri, 16 Dec 2011 03:23:06 +0400
    [Date] => Fri, 16 Dec 2011 03:23:06 +0400
    [subject] => direct test
    [Subject] => direct test
    [message_id] => <528E3B841BE4A049AE57E9508550FF005FBAD02A@...>
    [toaddress] => IT Helpdesk 
    [to] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => IT Helpdesk
                    [mailbox] => IT.Helpdesk
                    [host] => domain.com
                )

        )

    [fromaddress] => Bakr Mustafa 
    [from] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => Bakr Mustafa
                    [mailbox] => Bakr.Mustafa
                    [host] => domain.com
                )

        )

    [reply_toaddress] => Bakr Mustafa 
    [reply_to] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => Bakr Mustafa
                    [mailbox] => Bakr.Mustafa
                    [host] => domain.com
                )

        )

    [senderaddress] => Bakr Mustafa 
    [sender] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => Bakr Mustafa
                    [mailbox] => Bakr.Mustafa
                    [host] => domain.com
                )

        )

    [Recent] => N
    [Unseen] =>  
    [Flagged] =>  
    [Answered] =>  
    [Deleted] =>  
    [Draft] =>  
    [Msgno] =>    4
    [MailDate] => 16-Dec-2011 03:23:06 +0400
    [Size] => 1088
    [udate] => 1323991386
)
==================================================================
// THE FOLLOWING IS THE APPARENTLY MALFORMED FORWARDED MESSAGE HEADER DUMP
==================================================================
1stdClass Object
(
    [date] => Thu, 15 Dec 2011 10:34:02 +0400
    [Date] => Thu, 15 Dec 2011 10:34:02 +0400
    [subject] => RE: An IT Form is Pending Your Approval
    [Subject] => RE: An IT Form is Pending Your Approval
    [in_reply_to] => <2836d867-2071-4f1a-be7c-7904502db807@...>
    [message_id] => <9B46C58C0404A1429C605EAFCFF6904479D94FF6@...>
    [references] => <2836d867-2071-4f1a-be7c-7904502db807@...>
    [toaddress] => IT Helpdesk 
    [to] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => IT Helpdesk
                    [mailbox] => IT.Helpdesk
                    [host] => domain.com
                )

        )

    [fromaddress] => Paul Green 
    [from] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => Paul Green
                    [mailbox] => Paul.Green
                    [host] => domain.com
                )

        )

    [reply_toaddress] => Paul Green 
    [reply_to] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => Paul Green
                    [mailbox] => Paul.Green
                    [host] => domain.com
                )

        )

    [senderaddress] => Paul Green 
    [sender] => Array
        (
            [0] => stdClass Object
                (
                    [personal] => Paul Green
                    [mailbox] => Paul.Green
                    [host] => domain.com
                )

        )

    [Recent] => N
    [Unseen] =>  
    [Flagged] =>  
    [Answered] =>  
    [Deleted] =>  
    [Draft] =>  
    [Msgno] =>    3
    [MailDate] => 15-Dec-2011 10:34:02 +0400
    [Size] => 44256
    [udate] => 1323930842
)

The following is the script I used to generate the above

$authhost="......";
$user="......";
$pass="......";

if ($mbox=imap_open( $authhost, $user, $pass ))
        {
         echo "<h1>Connected to:</h1>\n".$authhost;
         
        } else
        {
         echo "<h1>FAIL!</h1>\n";
        }
echo "<hr size=1>";
		
$inbox = imap_open($authhost,$user,$pass,NULL,1) or die('Cannot connect to Gmail: ' . print_r(imap_errors()));


/* grab emails */
$emails = imap_search($mbox,'ALL');

/* if emails are returned, cycle through each... */
if($emails) {
  
  /* begin output var */
  $output = '';
  
  /* put the newest emails on top */
  rsort($emails);
  
  /* for every email... */
  foreach($emails as $email_number) {
   
    /* get information specific to this email */
    $overview = imap_fetch_overview($inbox,$email_number,0);
	$header   = imap_header($inbox, $email_number); 
    $message  = imap_fetchbody($inbox,$email_number,2);
    

    
    /* output the email body */
    $output.= print_r($header);
    echo $output.'<hr>';
  }
  
  
 
} 

/* close the connection */
imap_close($mbox);

I have tried setting "From header" to
- "Sender"
- "From"
-

no luck in three cases so far.

By the way..

Even inserting the:

watchdog('mailhandler',"hello");

, proved that it was not even invoked.

the watchdog table was empty.

This indicates the logic is not reaching even here.

function _mailhandler_get_fromaddress($header, $mailbox) {

//***************************************************************************************
 
+++++  watchdog('mailhandler',"hello");
 
  


  if (($fromheader = drupal_strtolower($mailbox->settings['fromheader'])) && isset($header->$fromheader)) {
    $from = $header->$fromheader;
  }
  else {
    $from = $header->from;
  }
  return array($from[0]->mailbox . '@' . $from[0]->host, (isset($from[0]->personal)) ? $from[0]->personal : '');
}

But for sure it got invoked with importing a normal direct message. that is, the watchdog entries where created normally.

By the way, I use the mimemail module, does it affect?

To your knowledge, Is it different from phpmailer?

So, if the $header (as per the patch) returned nothing in the watchdog table; What could be the deeper further step to dig for trouble within the module?

Keep in mind that the ajax test script for connectivity with the mailbox successfully identifies some messages as residing in the mailbox.

somewhere in between their is a "certain" extra exception handling that needs to be taken care to make thing less fragile, and thus eliminate the anecdotal behavior.

Best Regards
Bakr

Aha,

After setting the fetcher setting so filter "All", then trying to re-import, the result is the following error.

An error has occurred.
Please continue to the error page
An error occurred. /hd/?q=batch&id=217&op=do
Fatal error: Cannot instantiate abstract class MailhandlerFilters in F:\htdocs\hd\sites\all\modules\mailhandler\mailhandler.module on line 168

Import simply fails.

Hmmm!

function mailhandler_plugin_load_class($module, $plugin, $type, $id, $args = NULL) {
  ctools_include('plugins');
  if ($class = ctools_plugin_load_class($module, $type, $plugin, 'handler')) {
	 
++++	var_dump($class);
	
     return new $class($args);
  }
}

the $class variable was pointing to "MailhandlerPhpImapRetrieve"

Now we need to inspect that class file "MailhandlerPhpImapRetrieve.class.php"

I shall inject some var_dumps...

I think we are getting somewhere with this...

Ok... looking in the problematic class...

I noticed that in-between I had to restart apache, as php was claiming out of memory... there is a leak here, eventhough I have more than 1GB allocated for PHP - can you imagine!

anyway....

the $new variable dumps the following:

array(4) { [0]=> int(1) [1]=> int(2) [2]=> int(3) [3]=> int(4) }

So ... stepping next ....

The following var_dumps where injected (see +++)

 function retrieve($mailbox, $limit = 0, $encoding = 'UTF-8', $filter_name = 'MailhandlerFilters') {
    // This is cast as string in hook_menu, otherwise the url argument would get used.
    $limit = (int) $limit;
    if ($result = $this->open_mailbox((object)$mailbox->settings)) {
      $new = $this->get_unread_messages($result);
	  
+++	  echo " \n\n *** \$new =\n " .  var_dump($new);
	  
    }
    else {
      imap_errors();
      throw new Exception(t('Unable to connect to %mail. Please check the <a href="@mailbox-edit">connection settings</a> for this mailbox.', array('%mail' => $mailbox->mail, '@mailbox-edit' => url('admin/build/mailhandler/list/' . $mailbox->mail . '/edit'))));
    }
    if ($result && !empty($new)) {

+++	  $status = imap_status($result, $mailbox, SA_MESSAGES);
+++	   echo "\n\n *** \$status = " . var_dump($status)  . "\n";
	
+++	  echo "\n\n *** \$result = " . var_dump($result)  . "\n";
	  
		
      $messages = array();
      $retrieved = 0;
     
	  while ($new && (!$limit || $retrieved < $limit)) {

+++      echo "\n\n *** \$retrieved = " . $retrieved . "\n" ;
+++	  echo "\n\n *** \$this = " . var_dump($this) . "\n";

        $message = $this->retrieve_message($result, $mailbox, array_shift($new), $encoding, $filter_name);
		

		
		
        if ($message) {
          $messages[] = $message;
        }
        $retrieved++;
      }
	  
	 
	  
      $this->close_mailbox($result);
      return $messages;
    }
    else {
      $this->close_mailbox($result);
      watchdog('mailhandler', 'Mailbox %mail was checked and contained no new messages.', array('%mail' => $mailbox->admin_title), WATCHDOG_INFO);
    }
  }

and the result error message which appears during import was as such:

An error occurred. /hd/?q=batch&id=221&op=do

string(26) "MailhandlerPhpImapRetrieve" 

array(4) { [0]=> int(1) [1]=> int(2) [2]=> int(3) [3]=> int(4) } 

*** $new = bool(false) 
*** $status = resource(10) of type (imap) 
*** $result = 
*** $retrieved = 0 

object(MailhandlerPhpImapRetrieve)#3970 (0) { } 

*** $this = string(18) "MailhandlerFilters" 

<b>Fatal error</b>: Cannot instantiate abstract class MailhandlerFilters in <b>F:\htdocs\hd\sites\all\modules\mailhandler\mailhandler.module</b> on line <b>174</b><br />

So, no surprise, the failing line is:

$message = $this->retrieve_message($result, $mailbox, array_shift($new), $encoding, $filter_name);

Now ...

going to dig some var_dumps into the following function, to see what can be revealed..

function retrieve_message($result, $mailbox, $i, $encoding, $filter_name = 'MailhandlerFilters')

And i think, that because the above function fail unsafely some how, this may elaborate why the upper calls was failing to be instantiated.

hmmm..

All recent findings indicate that there is some hidden limitation in the filters handling mechanism.

As you previously recall, that my inbox was having two stale (forwarded) messages. the whole assumption is that they should be digested by the mail handler and cleared from mail server.

Now, something interesting happened, one of them got through, (while I was fiddling with the code), the other did not.

Now I get this error which further leads to the sneaky trouble area, or call it the root cause:

An error occurred. /hd/?q=batch&id=224&op=do 

string(26) "MailhandlerPhpImapRetrieve" 

array(1) { [0]=> int(1) } 

*** $new = bool(false) 
*** $status = resource(10) of type (imap) 
*** $result = 
*** $retrieved = 0 
object(MailhandlerPhpImapRetrieve)#3970 (0) { } 

*** $this = 
*** array_shift($new) = 1 
*** $encoding = 
*** $filter_name = MailhandlerFilters 
string(18) "MailhandlerFilters" 

*** $filter_name = string(18) "MailhandlerFilters" 

<b>Fatal error</b>: Cannot instantiate abstract class MailhandlerFilters in <b>F:\htdocs\hd\sites\all\modules\mailhandler\mailhandler.module</b> on line <b>174</b><br />

Now we are pointing fingers to:

class MailhandlerFilters

At last it works (Now mailhandler is able to import every message in the mailbox normally, regardless it was forwarded or directly sent), but after commenting the $filter (see below)

 function retrieve_message($result, $mailbox, $i, $encoding, $filter_name = 'MailhandlerFilters') {
	  
	
    $header = imap_header($result, imap_msgno($result, $i));
	
	drupal_set_message( "\n\n *** \$header = " . print_r($header) . "\n\n\n\n" );
	
	
    // Check to see if we should retrieve this message at all
/*     if ($filter =  mailhandler_plugin_load_class('mailhandler', $filter_name, 'filters', 'handlers')) {
	
      if (!$filter->fetch($header)) {
		 
        return FALSE;
	
      }
	  
...
...
...

Ration is, this is indeed in need of some review of the filtering.

Ofcourse this is not a solution, but a lead.

The filtering was creating all the issue in the up-chain of mailhandler funtion.

Findings Summary
============

• There is no issue at all with the imap library.
• There is no issue at all with the mail header malformation.
• There is no issue at all with process of retrieving the content from the mail server.

The following function is the root cause of trouble, yet I do not know the root of the root!

$filter =  mailhandler_plugin_load_class('mailhandler', $filter_name, 'filters', 'handlers'))

• It fails the import process, whether I use regardless of Fetch Mode: All/Comment/Nodes
• It also fails the import process whether I choose Mime settings as HTML/Text

I agree with #26.

That is what is happening.

and yes true the recent fix was a step forward but closing another parallel issue.

I do not know why does the filter reject ... if only i can get to know this.
Also, the filter, as a function is doing great, can't go without it... if it wasn't for this issue.

maybe as a short solution, would be to add a checkbox to be added in importer configuration screen, so that to bypass filtering is a message is detected being a "forwarded" message

Am not sure how far this can help.

My voice goes to option 1: the mailcomment.

This would segregate the issue into workable divide-and-conquer approach.