Hi,
I have mailhandler installed and want to create nodes using the feeds module and a regex parser. My understanding is that the fetcher and the parser are very loosely coupled which means that the raw data from the fetcher can be utilised by any parser.
I have for example an HTML fetcher which I parse using an Xpath parser.
OK - so my problem is that the mailfetcher passes nothing to the regex parser.
When I select the mailhandler fetcher and the Mailhandler IMAP stream parser I can correctly select elements from the email and create a node using them.
But when I select the REGEX Parser nothing is given to the parser.
It appears that the following function call $fetcher_result->getRaw() returns an empty string.
Can you help me please? Should the getRaw() function return the content of the mail?
Thanks
Rich
Comments
Comment #1
Anonymous (not verified) commentedHi guys,
any help on this issue would be really appreciated.
Thanks
Rich
Comment #2
danepowell commentedI've long advocated for Feeds to allow multiple parsers to run in succession, for situations just like this. I even wrote a 'multi-parser' parser to allow for this, but ran into some issues. You can probably find traces of that in the Feed queue.
Anyway, given the current system, I think you are on the right track. I'm not sure why getRaw() would return an empty string, since that's the very first line in the Mailhandler Parser as well. I think it should return an array that includes the raw message. I suspect there is something else wrong with your setup- either a misconfiguration with the Fetcher or Mailbox, or the regex parser is broken. I'd suggest swapping out components and adding debug statements to try to isolate the issue.
Also, I vaguely recall that someone came up with a module that allowed you to 'chain' Feeds importers or components of importers together, so that you could e.g. create a node with an attached CSV file and then parse that CSV file. But I don't remember what it was called, sorry.
Let me know what you come up with.
Comment #3
Anonymous (not verified) commentedHi Dane,
thanks for the response. I don't quite understand why I would need to have more than one parser because I understood that I need a fetcher (the email handler) and a single parser (regex). I use an HTML fetcher for example and parse it with Xpath.
ok, but I don't mind digging around in the code. Can you give me one pointer to set me off.... I can do PHP quite well and have no problems with classes etc. But I'm not an expert in the drupal system (yet). So normally when debugging I slap in a var_dump or something. What would I use to get debug information onto the drupal page or where can I find information about that quickly in the docs.
I know that's a little off topic but it would set me off in the right direction!
Thanks
Rich#
Comment #4
danepowell commentedWell I'm not sure that you'll need multiple parsers, but just from a theoretical standpoint- it's quite conceivable that you'd want to extract/parse some sources via regex (i.e. the message body) and other sources via Mailhandler (i.e. the authenticated user id).
At any rate, what you want to do is install and enable the Devel module and then use the dsm() function to print debug info. Dsm() can handle arrays, strings, etc... and prints them beautifully.
Comment #5
Anonymous (not verified) commentedHi,
just a quick update. The problem with using IMAP and REGEX is that the getRaw() function call to the IMAP fetcher is that it returns an array. The array contains an entry for each message in the mailbox and these in turn contain header elements and body-text & body-html elements. So it all gets a bit complicated.
Now the questions are:
- what should getRaw() return: raw text or an array.
- how should a parser handle multiple emails? Should the parser be invoked for each mail etc etc.
If getRaw() was to return just the body-text would the IMAP parser fall over?
Cheers
Rich
Comment #6
danepowell commentedWere you able to solve this?
Comment #7
danepowell commentedNo response
Comment #8
Anonymous (not verified) commentedHi Dane,
sorry, but was travelling for a while. No, I didn't get this fixed although I still stick to the paradigm that a fetcher and a parser should be independent and that it should be possible to parse an EMail message with regex
Cheers
Rich
Comment #9
danepowell commentedI guess it does seem that other Feeds Fetchers simply provide a string for getRaw(), rather than an array. However, Mailhandler is a little more complex than a fetcher that simply retrieves an HTML page. It doesn't matter how you get an HTML page- wget, curl, whatever- the content will look the same. But with emails, there is a tighter coupling between the 'raw' content and the retrieval method (in most cases, the PHP IMAP library.) This is why the Mailhandler Fetcher returns a structured array rather than the raw email, since you would likely need access to the retrieval library in order to actually decode the email.
That's not to say that this can't be fixed- getRaw() could return a raw string, and somehow we'd have to store or embed a link to the retrieval library, and extended the retrieval API so that the parser could then call PHP IMAP functions to build the message- but this seems like a LOT of work. I welcome any patches though.
Comment #10
kscheirerWouldn't it be a lot easier to use a different parser that's not a plain regex parser? If it could handle the outside array and then regex on the mail body that should be sufficient right?
Comment #12
danepowell commentedYeah, I just don't see a good way to fix this. The Feeds docs do say that getRaw() should return a string, but I think this was designed with a simple unstructured HTML page or CSV file in mind. Our "raw" data has inherent structure that comes from the PHP IMAP library - to take all of this data and serialize it as a string in the fetcher, only to unpack it in the parser, doesn't make much sense to me.