How to retrieve additional elements from source (e. g. images)

Is http://drupal.org/project/feeds_imagegrabber which is hinted in #624088-44: Mapper: ImageField/FileField (formalize feed elements) necessary?

In #721836-2: Where is the embedded media content field I created in Feeds? the answer is in using simplepie and map the 'enclosure' source to your image target.

It works for me. But the full node body (from description text) contains the image too :( ... so make sure your input filter does not allow img.

[edit] there is no need for feeds_imagegrabber.

Hmmm ... I tested the import on http://feeds.feedburner.com/arch-maaik with puts the image into my image field.

But the feed from http://drupal.org/node/feed is not mapped onto my image field.

Steps to make it work:
1. Add image field to your content type
2. Map target enclosures to your image field.
3. Import the feed

Working feeds (enclosure mapped on image field):

- http://api.flickr.com/services/feeds/photos_public.gne?tags=jump&lang=en...
- http://feeds.feedburner.com/arch-maaik

Not working feeds (image field stays empty):
- http://drupal.org/node/feed

Dear Clemens,

I am trying to map this feed http://feeds.feedburner.com/arch-maaik with no success.

That's what I've done:

1. I created a content type "A" wich only has an imagefield and no body.
2. I cloned "default feed" feed importer
3. I edited my custom feed importer with this settings (there's an export code below):
a) Basic settings: Attached to "Feeds" content type
b) Fetcher: I didn't change anything on it
c) Parser: SimplePie Parser
d) Processor: node processor with content type set to "content type A" (created at step #1)
e) Mapping: the same as feed defaults, only deleted "description" (because I don't want to import any description, only attached file) and added "enclosure" from source and targeted into the imagefield created at step #1
4. Added content to feed, which points to http://feeds.feedburner.com/arch-maaik
5. Imported feeds

The result after importing the feed is that the newly created node has a body and no image field!!! After checking which content type the feed has been imported to, I realized that it has been added into "Feed item", which is where defaults feed imports it, so I deactivated default Feed import and now, when I try to import new feed items I get the following error: "Download of failed with code -1003."

What am I doing wrong?

$feeds_importer = new stdClass;
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'feed_arch_maaik';
$feeds_importer->config = array(
  'name' => 'Feed: arch.Maaik',
  'description' => 'Feed de arch.Maaik',
  'fetcher' => array(
    'plugin_key' => 'FeedsHTTPFetcher',
    'config' => array(
      'auto_detect_feeds' => FALSE,
      'use_pubsubhubbub' => FALSE,
      'designated_hub' => '',
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsSimplePieParser',
    'config' => array(),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsNodeProcessor',
    'config' => array(
      'content_type' => 'archmaaik',
      'update_existing' => 0,
      'expire' => '259200',
      'mappings' => array(
        '0' => array(
          'source' => 'title',
          'target' => 'title',
          'unique' => FALSE,
        ),
        '1' => array(
          'source' => 'url',
          'target' => 'url',
          'unique' => 0,
        ),
        '2' => array(
          'source' => 'guid',
          'target' => 'guid',
          'unique' => 1,
        ),
        '3' => array(
          'source' => 'enclosures',
          'target' => 'field_img_maaik',
          'unique' => FALSE,
        ),
      ),
    ),
  ),
  'content_type' => 'feed',
  'update' => 0,
  'import_period' => '86400',
  'expire_period' => 3600,
  'import_on_create' => 1,
);

With the attached importer I just downloaded the image from the enclosure.

Change the created content type on /admin/build/feeds/edit/tester/settings/FeedsNodeProcessor

Check for the mapping on /admin/build/feeds/edit/tester/mapping

Navigate to /import/tester and paste the http://feeds.feedburner.com/arch-maaik

It works for me.

I des-installed the module and re-installed again and it worked fine!

It seems that there's no way to have different mappings providing from the same content type :( (It has its logic, but I wonder why there's a need to create a content type for only one node when only a url is wanted)

Multiple Mappings/Same Feed Content Type is a current bug/user interface problem. I could not come up with the search terms to find the issue that talks about it. If someone can post a link, would be helpful.

Sounds like the problem here is solved.

Sorry for setting this one back to 'active', but I thought I'd give up trying to figure this one out myself after trying for days now and ask for some help (at least some clarifications) before I start trying again...

The feed items with images I am trying to import are in this form:

<item>
  <title>&quot; some title text here&quot;</title>
  <link>http://news.provider.net/path/to/article.html</link>
  <description> 
    &lt;p&gt;&lt;a href=&quot;http://news.provider.net/path/to/article.html&quot;
    title=&quot;&quot; some title text here &quot;&quot;&gt;&lt;
    img src=&quot;http://image.hosting.net/provider/news/articles/article_number/some_image.jpg&quot;
    width=&quot;100&quot;
    height=&quot;80&quot;
    alt=&quot;&quot; some title text here &quot;&quot;
    align="left"
    border="0"
    /&gt;&lt;/a&gt; main article body text here .&lt;/p&gt;&lt;br clear="all"/&gt;
    
    &lt;br /&gt;&lt;small style='font-size: 12px;color: #6f6f6f;'&gt;&lt;a 
    href='http://www.twitter.com/provider_twitter_account'&gt;
    Follow us on Twitter &lt;/a&gt; for &lt;strong&gt; more &lt;/strong&gt; news! &lt;/small&gt;
  </description>
  <pubDate>Wed, 28 Apr 2010 09:09 +0500</pubDate>
  <author>some author here</author>
  <guid>http://news.provider.net/path/to/article.html</guid>
</item>

1st question: Do I need Feeds Image Grabber enabled or not? The way I see it, it doesn't help much since the image I am trying to map to the image field is included in the feed content and not in any enclosure or separate tag. This is the only reason I can think of for why my efforts using FIG bring no fruits :( ...please confirm.

2nd question: Looking at the source code of the working/non-working feeds provided in #3 and comparing them to my code above, I think that my case falls under the same category as the Drupal (non-working) feed. Is it so and should I simply give it up?

3rd question (kinda off-topic here): Is there a way to strip out content from the description/body using regular expressions. I am asking because I need to remove the links to twitter/facebook etc. that some feeds contain.

I did not use feeds image grabber for importing the feeds from #3. Having installed simplepie solved grabbing the closure images

1st answer: The working feeds from #3 have closure images. You could try the drupal feed with imagegrabber.

2nd answer: give up? I'm not sure ... I just wrote the location mapper which handles the target side. The embedded images in the description are at the source side. Is there a bug report of this? Please checkout. If not report it and then we should fix that :p

3rd: I guess you could use rules.module I guess to strip the body.

#8 3rd q: what can you NOT do with regexes? : P

But seriously: You could extend the FeedsSyndicationParser (or FeedsSimplePieParser), add an additional mapping source (getMappingSources()) and implement getSourceElement() - if getSourceElement() requests the new key added with getMappingSources() you could execute a regex on the item's description that peels out the image URL.

@Clemens: I will try the drupal feed as a test with FIG and report back. Will check for an existing bug and report if there is none. Will also check Rules module and report how it went.

@Alex: ...digesting :/

Thank you both for the prompt replies.

I am using Feeds and XML Parser and was wondering if I should bee able to map the images directly to the filefield, or have it download the images automatically.

Can Feeds do this with XML parser? Anybody have any experience doing this?

Thanks..

#12: You can map image URLs directly to filefield. This will download the images. Different issue though. For clarity, I am changing the subject here: This issue is not about mapping, it's about how to get the actual image URL (file URL, author name, etc.) out of a containing source element. The answer is #10, and yes: it will require reading up on documentation and code, and cracking open your text editor :-)

Fixed?

Sorry to reopen this, but the case of http://drupal.org/node/feed, where the img tag is embedded in the value of the description field is so common that I think this should be discussed further.. I extended FeedsParser based on the Common Syndication parser to identify another source element (description_img) which will extract the URL of the first image in the description and allow it to be assigned to an Image Field. It then removes all images from the description.

At least, it will be helpful to others, but I think that the existing RSS parser should consider an option to provide this kind of processing.

I won't commit this to feeds. Can be part of another project like http://drupal.org/project/feeds_imagegrabber

Please tell me where to add #15 file. I have the same case where I have to get image from description of the content.

@alex_b: Alex, can you please link to the feeds_imagegrabber project's page from feeds project page?

Thanx in advance.

PS: Sorry for setting this to 'active', I am merely trying to get this in your radar (I am guessing you are filtering closed issues out). You can set it back to closed once you've acted upon my suggestion ;)

@cnventura Thanks for this, saved me a job.

I'm not sure why feeds_imagegrabber is the best place for this, since from what I saw it's going off in a different direction of grabbing an image not from tags in the rss at all but from tags at the source url, making the job more heavy and unreliable for those who only want to grab something from the contents of rss.

I had to add the filefield validators to the filefield mapper for aesthetic/security reasons http://drupal.org/node/1230012

@jcnventura Sorry to bring this topic up again but can you write a patch at #15 for drupal 7? I have no idea when it comes to php so any help from anyone would be very much appreciated.

#20, @jcnventura, @MrNeko, I'd like to see this in drupal 7 version as well. But I guess a more general approach is to give user their own rules to post-process the content before saving to nodes fields. That'll be really awesome.

ok, i went ahead today to extract image src link from description, here's the patch. To serve my own needs, "Image source in description" option is added in Mapping source for Common syndication RSS parser.

Set the mapping target to Image, it'll import that link as a picture file.

@windmaomao Thanks a lot! That really made my day.

Have you missed #16 which refers to http://drupal.org/project/feeds_imagegrabber.

Unfortunately the project page is still not updated with a link to that project. This issue is kinda hijacked since comment #13.

To resolve this problem please report a new issue with patches #15 and or #22 if http://drupal.org/project/feeds_imagegrabber does not fill the reported issues.

I close this issue and please follow the hints above. The next links are hopefully helpful :-)

[Powered by #1115636: Issue Macros and Templates]

Status settings for an issue http://drupal.org/node/156119

Please read Making an issue report to improve this issue report.

Have you tried the steps from http://drupal.org/node/314185 before reporting this issue?

Wow #22 @windmaomao
been trying to get this to work with feeds_imagegrabber, feeds_querypath_parser, feeds_xpathparser and even started with feeds_regex_parser but with no result because the inside the description the image url was encoded..

But your patch is super!!
please add/commit this to the feeds plugin!!

this is the best way to import image urls inside of descriptions
thanx a lot!

:-)

Just manually patched this against the dev version of feeds 7.x-2.0-alpha8+8-dev
And it really works super great!
Best tool to scrape images url sin an < img > tag from a description field
Thanks again @windmaomao :)

Please commit this patch!

This patch still works 7.x-2.0-beta1 and I didn't run into any noticeable issues while importing images. Best solution since the Feed Image Grabber module is not working at the moment.