dman's sandbox: Feeds Tamper URL Scraper

Import content directly from the URL referred to in a Feed Item

Given an URL within a feeds item, go back to the source page, scrape more info from it, and use that instead of what was provided.

If you have a teaser-only feed, but you need the full text:

• Enable Feeds Tamper and this module.
• Set up the feed with GUID, tag, type and date and mappings as usual.
• When creating the mapping for the body, set its source to 'Blank source'
• Now 'Tamper'.
• For the body, add a tamper action 'Copy source value'. Copy the item URL.
• Add another tamper action "URL Scraper".
• Configure the scraper pattern to match the target region in the markup of the target page.

This will first copy the URL (we need to copy to avoid changing the original value)
and then fetch the content of that URL into the body field.

Further combinations of tampering are possible, depending on your needs.

Details

This process will result in multiple requests back to the originating site being made in succession.
In some cases, this may flood the target, or cause timeouts at your end.
USE SPARINGLY.

You may find that it's server firewall or proxy issues if the http fetching is failing.
Diagnostic logs will be in the Watchdog.

History

See also an earlier iteration of a similar task:
https://www.drupal.org/sandbox/dman/feeds_tamper_file_fetcher

This scraper is significantly simpler than that one.

See Also

It's possible to set up a similar sort of process using https://www.drupal.org/project/feeds_selfnode_processor

This has the advantage of queueing the remote requests as later tasks, and should be updatable in the future.

However, the configurations for that were a little dense, and for our needs we just needs additional detail, fetched once, applied to our existing simple content types by the feeds process. We didn't want to have to manipulate the data structures of the target items themselves.