Hi,

I need to create nodes from news in HTML-pages. So instead of repeatedly fetching news items from a feed, I'd like to repeatedly inspect a certain HTML-page, extract (maybe using regular expressions) news from it, and convert these to nodes.

I'd like to try to implement this and, if possible, would like to extend module Leech. Therfore: Do you think this is doable? Maybe by writting a new parser (if I am not mistaken, Leech can use different parsers).

I'd like to get some feedback by a Leech developer before I start, to coordinate with others and hear of the views of others.

Thanks,
Kaspar

Comments

alex_b’s picture

You don't need to touch leech for doing that. Write a screenscraper that produces an RSS feed. Subscribe then leech to that feed.

I bet that there are services like that out there. Let me know what you find.

kaerast’s picture

I've used Dapper in the past for this and there are indeed many other systems like it, it's far more sensible to use an external service like this because they have the time to spend coding a scraper.

hbfkf’s picture

Alex_b and kaerast, thanks a lot for your hints. I'll get back when I have taken a look at Dapper.

Best,
Kaspar

alex_b’s picture

Ok, I ll close this issue for the time being, feel free to reopen it when you want to follow up here on scraping html pages.

alex_b’s picture

Status: Active » Closed (fixed)

eem.

hbfkf’s picture

With your help I managed to screen-scrape the HTML-page to an RSS feed and use leech to import the news items into my system. Worked like a charm!

For the sake of completeness (maybe somebody else can use it as a starting point), here's my little screen scraper.

<?php
// Synopsis: Simple screen-scraper to convert the news on
// http://transparency.org/news_room/latest_news/press_releases to an RSS feed
// Please notice that this is just an example; this script must not be used unless you
// have permission by transparency.org to actually publish their content on your site.

// get page:
$url = "http://transparency.org/news_room/latest_news/press_releases";
$data = implode("", file($url)); 

// match entities of interest:
preg_match_all('/<p>\s*<a href="([^"]*)">([^<]*)<\/a><br \/>\s*(([^<]*)<br \/>)?
\s*([^<]*)<\/p>/sUm', $data, $matches);

header ("Content-Type: text/xml; charset=utf-8");
echo "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n";
?>
<rss version="2.0"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:content="http://purl.org/rss/1.0/modules/content/"
  xmlns:admin="http://webns.net/mvcb/"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <channel>
        <title>Transparency News</title>
        <description>The latest news from Transparency.org (screen scraped)</des
cription>
        <link>http://transparency.org/news_room/latest_news/press_releases</link
>
        <language>en-us</language>


<?php
for ($i = 0; $i < count($matches[0]); ++$i) {
    echo "<item>\n";
        echo "\t\t\t<title>" . strip_tags($matches[2][$i]) . "</title>\n";
        echo "\t\t\t<link>http://www.transparency.org" . strip_tags($matches[1][
$i]) . "</link>\n";
        echo "\t\t\t<guid>http://www.transparency.org" . strip_tags($matches[1][
$i]) . "</guid>\n";
        echo "\t\t\t<description>" . strip_tags($matches[3][$i]) . "</descriptio
n>\n";
        echo "\t\t\t<content:encoded><![CDATA[ \n";
        echo $matches[3][$i] . "\n";
        echo " ]]></content:encoded>\n";
    echo "\t\t</item>\n";
}
?>
</channel>
</rss>

Best,
Kaspar

alex_b’s picture

Awesome work - and on top of that, the code looks really compact.

kenorb’s picture

+1

tanc’s picture

Great work Kaspar, this is the neatest implementation I've seen so far for scraping sites.

Using your code as a base I created a feed for the Drupal.org modules page, allowing people to subscribe to a feed of latest module releases for Drupal 6. I've used Feedburner to present the feed which also caches the feed. This should alleviate any load on drupal.org.

Here is the feed: http://feeds2.feedburner.com/drupal/d6modules

And here is my code:

<?php

// get page:
$url = "http://drupal.org/project/modules?filters=drupal_core:87&solrsort=ds_project_latest_release_87%20desc";
$data = implode("", file($url)); 
$domain = "http://drupal.org/";
// match entities of interest:
preg_match_all('/<div class="project-item.*?<h2><a href="(.*?)">(.*?)<\/a><\/h2>.*?<div class="by-line">By <a href=".*?">(.*?)<\/a>.*?<\/div>(.*?)<table.*?<td class="release-date">(.*?)<\/td>/s', $data, $matches);

header ("Content-Type: text/xml; charset=utf-8");
echo "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n";
?>
<rss version="2.0"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:content="http://purl.org/rss/1.0/modules/content/"
  xmlns:admin="http://webns.net/mvcb/"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <channel>
        <title>Drupal Modules</title>
        <description>Drupal modules organised by latest release</description>
        <link>http://drupal.org/project/modules</link>
        <language>en-us</language>

<?php
for ($i = 0; $i < count($matches[0]); ++$i) {
    $pubdate = date("r", strip_tags($matches[5][$i]));
    
    echo "<item>\n";
        echo "\t\t\t<title>" . strip_tags($matches[2][$i]) . "</title>\n";
        echo "\t\t\t<author>" . strip_tags($matches[3][$i]) . "</author>\n";
        echo "\t\t\t<pubDate>" . date("r", strtotime($matches[5][$i])+$i) . "</pubDate>\n";
        echo "\t\t\t<link>http://drupal.org" . strip_tags($matches[1][$i]) . "</link>\n";
        echo "\t\t\t<guid>http://drupal.org" . strip_tags($matches[1][$i]) . "</guid>\n";
        echo "\t\t\t<description>" . str_replace('href="/', 'href="' . $domain, $matches[4][$i]) . "</description>\n";
        echo "\t\t\t<content:encoded><![CDATA[ \n";
        echo str_replace('href="/', 'href="' . $domain, $matches[4][$i]) . "\n";
        echo " ]]></content:encoded>\n";
    echo "\t\t</item>\n";
}
?>
</channel>
</rss>

If anyone is offended or thinks this feed should be removed please let me know.

stoltoguzzi’s picture

beginner question
where to store this example, in a node?
or do I need some other modules