Closed (fixed)
Project:
Leech
Version:
5.x-1.6
Component:
General
Priority:
Normal
Category:
Feature request
Assigned:
Unassigned
Reporter:
Created:
23 Apr 2007 at 08:22 UTC
Updated:
30 Aug 2009 at 20:51 UTC
Hi,
I need to create nodes from news in HTML-pages. So instead of repeatedly fetching news items from a feed, I'd like to repeatedly inspect a certain HTML-page, extract (maybe using regular expressions) news from it, and convert these to nodes.
I'd like to try to implement this and, if possible, would like to extend module Leech. Therfore: Do you think this is doable? Maybe by writting a new parser (if I am not mistaken, Leech can use different parsers).
I'd like to get some feedback by a Leech developer before I start, to coordinate with others and hear of the views of others.
Thanks,
Kaspar
Comments
Comment #1
alex_b commentedYou don't need to touch leech for doing that. Write a screenscraper that produces an RSS feed. Subscribe then leech to that feed.
I bet that there are services like that out there. Let me know what you find.
Comment #2
kaerast commentedI've used Dapper in the past for this and there are indeed many other systems like it, it's far more sensible to use an external service like this because they have the time to spend coding a scraper.
Comment #3
hbfkf commentedAlex_b and kaerast, thanks a lot for your hints. I'll get back when I have taken a look at Dapper.
Best,
Kaspar
Comment #4
alex_b commentedOk, I ll close this issue for the time being, feel free to reopen it when you want to follow up here on scraping html pages.
Comment #5
alex_b commentedeem.
Comment #6
hbfkf commentedWith your help I managed to screen-scrape the HTML-page to an RSS feed and use leech to import the news items into my system. Worked like a charm!
For the sake of completeness (maybe somebody else can use it as a starting point), here's my little screen scraper.
Best,
Kaspar
Comment #7
alex_b commentedAwesome work - and on top of that, the code looks really compact.
Comment #8
kenorb commented+1
Comment #9
tancGreat work Kaspar, this is the neatest implementation I've seen so far for scraping sites.
Using your code as a base I created a feed for the Drupal.org modules page, allowing people to subscribe to a feed of latest module releases for Drupal 6. I've used Feedburner to present the feed which also caches the feed. This should alleviate any load on drupal.org.
Here is the feed: http://feeds2.feedburner.com/drupal/d6modules
And here is my code:
If anyone is offended or thinks this feed should be removed please let me know.
Comment #10
stoltoguzzi commentedbeginner question
where to store this example, in a node?
or do I need some other modules