Aggregation module with Drupal 6.3. Recently, I notice that aggregation_item from WordPress2.6 RSS2.0 (.../wordpress/wp-rss2.php) is duplicated. Core aggregator from the same WordPress blog works normaly.

How to avoid duplication?

Comments

Ashraf Amayreh’s picture

Ashraf Amayreh commented

Could you provide me a URL to an example feed that shows the duplication problem?

Diseree’s picture

Diseree commented

I've sent you URL of test site from contact form. Please check it out.
Test site is new installation of Drupal6.3 and Wordpress2.6.
When I run cron manually twice, duplicated feed_item is registered.
I do not remember well, this phenomenon was not occur with WP2.5.1.
It may be the problem of RSS2.

kkuphal’s picture

kkuphal commented

I'm seeing duplicate imports using this RSS feed:

http://images.apple.com/trailers/rss/newtrailers.rss

chrissearle’s picture

chrissearle commented

Same here on this one (again - drupal 6.3):

http://test.longship.org/node

Feed is http://www.speiding.no/rss/

Diseree’s picture

Diseree commented

I've checked the PHP code of aggregation.module. This module decide whether the fetched RSS is new or not by checking the return code from httpd. If httpd returns 200, this module makes "INSERT INTO" query. If httpd returns 304, this module assumes that the RSS is unchanged since last fetching. So the duplication depends on the behavior of httpd. This implementation is simple and lightweight, but not accurate.

If this module makes query to check duplication before INSERT, the performance may be too slow. But I want this function.....

chrissearle’s picture

chrissearle commented

It would be nice if we could rely on that - but - as you say - this can't be relied on. Could we do whatever it is the in-built aggregate module is doing? I know it isn't creating nodes - but it is creating rows in a table.

chrissearle’s picture

chrissearle commented

Just found i forgot to disable the module on a test site :)

200 pages of feed items in the admin content page (thousands of records).

Glad I found http://drupal.org/node/136740 (change product with aggretation_item)

jkopel’s picture

jkopel commented

I spent a long time looking at the code since I too was getting duplicate items.
I think that I found the problem, but I am not sure of the best way to fix it.
When the feed_items are stored they have a guid assigned to them which is created using sprintf('%u', crc32())
I think there is one too many of these and the end result is that the value stored is the CRC of a CRC.
So when the feed is read a second time the guids stored in the feed_items are never equal to the guids generated from the feed, and new items get created.
I solved the issue by commenting out line 844,

$node->story_guid = sprintf('%u', crc32($node->story_guid));

but I am not sure this is the best place to fix it.

Diseree’s picture

Diseree commented

Feed_Api module can resolve this issue. Feed_Api does not require cURL library and modification of php.ini (but simplepie php library is mandatory).

oriol_e9g’s picture

oriol_e9g
Primary language Catalan
Location Barcelona
commented
Category: support » bug

Same problem duplication with this feed:

http://www.bastoners.info/rss.php?summary=1&type=postslist

LeonidShamis’s picture

LeonidShamis commented

Here is another example of the feed that results in duplicate feed items created: http://feedrinse.com/services/rinse/?rinsedurl=b588d7fce0289499c490fcdbb...

Lucky I didn't promote feed items to the front page by default so I had an opportunity to notice this behaviour when retrieving new content (feed items) created after a couple of cron runs, each adding the same feed items :(

I have selected Aggregation module because it supports images and I wanted to be able to fetch feeds including images...
I wanted to use FeedAPI at the beginning, but its companion module Feed Element Mapper was not available for Drupal 6. I have just noticed Alpha release, so I'm going to test it in combination with
Embedded Media Field.

Ashraf Amayreh’s picture

Ashraf Amayreh commented
Status: Active » Closed (fixed)

Fixed in 1.1 due to come out today