I am using Calais with feeds, and I know there is initially taxonomuy created. The question is will my server become overloaded with say 70k feeds? In my past experiances with feedapi I know my server became somewhat bogged down and I am not sure if it was because I had such a large number of feeds on chron jobs to run, or because I set them so they would ne ver expire.

Thanks for the input!

Also, if you have a feed in the nodequeue, and it expires after one day, will it still remain in the nodequeue?

Thanks !

Comments

alex_b’s picture

alex_b commented
Title: If feeds never expire, will the server be overloaded? Is there any interaction with capatured sources after initial scraping ? » Aggregate 70k feeds?
Priority: Critical » Normal

70k feeds is a lot and you *will* experience performance problems. You can use Drupal Queue to mitigate the issue somewhat http://drupal.org/project/drupal_queue but basically, you're in for a research project. You may want to consider using sth else than Drupal (a CMS like framework!) for aggregation. You may be able to use PubSubHubbub with Superfeedr - see http://developmentseed.org/blog/2010/feb/23/pubsubhubbub-support-drupal

Also, if you have a feed in the nodequeue, and it expires after one day, will it still remain in the nodequeue?

No.

PS: Please don't use "critical", we use "critical" on this queue for release planning, to mark issues that will be addressed for the next release.

AntiNSA’s picture

AntiNSA commented

Ok, I guess I need to rephrase .... I had aggregated about 300 feeds creating 70k feed items from those feeds....... Would this still cause issues that taxes out drupal?

alex_b’s picture

alex_b commented

300 feeds should work much better. You may find that it will be hard to refresh them in a high frequency (e. g. every 30 minutes).

You can step up refreshing frequency by using and tuning Drupal Queue (see its readme.txt for how to set it up) or again, by using a dedicated pubsubhubbub hub like Superfeedr.

Note that when you step up refresh frequency by using drupal queue you will have more feeds processing simultaniously which will mean more server resources allocated to feed processing at any given time.

AntiNSA’s picture

AntiNSA commented
Assigned: AntiNSA » Unassigned
Priority: Normal » Minor
alex_b’s picture

alex_b commented
Status: Active » Closed (duplicate)

Please see #754626: Performance - max number of feeds