Keep exceeding cron time with FeedAPI

You could try to reduce the cron interval.

I think it would be useful to have the possibility to create groups of feeds to be run indipendently.
Eg. feeds frequently updated should be run every 30 minutes, feeds updated once a week could be run every day (or even every week).

Moreover, it would be useful a "multi-threaded" version of cron, but I think this is not a FeedAPI-only issue.

Hey there, I also seem to have this problem, as it seems nearly impossible to use feedAPI and Cron on my site. I don't have much feeds but it seems having max_execution_time at 300 and running cron every 45 minutes is not good enough.

When I run cron it says that it's already been running for more than an hour, and then I get an error 500.

Is there anything we can do to debut the feedAPI cron? For example how can we see how many seconds it would need to refresh, etc

thanks,
Patchak

"set feedapi to be 75%" - In fact if you get those timeout messages, the good way is to decrease feedapi percentage and increase the php max execution time.
Maybe you have such a feed that takes too much time to download, i just guess.
What other modules do you have that do something in cron time?

There is something weird, I set up the "php_max_execution" time at 1200 seconds and still in the feedAPI settings I only see that feedAPI will try to run at much less time.

"Percentage of maximal PHP execution time (currently 240 seconds). At current settings, the FeedAPI cron process can run for up to 120 seconds."

I should get other values here no?

Even after cleaning cache and refeshing browser, no changes.. is there a variable in the variables table that would need to be reset??

Or maybe this is not related to the php max execution time, but to another value like how much time php can communicate with the web server??

Patchak

I tried also to setup the max_execution_time directly in the settings.php for the site (I use a multisite install) but still, feedAPI does not recognize the new value of php max execution... Maybe there is a variable to reset?

Patchak

Well, Drupal behaves nasty in this question :)
http://drupal.org/node/375578
So in common.inc, there is a hard limit, 240 seconds for cron!! You need to modify this value by hand, it seems.

I have tried increasing the max_execution_time in php.ini, by adding the instruction to settings.php, and by changing the limit in common.inc. Drupal 6.9 still reports 240 seconds as the max time. I have also tried increasing the percentage of time for feedapi. Nothing seems to make this error go away.

I am working on a small, new site with one feed from blogger.com that has about 300 posts. Even with the apparent ceiling of 240, despite all of the changes I have tried, cron seems to crash within the first 60 seconds. The run starts on the hour and feedapi is the last logged module, still in the first minute. So, I do not know what is causing the problem, but it does not appear to be that all of the time is being used up.

subscribing, same issue

Yes I have the same problem as my site is pulling many feeds. Problem seems that it takes a long time for all the feeds to be pulled. I'm running on 75% as well. I used to run the cron every 30 minutes, now will try to run every 20 minutes. I'll see if my feeds gets updated fast enough. Should I decrease it even more?

Anyhow, definitely a thread I need to follow up...

My problem doesn't seem to be related to allowed time of execution - cron just hangs forever until I kill the process. I set up some logging for cron to find out which module was causing it and FeedAPI was never returning. I only have 9 feeds on my site - so I don't see why there should even be a problem.

Sometimes it works fine..most of the time it doesn't...

Any suggestions? How would you know if it was a time execution problem?

Thanks

subscribing; same issue

feedapi 6.x-1.6, simplepie 1.1.3, feedapi_node enabled

See feedapi_cron()

FeedAPI takes 100 (FEEDAPI_CRON_FEEDS constant) feeds at a time and updates them. After each time, it checks whether there is still time available on cron. It seems that in the cases that are being described here, cron hangs because PHP times out while FeedAPI is somewhere in a 100 feeds patch.

Can you lower the number of feeds drastically by setting FEEDAPI_CRON_FEEDS to say, 10 and see whether the problem persists?

I have 9 feeds on my site. And updating hourly, there aren't many items to grab.

mikesteff: are these 9 feeds really big? Can you make sure that there is no other module enabled that consumes a lot of cron time? Usual suspect would be search...

No they aren't and being updated hourly there aren't many items to grab per cron run. There aren't a lot of other modules enabled - more importantly, ones that have large cron jobs. If cron time exceeded wouldn't it just end and return - not stay stuck up and unresponsive? Cron hangs and won't run again until I kill the process manually and delete the semaphor.

I had this problem on my site and changed to use drush to execute the cron. See the drush project for more info:

http://drupal.org/project/drush

Since I switched I haven't had any time exceeded errors.

I am suspecting that people are reporting different issues with the same symptom: FeedAPI timing out on cron.

This patch might help in cases where the 100 feeds per batch on cron is too big. It introduces a variable that allows for adjusting the number of feeds per batch.

E. g. set it to 20 and see whether the problem persists.

I have 9 feeds.

I changed the cron value to 480 in common.inc but FeedAPI doesn't see! still shows 240.

See common.inc line 2622 (drupal_cron_run). This is the reason that it still shows 240.

I know, doesn't matter though what it shows. important thing is what PHP memory limit is. (in php.ini)

I am gonna try php memory limit in .htaccess. I have the same problem. But, I am pulling huge feeds from several new sites like the LATimes and Huffington Post etc...

I have it set to 32m at the moment but I'll try 128m and see what happens?

Plus I might install Drush.

http://piggypalace.goodtimessociety.com/

I also have Elysia Cron http://drupal.org/project/elysia_cron installed.

I set that to run feed api at every five minutes. I cranked up feed api to 75% and I set .htaccess to php 128m.

(Plus, I disabled Urllist which always times out. And, I disabled Google Analytics which seems to time out too.)

AND... I'll tell you in a minute...

NOTHING!

brb

And, I used this patch:

define('FEEDAPI_NEVER_DELETE_OLD', 0);
define('FEEDAPI_TIMEOUT', 1);
// Number of feeds to process for each step in cron.
-define('FEEDAPI_CRON_FEEDS', 100);
+define('FEEDAPI_CRON_FEEDS', variable_get('feedapi_cron_feeds', 100));
// Minimum time that must elapse before a feed can be refreshed again on cron.
define('FEEDAPI_CRON_MIN_REFRESH_TIME', 1800);
// Prune FeedAPI stats 4 weeks

and changed number of feeds to 20 like this

define('FEEDAPI_NEVER_DELETE_OLD', 0);
define('FEEDAPI_TIMEOUT', 1);
// Number of feeds to process for each step in cron.
-define('FEEDAPI_CRON_FEEDS', 20);
+define('FEEDAPI_CRON_FEEDS', variable_get('feedapi_cron_feeds', 100));
// Minimum time that must elapse before a feed can be refreshed again on cron.
define('FEEDAPI_CRON_MIN_REFRESH_TIME', 1800);
// Prune FeedAPI stats 4 weeks

in feed api module.

Now we wait...

OK. I did all that. I removed Elysia Cron just in case. I set .htaccess to 128m. I used the patch and set Feedapi to get only 20 feeds per cron. I set cron to run every 15 minutes and I set Feedapi to take up 75% of cron time.

Now we see...

http://piggypalace.goodtimessociety.com/

And, I also disabled Urllist and Google Analytics.

These two modules seemed to time out all by themselves anyways.

And, it worked!

@hotlikecold (and who is using Elysia cron)

I'm the maintainer of Elysia Cron.

An old version of Elysia Cron (feb.2009) was handling cron_semaphore in a way that conflicted with FeedAPI (http://drupal.org/node/382796)
Now that issue is solved, and there should be no more problems using the two modules togoether.

However if you think the problems is using feedapi with elysia_cron i can investigate further.

Are you using Elysia Cron by calling standard drupal cron or by calling elysia_cron/cron.php (advanced install)?

@hotlikecold (and who is using Elysia cron)

I've just released a new version of Elysia Cron with the a settings to change the time limit for job execution (and other improvements).
Try to adjust it to match the needs of feedapi.