The initial creation of the feed goes well (content is imported), but cron is not causing the feeds to update. The only way to get new feed content is by manually clicking the "import" button on the feed.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Anonymous’s picture

subscribing

Sorry to ask, but I did not know that Feeds ran when cron ran. If that's true, that would be great. Are there some settings that need to be set to enable a Feed to do this? Is that what the 'minimum refresh period' setting is for?

alex_b’s picture

#1: README.txt

- Look at admin/build/feeds, double check importer settings http://skitch.com/alexbarth/nuhit/edit-feed-d6,
- Look at admin/reports/dblog - any messages by FeedsScheduler?
- Is Drupal Queue installed? Are you running cron for Drupal and for Drupal Queue? See README.txt in Drupal Queue.

Anonymous’s picture

@alex_be: Thanks for the clarification. I am running cron.php as a UNIX cron job. I don't have Drupal Queue installed, but probably I don't need it?

I think everything is running, I see this in my logs:

Type FeedsScheduler
Date Friday, February 5, 2010 - 23:45
User Anonymous
Location http://www.cachekuni.com/cron.php
Referrer
Message Finished processing schedule after 0 sec.
Severity notice

easp’s picture

I am running into the same problem with Feeds 6.x-1.0-alpha9.

Could the problem be permissions?

Eidolon Night’s picture

I have the same entry as totsubo. I don't have Drupal Queue installed either (only 4 feeds).

Eidolon Night’s picture

Installed the Drupa Queue (even read the directions), but still no luck.

Anonymous’s picture

I don't have Drupal Queue installed and Feeds *seems* to be running on schedule.

That being said I haven't had any new nodes to create. I just assume it is working because I see this message in the log file:

FeedsScheduler 02/17/2010 - 11:45 Finished processing schedule after 0 sec.

Vacilando’s picture

Version: 6.x-1.0-alpha10 » 6.x-1.0-alpha11
Priority: Normal » Critical
FileSize
14.8 KB

Well, it does not update on cron for me either.

Cron does run every three hours. I see it on the status page. Also ran it manually now.

Cleared all caches, including views, etc., all of it.

The feed is set to import every 6 hours - see the screenshot.

Yet for the past 24 hours the 8 new feed items I see in Google Reader have not been fetched by Feeds. On initial creation everything that was available got imported. But nothing by cron. :-(

EDITED: Even after 3 days, although news readers do show there are new news items, no new items are fetched using Feeds. Cron runs and finishes normally. Please let me know what can I test or do... this is a critical problem.

Anonymous’s picture

If you run the feed import manually after the initial import does it import anything new?

Vacilando’s picture

I've installed the latest dev version and it has fetched new items on cron!

Anonymous’s picture

Priority: Critical » Normal
Status: Active » Closed (fixed)

Closing issue since it's fixed for me and vacilando seems to have fixed his issue also

Izz ad-Din’s picture

Version: 6.x-1.0-alpha11 » 6.x-1.0-alpha12
Priority: Normal » Critical
Status: Closed (fixed) » Active

I have the same problem on a fresh install, with a small number of feeds. Cron interval± tried 5, 10, 15 and 30 minutes.

alex_b’s picture

Priority: Critical » Normal

#12: I can't confirm this. I'm running Feeds alpha 12 in a couple of production environments and it downloads on cron just fine. Do you have Drupal Queue installed? It's not a requirement, but it needs to be configured correctly. Could you share your PHP version, and the configuration of the importer (admin/build/feeds) you are using?

Thank you.

Izz ad-Din’s picture

Thanks for looking into this.

I'm using PHp 5.2.5 and mt configuration of feed is:

Attached to: Feed
Refresh: every 15 min
Import on create

Fetcher
HTTP Fetcher
Download content from a URL.

Parser
Common syndication parser with term extraction
Parse XML feeds in RSS 1, RSS 2 or Atom format, extract terms from titles and descriptions.

Processor
Create data table records.

However, I discovered today that it is not that items don't get aggregated, but they don't get aggregated in time;

cron runs however every 20 minutes, but some items take 1-2 hours to appear.

alex_b’s picture

How many feeds are you aggregating? Is there otherwise a lot of content on the site? Is search enabled? All these factors will contribute to a declining performance of aggregating feeds...

Izz ad-Din’s picture

I was aggegrating about 30 feeds, with about 10 from major news sites (al-Jazeera, bbc etc.).

I'm currently reinstalling and trying out elysia cron, so divide cron tasks (no need to check for updates every 15 minutes).

alex_b’s picture

#16 Couple of options:

A. Run cron more often, like every 6 minutes. You should do this either way.
B. Run cron with drush. You should do this eather way, it's faster.
C. Download and install drupal_queue module, be sure to follow its README file closely to set it up correctly.
D. Alternatively, use superfeedr http://superfeedr.com as a dedicated pubsubhubbub hub (see Feeds README file).

Use feeds head if going with option C or D.

alex_b’s picture

Status: Active » Fixed

I've added a performance and a performance tuning section to the docs: http://drupal.org/node/622698

Can we consider this issue fixed?

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

XiaN Vizjereij’s picture

Version: 6.x-1.0-alpha12 » 6.x-1.0-alpha15
Priority: Normal » Critical
Status: Closed (fixed) » Active

I ran into the same problem now. My setup worked fine for 25 feeds, but now with 35 its showing the same issues.

Setup

-> 35 Feeds
-> Cronrun every 5mins ( every cronrun is succesfull )
-> PHP_MAX_EXECUTION_TIME 180s
-> PHP MEMORY LIMIT 60M
-> No drupal_queue
-> In the watchdog it shows a lot of "FeedsScheduler ... - Finished processing schedule after 5 sec"
-> Manual Import via "Import" imports the items correctly

Help would be highly appreciated!

With best regards
Mario

alex_b’s picture

The same issues as in none of the feeds updating?

XiaN Vizjereij’s picture

Yes. No items are found on the cronrun, but they will be imported if i hit "Import" in the Feed Parent.

EDIT

Just noticed that all feeds have an "last_executed_time" and "scheduled" of 0 in the {feeds_schedule} table, even if i run the import manually via the "Import" button.

I've cleared the all drupal caches numerous times without any success.

And it might be not related to that issue ( as far as i can see it in the feeds code ), but some time ago i get my cron kind of "locked". I needed to delete "cron_last" and "cron_semaphore" {variables} table. The cron started to work again.

I can't say if that was the time the feeds started to not creating new items. The only thing i know is that, it stopped working somewhere between 25 and 35 feeds and it worked without any problems before.

I also see 6 times "No new content" underneath each other if i run the cron from the status page.

alex_b’s picture

Just noticed that all feeds have an "last_executed_time" and "scheduled" of 0 in the {feeds_schedule} table, even if i run the import manually via the "Import" button.

That's how it should be.

I also see 6 times "No new content" underneath each other if i run the cron from the status page.

That's normal.

I hate to ask this, but: Sure you ran all DB updates with update.php? Double checked refresh time on basic settings of your importer configuration?

XiaN Vizjereij’s picture

The update.php shows no available updates and refresh time is set to "Refresh: as often as possible".

XiaN Vizjereij’s picture

Component: Miscellaneous » Code
Category: support » bug

Ok, so i looked once more into that and i'm now fairly certain, that its a bug in feeds itself.

What i did today

-> I rechecked every possible setting of my drupal site twice. Everything seems fine.
-> I compared two mysql backups i did ( one before and one after this issue occurred ). The only mayor difference between these two are the number of feeds and feed items ( 24 before and 38 now )
-> I checked if every part of the cron is running correct ( search index part, feeds part, my custom part and all the other tasks ). Everything seems to work and the feeds part is running correct too ... except it doesn't create/updates the nodes. It kind of feels, that everything works fine ( getting the feed xml, checking for updated items and so on ), but it doesn't update nodes nor create them if there are new items. I can't fully verify it, but based on the time the cron is running it seems that there is no real time difference between the time it worked and now.
-> I noticed that sometimes nodes get created. Today i noticed that ONE random node was created from the feeds ( there are way more new items on that feed and on all the other feeds ). I noticed that behavior yesterday, too. It seems to import nodes from one feed for about 2-3 hours and then stopped for no visible reason. Non of the other feeds imported nodes during that time.

I'm not saying that its not possible that there are some problems on my side, but from my 15 years of programming i would say chances are 80-20 its something in the feeds code.

Edit : Issue still present with Alpha 16. I only applied the Emfield, the Inherit Tax and this patch http://drupal.org/node/625196#comment-2336760

XiaN Vizjereij’s picture

Help would be highly appreciated.

I forgot my specs in the previous posts :

MySQL database 5.0.45
PHP 5.2.13

Alex, may i send you an SQL dump + the drupal folder so you can try to reproduce the problem?

EDIT

What i did today :

I mirrored my site ( drupal directory and sql dump ) to an local installation ( the same php/mysql version and php.ini settings as the webserver ) of xampp and started debugging the feeds->cron() with Eclipse.

What i get out of it :

-> It basically confirmed my observations above : No node are created, even if there are new items on the feed. It does not update nor create nodes from it. Items are created and updated if Import is called manually.
-> After i manually F6'ed through all the functions it seemed to created the new items from at least 2 ( random, they never imported after it stopped to work ) of the 43 feeds.

As i write this 2 new items from a totally random feed got imported at the website. Its the first time since a week that new items got imported from cron.

I will stop investigating this now. Without the fundamental understanding of the underlying structure and workflow it just doesn't make sense to continue. Maybe i'm switching back to the old lady feedsAPI.

Lets hope ( of cause we're not :D ) that Alex run into this issue on his productive sites one day and can spend some hours on this one.

alex_b’s picture

XiaN Vizjereij: sorry that I can't be of more help right now. I'm running Feeds in a couple of production situations and I haven't observed this behavior...

Can you share the feeds you are aggregating?

XiaN Vizjereij’s picture

I could send you an MySQL dump and the drupal dir. Would that help?

alex_b’s picture

If you can export only the feed URLs and the importer configuration you're using that would be good. You can attach both here as a text file.

XiaN Vizjereij’s picture

FileSize
1.55 KB

Here you go. Thanks for taking a look at it.

field_feed_video_video is an emfield

$feeds_importer = new stdClass;
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'video';
$feeds_importer->config = array(
  'name' => 'Video',
  'description' => '',
  'fetcher' => array(
    'plugin_key' => 'FeedsHTTPFetcher',
    'config' => array(
      'auto_detect_feeds' => FALSE,
      'use_pubsubhubbub' => FALSE,
      'designated_hub' => '',
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsSyndicationParser',
    'config' => array(),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsNodeProcessor',
    'config' => array(
      'content_type' => 'feed_video_item',
      'update_existing' => 1,
      'expire' => '-1',
      'mappings' => array(
        '0' => array(
          'source' => 'guid',
          'target' => 'guid',
          'unique' => 1,
        ),
        '1' => array(
          'source' => 'url',
          'target' => 'url',
          'unique' => 1,
        ),
        '2' => array(
          'source' => 'url',
          'target' => 'field_feed_video_video',
          'unique' => FALSE,
        ),
        '3' => array(
          'source' => 'title',
          'target' => 'title',
          'unique' => FALSE,
        ),
        '4' => array(
          'source' => 'timestamp',
          'target' => 'created',
          'unique' => FALSE,
        ),
        '5' => array(
          'source' => 'description',
          'target' => 'body',
          'unique' => FALSE,
        ),
      ),
      'inherit' => array(
        'taxonomy' => 1,
        'user' => 0,
      ),
      'author' => 0,
    ),
  ),
  'content_type' => 'feed_video',
  'update' => 0,
  'import_period' => '0',
  'expire_period' => 3600,
  'import_on_create' => 1,
);
alex_b’s picture

Thank you - need to make some time to look at this.

ajayg’s picture

I am facing the same issue with feeds 6.x-1.0-alpha16. I have more than 100 feeds and for each cron run it seems to be processing max 5 feeds. I basically went to reports-status page and repeatedly kept pressing manual run of cron link. With every cron link it shows only 5 feeds being processed. And depending on if there is new items it will show message 10 feed items created or no new data. I have confirmed CPU or memory is no issue at all since there is no uptick on any of those stats. So it is baffling why it won't process more feeds in one cron run when plenty of CPU and memory available.

ajayg’s picture

From whatever I can tell feeds basically doing the processing in chunks of 5. SO if you have say 20 feeds, with one cron run only feeds 1-5 get fetched. Next cron run 6-10 getting processed and so on. So it gives appearance of random feed update or no update at all. But in reallity the feed you are testing may not have its turn yet.

Now question, is there a paramaeter which sets this limit to 5? Can I increase to say to 20? I have noticed so far no cpu or memory issuses with 5 feeds and can gladly try upping to 20.

UPDATE: issue resolved. As mentioned in READEME there is a hidden variable feeds_schedule_num that sets this limit after chaging this in my settings.conf I can process more feeds per cron run.

Agogo’s picture

I just want to add that I have the same problem as the original issue in this thread.
Cron doesnt update anything when there is new items in the feed.

However, when I flush cache and then run cron manually - Feeds finds the new ones and imports them.

The importer settings are:
Refresh: as often as possible

Cron runs with UNIX every 6 hours

Minimum cache lifetime is set to 3 hours

EDIT: When I set the cache lifetime on /admin/settings/performance to 3 hours the cron updates and import the new feeds as it should. So, problem solved in my case.

endiku’s picture

Version: 6.x-1.0-alpha15 » 6.x-1.0-beta4

Cron not updating any feeds. This worked a few versions back. With a 6 minute cron and feeds_schedule_num set to a high ammount it is still never updating the created nodes. The feeds_schedule_num is even set to more than the amount of nodes. Refresh is set to often as possible.

Don't really see what else there is to set to make this work. Seems broken.

Spark_man’s picture

Feeds not updating on my site either. cron is running on the hour and Min Refresh time is set to 15 minute -although I have tried various other settings and it doesn't seem to make any difference. I concur that this was working a couple of releases back but it doesn't seem to be doing much of anything at present. I am getting the message in the logs that the FeedsScheduler "Finished processing schedule after 0 sec." whenever cron runs. I only have a single feed importer specified and I can of course import feeds fine manually. But that sort of defeats the purpose. Something is definately broken here.

alex_b’s picture

Priority: Critical » Normal

I can't confirm these reports. All feeds aggregating fine on my farm of 20+ Feeds installations. #36 from which version of Feeds did you upgrade?

Spark_man’s picture

I have been following your development for some time and I probably started off with an early Alpha version. I am currently running the latest Dev version (Beta 4). I just loaded a "clean" test site on my PC (php 5.3.2, MySql 5.1.44, Drupal 6.19, Feeds 6x-1.0-beta4). I am using the SimplePie parser version 1.1.3 (I had a problem, SimplePie version 1.2 mangles hypertext links). Everything else is standard and all other modules are up-to-date. I can provide a list if you like. I can also export and forward you the settings for the one feed importer I have defined, if would help.

Spark_man’s picture

alex_b:

Question ... Does the feed have to be "published"?

Since I am displaying the feed-items with a view, I didn't see any need to mark the feed-items as "published" and likewise didn't find any added value in marking the feed node as "published". I changed the status of the Feed (not the feed-item nodes) to "Published" and now it seems to update on my test system. On my production system I got an error in the dblog: "array_intersect_key() [function.array-intersect-key]: Argument #1 is not an array in .../sites/all/modules/feeds/includes/FeedsConfigurable.inc on line 126." However, I was messing with the feed importer when cron ran, so that may have had something to do with it. I am going to wait until the next time cron runs to see if it happens again.

Spark_man’s picture

Yea! It ran, updated my lonely little feed and didn't report an error this time!!!

Spark_man’s picture

Bad news, good news. Well, I recreated the feed importer, feed and feed items in my production system and it worked sporadically overnight, and eventually stopped updating altogether. Confused, I started looking to see the differences in the database. The feed_schedule table had a 0 value in the last_executed_time field and a large number in the scheduled field on the production system, but my test system had it reversed. Also, I noticed that my production system had a queue table with a bunch of records. So, I looked and sure enough, the production system had queue api installed. I removed that module and uninstalled it completely. The queue table naturally disappeared. But, the values in the feed_schedule table were still reversed. I manually swapped the values in the last_executed and the scheduled fields and the thing updated properly on the very next cron run. I will continue monitoring the updates and report back if it stops working.

UPDATE:

After 5 cron runs, the feeds are updating as they should and looking at the feeds_schedule table, there the last_executed_time field is incrementing and the scheduled field still contains a 0. I'm not sure I trust this yet, but I will continue to monitor because that is all I can do. As my grandmother used to say "The good Lord helps those who help themselves."

I know there was some work on modifying the update methodology (last run vs next scheduled) and I suspect that somewhere in there, the code got messed up. I am hopeful that alex_b will take a closer look at this issue, because there IS a bug someplace in the code for this to be happening. If I had more time, I would dig into this further, but for now, like many others, I just need it to work.

Spark_man’s picture

UPDATE #2:

Ok, the cron messages were killing me (I had it set cron to run every 10 minutes with the refresh "as often as possible" and the expire nodes set to after 1 hour), but it was working. So, I set cron to run twice and hour and then set the refresh period to 1 hour and left the nodes expires at 1 hour. It was updating the feed item every third cron run (every 1 1/2 hours the update ran and imported items). I assume this is just a timing issue? I just set the refresh period to 30 minutes hoping to get the update to happen at least once every hour. I will try to post what it actually does tomorrow sometime.

It occurs to me that this is process is imprecise at best. But, I am at a loss as to handle this in a way that offers better control. Perhaps some better documentation - oh, and fixing the code so it puts the correct values in the correct fields would be nice too.

XiaN Vizjereij’s picture

Spark Man give http://drupal.org/node/625196 a try. It fixes the annoying array_intersect_key warnings. Dunno why this is not in the cvs yet :)

alex_b’s picture

Why do you use an expiry setting of just one hour?

From expriy setting's description:

"Select after how much time nodes should be deleted. The node's published date will be used for determining the node's age, see Mapping settings."

Spark_man’s picture

Thanks. I will give it a try, but to be honest, I only got the error that one time. Haven't seen it since.

Spark_man’s picture

To be honest, I set it to 1 hour as a test. I can't see any discernable difference no matter what the value is. In reality, this particular RSS feed is only changing once a day and only really needs to be updated right after midnight, but because both the update frequency is so loosely defined and the expiry hasn't seemed to work at all (maybe because I am not publishing the feeds items, they are being displayed though a view?), I think I just set it to 1 hour, just to see if I could get some kind of activity. Again, if I could lock in the exact time to run the feed importer, then I might need a specific time to expire the imported node. For now, hourly is working and I am reluctant to change things.

budda’s picture

Isn't this problem related to the feed items being created as an Anonymous user? Noted in comment #30 http://drupal.org/node/705444#comment-3127284 above the author id is 0.

Might be related to #656198: Node Processor silently drops nodes ?

alex_b’s picture

Version: 6.x-1.0-beta4 » 6.x-1.x-dev
Status: Active » Postponed (maintainer needs more info)

I can't reproduce this. After refactoring scheduling with #908964: Break out job scheduler I'd be interested whether these problems persist. Please upgrade and report back. Review your job_schedule tables before reporting, it contains valuable information of what's scheduled when.

imb123’s picture

Version: 6.x-1.x-dev » 6.x-1.0-beta5
Category: bug » support
Priority: Normal » Major
Status: Postponed (maintainer needs more info) » Active

Hi,

I'm hoping someone on here can help me. I have about 30 feeds that are being fetched by feeds, but on cron run it seems only a few few feeds are updated, but other feeds are never automatically updated. I have to manually import them in order to update.
My settings are to update as often as possible, and cron runs every few hours. I think i might have to add following code to settings file, but I'm not sure how it should be formatted. Could anyone include a screencap of what is should look like.

Name: feeds_schedule_num
Default: 5
Description: The number of feeds to import on cron time.
Only has an effect if Drupal Queue is *not* enabled.
http://drupal.org/project/drupal_queue

Also, is it possible that feeds times out after X seconds. I'm thinking that some feed might take a little longer to process and might just time out before completion.

Finally, I see some people have changed the Minimum cache lifetime, and I was wondering what the purpose of the function is.

Thanks so much for any help.

jscm’s picture

Priority: Major » Critical

I have the same problem.
http://drupal.org/node/917980

imb123’s picture

Hmm, now none of my feeds update on crotn, only on manual import. Does anyone have suggestions.

jscm’s picture

Title: Feed Nodes don't import on Cron with Feeds Beta7 + Job Scheduler » Feeds Node don't import on Cron with Feeds Beta7 + Job Scheduler
Version: 6.x-1.0-beta7 » 6.x-1.0-beta5
Category: bug » support

I have not the know-how to give you a better description.
I can report you log details.

I'm in shared hosting but this never has been a problem.

Cron run every 30 minutes.

on Cron, Job Scheduler log return always "Finished processing schedule after 0 sec.".
This is strange, because with Feeds Beta5 I got several seconds to complete FeedsHTTPFetcher.

In Feed Importer settings I checked "Auto detect Feeds", "Use PubSubHubbub" (without PubSubHubbub url).
I use SimplePie Parser.

In my logs, although I have 30 Feed Nodes, I find out that FeedsHTTPFetcher logs are only for 2 Feed Nodes, because there are on cron only 2 Verifying Subscriptions logs. The Feed Nodes imported are the olders.

This is all I can do for you.

Sorry for my bad english, I don't speak english frequently.

Jerry
Milan (Italy)

jscm’s picture

Title: Feeds Node don't import on Cron with Feeds Beta7 + Job Scheduler » Feed Nodes don't import on Cron with Feeds Beta7 + Job Scheduler

I hope this can help you to fixing problem.
I can assume that real problem is that something stop too soon Job Scheduler.. Maybe.

This is a CRITICAL Problem.

jscm’s picture

Version: 6.x-1.0-beta5 » 6.x-1.0-beta7

...

There is a way to Downgrade from Feeds Beta7 to Feeds Beta5 without damage
...waiting Fixing in Beta8?

jscm’s picture

Title: Feeds not updating on cron » Feed Nodes don't import on Cron with Feeds Beta7 + Job Scheduler
Version: 6.x-1.0-beta5 » 6.x-1.0-beta7
Category: support » bug

...

XiaN Vizjereij’s picture

Category: support » bug

There is a way to Downgrade from Feeds Beta7 to Feeds Beta5 without damage
...waiting Fixing in Beta8?

No.

Btw. the feeds_schedule_num has changed to job_schedule_num ( see the job scheduler readme ). Maybe setting that value to a higher than the default 5 fixes your problem?

alex_b’s picture

jscm: please follow these steps:

- back up your database.
- review job_schedule table - is it empty? if so, please report here.
- look at some of the node nids of your feeds (the nodes that contain the URLs you are downloading to drupal), try to find them in the job_scheduler's id field. Are these nids not contained in the id field? If so, please report here.

If the job schedule table contains the feed nids that should be updated:

- disable pubsubhubbub ( "Use PubSubHubbub").
- update the job schedule table and reset the next schedule time by running UPDATE job_schedule SET next = 0; on it.
- run cron several times, observe whether feeds are updating.
- report results here.

Thank you!

jscm’s picture

FileSize
255.07 KB
237.76 KB
284.21 KB
256.48 KB

Hi,
I have explored DB tables, using phpMyAdmin, and I found out that job_schedule table is empty.

So, I attach Screenshoots from phpMyAdmin output about tables, feed_nid, feed_source, job_schedule and other.. for your analyses.
If you need tables exported please ask me.

I have several websites with equal settings, modules and functions. This problem is in all sites, but there are sites where feeds are imported only from 3-4 Feed Nodes (the oldest), and sites where feeds are imported from only 2 Feed Nodes (the oldest).

In the DB here reported by phpMyAdmin Screenshoots, are imported feeds from only 2 Feed Nodes (the oldest) that you can see also in feed_push_sub.png.

Thank you for you support.

best regards

jerry
Milan (Italy)

alex_b’s picture

jscm:

Try this on your Feeds beta 7 installation:

- Go to the importer on admin/build/feeds that you use for the feeds that aren't aggregating right.
- Turn of pubsub support.
- Change the "Minimum refresh period" to some other value then the current one.
- Change the "Minimum refresh period" back to the value before you changed it.
- Run cron.
- Review job_schedule table, there should be entries now.

This should cause Feeds to reschedule all your subscriptions.

alex_b’s picture

jscm: for the record, what I think has happened is that you ran into #916962: Fix undefined feeds_reschedule() when upgrading (feeds not updating on cron) and Feeds failed to set the 'reschedule' flags when you upgraded from beta5.

jscm’s picture

FileSize
133.89 KB
250.72 KB

Hi Alex,
I followed your instructions. and now I see that job_schedule table is populated (see attachment).

I Turned Off pubsub support, but Now Can I Turn it On? (see attachement)

The PubSub table although Pubsub is off, is always populated.

I see that Feed Node Items are imported and job_schedule is updated.

Do you need more reports and informations?

alex_b’s picture

Status: Active » Closed (duplicate)

Great, you can turn pubsubhubbub back on, I will close this issue as a duplicate then.

The fix to #916962: Fix undefined feeds_reschedule() when upgrading (feeds not updating on cron) will completely reschedule all enabled importers.

jusyjim’s picture

Version: 6.x-1.0-beta7 » 6.x-1.0-beta8
FileSize
38.74 KB

Tuning in late... but I am having the same problem with the cron and job scheduler not updating the (3) feeds. Manually importing works fine. I have a question regarding the job scheduler hidden settings and something alex_b said in #56 above: Are the hidden settings for the job scheduler inherent in the module (in other words, is installing the module all that needs to be done and it works on it's own with the feeds module) or do the hidden settings actually need to be written into the settings.php file?

I've attached a screen shot of my importer settings.

sHaDoW_LiGhT’s picture

Hello,

I have also the same problem, the importation is done the first time I add a new feed, then nothing. For some feeds I can load new contents by clicking manually on import and for other even when I click on import I have the message "There is no new content". The source feeds are daily newspapers.

wwhurley’s picture

This is due to the fact that feeds_cron calls feeds_reschedule(FALSE) at the end. As a result later calls to feeds_reschedule will always return FALSE and thus no sources will ever get scheduled. To make it work comment out line 38 in feeds.module:

//feeds_reschedule(FALSE);

alex_b’s picture

#65

This is due to the fact that feeds_cron calls feeds_reschedule(FALSE) at the end.

Untrue.

feeds_reschedule() reschedules all feeds on all cron runs. I wonder how you fix anything with this. Feed Sources / Feed Importer only need to be rescheduled if their import interval changes. Once scheduled, they stay scheduled until removed.

One problem that could occur is that the rescheduling routine on cron craps out and does not reschedule all feeds.

How many feeds are you tracking - i. e. how long is your feeds_source table?

sHaDoW_LiGhT’s picture

I have only 4 feeds. As this problem occured after I upgraded from beta5 (before the use of job scheduler) to beta7. I suspected the module version. So I deleted all my feeds, unistalled the module, deleted all files and reinstalled beta5 but always the same problem.

N.B : I use the Elysia Cron module

wwhurley’s picture

Status: Closed (duplicate) » Needs work

In feeds_cron() it has the following:

function feeds_cron() {
  if ($importers = feeds_reschedule()) {
    foreach ($importers as $id) {
      feeds_importer($id)->schedule();
      $result = db_query("SELECT feed_nid FROM {feeds_source} WHERE id = '%s'", $id);
      while ($row = db_fetch_object($result)) {
        feeds_source($id, $row->feed_nid)->schedule();
      }
    }
    feeds_reschedule(FALSE);
    return;
  }
}

This calls feeds_reschedule() that has

function feeds_reschedule($importer_id = NULL) {
  $reschedule = variable_get('feeds_reschedule', FALSE);
  if ($importer_id === TRUE || $importer_id === FALSE) {
    $reschedule = $importer_id;
  }
  elseif (is_string($importer_id) && $reschedule !== TRUE) {
    $reschedule = is_array($reschedule) ? $reschedule : array();
    $reschedule[$importer_id] = $importer_id;
  }
  variable_set('feeds_reschedule', $reschedule);
  if ($reschedule === TRUE) {
    return feeds_enabled_importers();
  }
  return $reschedule;
}

So what happens if FALSE is passed to feeds_reschedule it sets the variable to FALSE, and unless $reschedule is TRUE it return FALSE. So once variable_get('feeds_reschedule', FALSE) return FALSE, nothing happens. So when feeds_reschedule returns FALSE feeds_cron short-circuits and does nothing. Since it's the only thing that implements hook_cron, if nothing happens in feeds_cron, nothing happens.

Let's walk through this. feeds_cron calls feeds_reschedule with FALSE. This sets the variable feeds_reschedule to FALSE. If feeds_reschedule is not TRUE (i.e., is FALSE) it does not call feeds_enabled_importers and returns FALSE. As a result $importers in feeds_cron is FALSE and feeds_cron exits without doing anything. The scheduler is never called, so the importers and processors are never triggered.

alex_b’s picture

Status: Needs work » Closed (duplicate)

Since it's the only thing that implements hook_cron, if nothing happens in feeds_cron, nothing happens.

Not true. Import and expiry jobs are dispatched by job_scheduler_cron(). feeds_cron()'s responsibility is only to reschedule all jobs after certain events like for instance the import period of an importer is changed.

sHaDoW_LiGhT’s picture

Okay but do you have an issue to fix that ? when I try to import a feed I get the message "There is no new content" even if the feed has new content !

gigantortron’s picture

I'm having the same issue.

My setup involves about 390 feed nodes where each feed node must fetch content from a separate XML feed (identified by its ID).

When I run cron only 5 of my feeds are updating at a time.

kenorb’s picture

pgrond’s picture

Status: Closed (duplicate) » Needs work

I think whurleyf1 is right here. The JobScheduler does a job for all jobs returned by the query

$result = db_query_range("SELECT * FROM {job_schedule} WHERE scheduled = 0 AND next < %d ORDER BY next ASC", JOB_SCHEDULER_REQUEST_TIME, 0, $num);

So if scheduled != 0, the job will not run.

The only place scheduled is set to 0 is in the set method of the JobScheduler, and that is only called from FeedsSource->schedule() and FeedsImport->schedule(). And those are only triggerd if feeds_reschedule returns importers.

pgrond’s picture

Status: Needs work » Closed (duplicate)

Ok, sorry I was wrong. It is working and the entry stays on 0 after a succesful import. It fails after an import that did not finish succesfuly though.

PixelClever’s picture

I found this same issue of feeds not importing anymore after update and I fixed it by going directly into the database and setting the 'last' and 'next' fields to 0 in the job_schedule table. I think this indicates that there is something else going on other than what is described on the other issue. In any case, I am posting this here so that other people who are struggling with this will see that there is another possible solution.

nguyentran’s picture

Hello,

when everything work well until some days It has this problem, you can see in the status of importer, the last import zero second (set value for periodical).That mean something wrong, check download content from a URL, check cron. Last dont forget to see the import link or import file in /sites/default/files/feeds.

Hope this help someone.

saurabh.tripathi.cs’s picture

Issue summary: View changes

Using below modules:
Views 7.x-3.7
Entity API 7.x-1.2
feeds 7.x-2.0-alpha8

Feeds are not importing on cron..Below is my job_schedule table:

+---------+---------------------+------+-----+--------+---------+------+--------+---------+------------+----------+------------+-----------+
| item_id | name | type | id | period | crontab | data | expire | created | last | periodic | next | scheduled |
+---------+---------------------+------+-----+--------+---------+------+--------+---------+------------+----------+------------+-----------+
| 15165 | feeds_source_import | feed | 408 | 0 | | NULL | 0 | 0 | 1438235549 | 1 | 1438235549 | 0 |
+---------+---------------------+------+-----+--------+---------+------+--------+---------+------------+----------+------------+-----------+

I tried to set last and next to zero and ran the cron but still no success.Any help would be great.
Thanks.