Missing import tab and feeds aren't updating at the assign time [#1705930]

I have two issues with the feed module for one of my sites that I am working on.
1) When I create a node type that holds the feed URL, the import menu tab is missing.
2) I set my feed importer to update every 30 minutes or as often as possible and the feeds are coming in 2-5 days later.

Here's a description about what I did and the problem I saw:

Things I have installed:
-pressflow
-memcache
-supercron

I created over a hundred feed importers. Here are my settings on each of them.
Basic Settings:
- Minimum import date: set to 30 minutes
Node Processor->settings
- Expire nodes: never
- Don't update existing node content
Node Processor->Mapping:
Source Target
Title<----> Title
Description<----> Body
Feed node: Taxonomy: Category<----> Taxonomy: Category
Title<----> GUID
Item URL (link)<----> Feed Link (URL)
Published date<----> Published date

I am able to create the node that contains the feed source. The new feed item gets imported fine. I noticed that the feed import menu tab is missing for each node. At first I thought the feed module wasn't working because nothing new was getting imported. After a couple of days, I noticed that the feeds were getting updated. But it wasn't happening every 30 minutes. It took a couple of days for each feed importer to update. None of the feed importers seem to update at the same time. Do you think this is because I created a lot of feed importers. I don't think it's a cron issue because it does update. I checked the supercron settings and the average execution time is 0 seconds, which is kind of strange.

I have this installed on another site and it works fine. For the site that works, the import menu tab is present and the execution time is under a minute.
I did install a different version of the feed module. The version is 6.x-1.0-beta12.
I rather not update the module if I don't have to. But if you think it would fix my problem, I guess I would have to try it.

So I'm not sure why it takes a couple of days to import new feeds. I read another post that the amount of feed importers could be an issue, but I don't feel like this would cause the problem at hand. I'm also not sure why I can't re-import feeds. I think this will help update the RSS feeds. Could this be a permission issue? I checked everything in the permissions and I still can't see the import menu tab. I'm wondering if the missing import tab item and the delayed feed import are related somehow?
I have been working on this problem for about a week and don't have any leads about how to fix this problem.

Comments

Comment #1

jbunyan CreditAttribution: jbunyan commented 31 July 2012 at 00:59

I just noticed something about the feed importer. So it looks like the feeds didn't update until the 3rd day. It would only update one feed at a time per hour. So let's say the feedsource_1 updated at 5am. Then feedsource_2 imported feeds at 6am. This kept going on every hour until today at 7am. So each hour represented each feed importer I created. It hasn't updated since then. I'm not sure when it will start again.

Comment #2

twistor CreditAttribution: twistor commented 31 July 2012 at 02:40

Why on earth did you create over a hundred importers with the same configuration?

Comment #3

jbunyan CreditAttribution: jbunyan commented 31 July 2012 at 15:19

I had to create different users names for each feed source. So all of the feed importers are suppose to create a node called feed item. Base on the feed source, there will be a different user creating the feed item. I wasn't sure if I could do it with one configuration.
If there was a way to assign each author with out creating hundreds of importers, then that would be better.

Comment #4

twistor CreditAttribution: twistor commented 31 July 2012 at 15:35

You can map something to the user. In dev, you can map to user id, username, user email.

Comment #5

jbunyan CreditAttribution: jbunyan commented 31 July 2012 at 16:46

Ok I didn't realize that.
I do see
Feed Node: User ID
so I guess that would map to user ID. Ok that make sense. I created a dev site and only create a few feeds and it's still producing the same problem. So at least I know it wasn't the amount of feed importers I created. I'm trying to run some more test to see whether it's the server, feeds module, or other conflicting modules.

Comment #6

jbunyan CreditAttribution: jbunyan commented 1 August 2012 at 17:20

So I don't think it's the module version because I tried this module on other sites and it works. I did notice something. I am unable to uninstall the module. When I try to uninstall the module, it keeps taking back to the "Confirm Uninstall" page. This happens to all of the modules that I try to uninstall. There was an early version of the feeds module install before. I'm not sure what version it was. I did try to use update.php, would that replace the old database with the current feed module. Whether or not the old module was installed in the database, the import menu tab still doesn't show. I don't think this has anything to do with the database since the information about the import tab is inside the module. So shouldn't it show the import tab? I'm basically trying to remove all traces of the feed module and install it fresh.
I also removed multiple modules and changed the theme, and the 2 problems I mention above still persist.

To summarize, would the problem with uninstalling a module affect the feed module to update the database?

Comment #7

jbunyan CreditAttribution: jbunyan commented 27 August 2012 at 23:08

I found out why the import tab wasn't showing. The import tab showed up after I disabled the node gallery module. Node gallery has an import tab as well, so I'm pretty sure that it was overwriting the feed import tab.
I still have a problem with cron not working. I tried to run cron manually multiple times, but nothing get's created. Feeds are only created when I click import. Any idea what could be causing cron to run create these feeds? I think the feeds are still updating irregularly as I mention above.
My only guess is that whatever interval you choose, the time is different in the database.

Comment #8

jbunyan CreditAttribution: jbunyan commented 29 August 2012 at 23:21

I think I found out the reason why cron was updating irregularly.

---------------------+----------------------------------+--------+------------+--------+------------+----------+-----------+
| callback | type | id | last | period | next | periodic | scheduled |
+---------------------+----------------------------------+--------+------------+--------+------------+----------+-----------+
| feeds_source_import | feed | 313625 | 1345834802 | 0 | 1345834802 | 1 | 0 |
| feeds_source_import | feed | 313480 | 1345822201 | 0 | 1345822201 | 1 | 0 |
| feeds_source_import | feed | 313497 | 1345822201 | 0 | 1345822201 | 1 | 0 |
| feeds_source_import | feed | 313452 | 1345824001 | 0 | 1345824001 | 1 | 0 |

There are over 300 of records like this in the job schedule table. I removed all of these, and cron ran better. I had to run it a couple of times in order get all of the feeds to be pulled. This was on the dev site so I manually deleted these records from the database. Is there a better way to do this than using a mysql command. I'm not sure there is. I didn't see a problem afterwards, so I don't see this as an issue in the future. Cron still doesn't pull all of the latest feeds in one cron job. I'm not sure if this is possible with cron job. I'm guessing cron times out. Is it possible to increase the cron job time so that it doesn't timeout?

Comment #9

jbunyan CreditAttribution: jbunyan commented 31 August 2012 at 16:59

I deleted all of the unnecessary job schedule records. I realized that every cron job pulls in 5 feed sources at a time. So basically 5 records in the job_schedule table runs per cron. Is there a way to increase this value? I'm not sure if it's the feed module or the job_schedule module that limits the amount of feeds pulled per cron.
I have about 100 feeds so I guess it would take 20 cron jobs to pull in all of the feeds. One more thing to note, I did increase the php execution time for the server. IT still pulls in 5 feed sources per cron.

Comment #10

hanksterr7 CreditAttribution: hanksterr7 commented 20 January 2014 at 22:02

I had a similar issue with the Import tab not showing. Something was stepping on it (not sure what, with all the modules I have installed). Solved it by renaming import to import1 in one spot in feeds.module, hook_menu(). See https://drupal.org/node/2161343

  $items['node/%node/import1'] = array(
    'title' => 'Import',
    'page callback' => 'drupal_get_form',
    'page arguments' => array('feeds_import_tab_form', 1),
    'access callback' => 'feeds_access',
    'access arguments' => array('import', 1),
    'file' => 'feeds.pages.inc',
    'type' => MENU_LOCAL_TASK,
    'weight' => 10,
  );

Comment #11

hanksterr7 CreditAttribution: hanksterr7 commented 21 January 2014 at 00:41

In my site, I have limesurvey_sync module as well as feeds. I think they are stepping on each other for node/%node/import:

In feeds.module there is:

function feeds_menu() {
  $items = array();
...
  $items['node/%node/import'] = array(
    'title' => 'Import',
    'page callback' => 'drupal_get_form',
    'page arguments' => array('feeds_import_tab_form', 1),
    'access callback' => 'feeds_access',
    'access arguments' => array('import', 1),
    'file' => 'feeds.pages.inc',
    'type' => MENU_LOCAL_TASK,
    'weight' => 10,
  );

and in ls_survey.module there is

function ls_survey_menu() {
  $items = array();
...
  $items['node/%node/import'] = array(
    'title' => 'Synchronization',
    'page callback' => 'drupal_get_form',
    'page arguments' => array('ls_survey_import', 1),
    'access callback' => 'ls_survey_import_access',
    'access arguments' => array(1),
    'type' => MENU_LOCAL_TASK,
    'weight' => 15,
    'file' => 'ls_survey.import.inc',
  );