Problem/Motivation

Under certain circumstances content gets imported via a manual import, but not when running cron.

Proposed resolution

Unknown. The cause of the issue should be examined first. The problem has been reported in multiple issues, so these issues should be sorted out first to gather clues. Perhaps there is more than one cause. When the cause is known, it would be best if the bug could be replicated with an automated test. This makes it easier to find out if a proposed fix really fixes the issue or just partly.

Remaining tasks

  1. Sort out the issues that reported this problem (and close duplicates).
  2. Find the cause of the issue.
  3. Replicate the issue with an automated test, if possible.
  4. Propose a fix for the issue.

Temporary Solution

The temporary solution is to manually kick off the feed import with a custom hook_cron(). For feeds not attached to nodes, the code will look like this:

function MODULE_NAME_cron() {
  $name = 'FEED_NAME';
  $source = feeds_source($name);
  $source->import();
}

For feeds that are attached to nodes, you'll need to load a list of node ids then run the import on each one:

function MODULE_NAME_cron() {
    $node_type = 'CONTENT_TYPE';
    $feed_type = 'FEED_NAME';
    $query = new EntityFieldQuery();
    $query->entityCondition('entity_type', 'node');
    $query->entityCondition('bundle', $node_type);
    $result = $query->execute();
    $feed_nids = array_keys($result['node']);
    foreach ($feed_nids as $feed_nid) {
      $source = feeds_source($feed_type, $feed_nid);
      $source->import();
    }
}

Related issues

These issues may be reports of the exact same problem:

These issues look like a different problem, but are related to cron too:

Closed cron issues:

Original report by tamarackmedia

I am importing a CSV via a URL (HTTP Fetcher), set to periodic import "as often as possible."

I had successfully used the exact same setup to import via cron on another site running Feeds 7.x-2.0-alpha7.

Manual import works fine on the new site (running latest release, 7.x-2.0-alpha8), but nothing happens on cron. I rolled back to Feeds 7.x-2.0-alpha7 and it works fine.

Files: 
CommentFileSizeAuthor
#13 feeds_periodic_import.png20.72 KBdineshw

Comments

twistor’s picture

Status: Active » Postponed (maintainer needs more info)

I just setup an importer to run as often as possible with alpha7, upgraded to alpha8, and it works fine.

Can you provide your configuration?

essenceofginger’s picture

I'm seeing the same thing. Other than the 'periodic import' setting, I can't figure out how else I should be configuring the import to run on cron. Driving me nuts!

twistor’s picture

Issue summary: View changes
Status: Postponed (maintainer needs more info) » Closed (cannot reproduce)
ajayg’s picture

I am facing this issue and driving me nuts. I have a drupal multisite and it used to work for both sites and now it stopped only for one site.
Manual import works fine on both site but through cron works only for one site. On site where it does not work, The logs says cron ran and job scheduler processed n records. But the feed logs does not reflect new items created. No other error.

ajayg’s picture

Status: Closed (cannot reproduce) » Active
metabrown’s picture

+1 upgrade to Feeds 7.x-2.0-alpha8 broke cron updating, rolling back to alpha7 fixed the issue.

ajayg’s picture

Confirming the multisite which was upgraded , cron does not work. But using same multisite code , a new site created on scratch, it works fine. So same code is bahaving differently which points to some database or variable discrepancy.

MhueD’s picture

Feeds 7.x-2.0-alpha8 AND 7.x-2.0-beta1 -- Cron hook has completely stopped working (on our sites) and is rather easy to detect. For instance, this code:

function bupe_utility_cron() {
        run_buprenorphine_physician_feed();
}

function run_buprenorphine_physician_feed () {
        $importer_id = 'bup_info_graphic';  // Machine name of the importer
        $source1 = feeds_source($importer_id); // Load the Feeds Source object
        $source1->startImport();
        watchdog('bupe_utility', "Buprenorphine Info-graphic Feed has been initiated.", null, WATCHDOG_INFO);
}

In this case the watchdog record DOES get written, and yet the Feed itself never runs. This last part is also easy to establish, because one can put a hook like:

function bupe_utility_feeds_after_import (FeedsSource $source) {
        switch ($source->id) {
                case 'bup_info_graphic':
                        watchdog('bupe_utility', "Buprenorphine Info-graphic Feed has been completed.", null, WATCHDOG_INFO);
                        break;
        } // end switch
}

Also, if we try:

$source1->startImport() or die( "Could not start bupe import!" );

...the 'die' statement runs...so I guess in the case above the startImport is failing but control continues getting passed on to the Watchdog call anyway.

FYI: This feed only takes a few seconds to run, and it DOES run completely successfully as a standalone form from the /import UI. Perhaps even stranger...it doesn't seem to fix the Cron call if the Feed is reset to work off of a node (a common fix listed in some threads on this topic). The site itself is 7.37 Core. Also, we have tried sandboxing with Ultimate Cron with no effect. If this is a setting/db-change, I would love to have some idea of what direction to take that research, because chasing down every conceivable 'cross-talk' interaction with every other module is beyond our scope at the moment!

Below is the export code of the Feed Definition. Please note, however, that we have changed quite a few of the settings, especially those effecting timing, and the behavior stays the same. For instance, having the import run 'as often as possible' has no effect on the success of each attempt, though it does initiate the attempts as requested.

$feeds_importer = new stdClass();
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'bup_info_graphic';
$feeds_importer->config = array(
  'name' => 'Bup Info Graphic',
  'description' => 'Imports data for Buprenorphine Info Graphic',
  'fetcher' => array(
    'plugin_key' => 'FeedsHTTPFetcher',
    'config' => array(
      'auto_detect_feeds' => FALSE,
      'use_pubsubhubbub' => FALSE,
      'designated_hub' => '',
      'request_timeout' => NULL,
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsExJsonPath',
    'config' => array(
      'sources' => array(
        'certified_30_total' => array(
          'name' => 'Certified-30 Total',
          'value' => 'certified_30_total',
          'debug' => 0,
          'weight' => '1',
        ),
        'certified_100_total' => array(
          'name' => 'Certified-100 Total',
          'value' => 'certified_100_total',
          'debug' => 0,
          'weight' => '2',
        ),
        'certified_30_past_year' => array(
          'name' => 'Certified-30 Past Year',
          'value' => 'certified_30_past_year',
          'debug' => 0,
          'weight' => '3',
        ),
        'certified_100_past_year' => array(
          'name' => 'Certified-100 Past Year',
          'value' => 'certified_100_past_year',
          'debug' => 0,
          'weight' => '4',
        ),
        'certified_30_past_6_months' => array(
          'name' => 'Certified-30 Past 6 Months',
          'value' => 'certified_30_past_six_months',
          'debug' => 0,
          'weight' => '5',
        ),
        'certified_100_past_6_months' => array(
          'name' => 'Certified-100 Past 6 Months',
          'value' => 'certified_100_past_six_months',
          'debug' => 0,
          'weight' => '6',
        ),
        'certified_30_past_90_days' => array(
          'name' => 'Certified-30 Past 90 Days',
          'value' => 'certified_30_past_90',
          'debug' => 0,
          'weight' => '7',
        ),
        'certified_100_past_90_days' => array(
          'name' => 'Certified-100 Past 90 Days',
          'value' => 'certified_100_past_90',
          'debug' => 0,
          'weight' => '8',
        ),
        'certified_30_past_60_days' => array(
          'name' => 'Certified-30 Past 60 Days',
          'value' => 'certified_30_past_60',
          'debug' => 0,
          'weight' => '9',
        ),
        'certified_100_past_60_days' => array(
          'name' => 'Certified-100 Past 60 Days',
          'value' => 'certified_100_past_60',
          'debug' => 0,
          'weight' => '10',
        ),
        'certified_30_past_30_days' => array(
          'name' => 'Certified-30 Past 30 Days',
          'value' => 'certified_30_past_30',
          'debug' => 0,
          'weight' => '11',
        ),
        'certified_100_past_30_days' => array(
          'name' => 'Certified-100 Past 30 Days',
          'value' => 'certified_100_past_30',
          'debug' => 0,
          'weight' => '12',
        ),
      ),
      'context' => array(
        'value' => '$.[]',
      ),
      'display_errors' => 0,
      'source_encoding' => array(
        0 => 'auto',
      ),
      'debug_mode' => 0,
    ),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsNodeProcessor',
    'config' => array(
      'expire' => '-1',
      'author' => 0,
      'authorize' => 1,
      'mappings' => array(
        0 => array(
          'source' => 'certified_30_total',
          'target' => 'field_cert_30_total',
          'unique' => FALSE,
        ),
        1 => array(
          'source' => 'certified_100_total',
          'target' => 'field_cert_100_total',
          'unique' => FALSE,
        ),
        2 => array(
          'source' => 'certified_30_past_year',
          'target' => 'field_cert_30_past_year',
          'unique' => FALSE,
        ),
        3 => array(
          'source' => 'certified_100_past_year',
          'target' => 'field_cert_100_past_year',
          'unique' => FALSE,
        ),
        4 => array(
          'source' => 'certified_30_past_6_months',
          'target' => 'field_cert_30_past_6_months',
          'unique' => FALSE,
        ),
        5 => array(
          'source' => 'certified_100_past_6_months',
          'target' => 'field_cert_100_past_6_months',
          'unique' => FALSE,
        ),
        6 => array(
          'source' => 'certified_30_past_90_days',
          'target' => 'field_cert_30_past_90_days',
          'unique' => FALSE,
        ),
        7 => array(
          'source' => 'certified_100_past_90_days',
          'target' => 'field_cert_100_past_90_days',
          'unique' => FALSE,
        ),
        8 => array(
          'source' => 'certified_30_past_60_days',
          'target' => 'field_cert_30_past_60_days',
          'unique' => FALSE,
        ),
        9 => array(
          'source' => 'certified_100_past_60_days',
          'target' => 'field_cert_100_past_60_days',
          'unique' => FALSE,
        ),
        10 => array(
          'source' => 'certified_30_past_30_days',
          'target' => 'field_cert_30_past_30_days',
          'unique' => FALSE,
        ),
        11 => array(
          'source' => 'certified_100_past_30_days',
          'target' => 'field_cert_100_past_30_days',
          'unique' => FALSE,
        ),
      ),
      'update_existing' => '0',
      'input_format' => 'plain_text',
      'skip_hash_check' => 0,
      'bundle' => 'bup_info_graphic',
    ),
  ),
  'content_type' => '',
  'update' => 0,
  'import_period' => '-1',
  'expire_period' => 3600,
  'import_on_create' => 1,
  'process_in_background' => 0,
);

MegaChriz’s picture

MegaChriz’s picture

Priority: Normal » Major
Issue summary: View changes

I've created a summary of the issue. While I do know possible causes have been reported in this issue (and perhaps in other issues too), I've stated that the cause of the issue is still unknown. Because there are so many issues open about this problem, I have no overview of all possible causes and as such I can not judge if one of them is a real cause or a misunderstanding of how something in Feeds is supposed to work.

The first step is to consolidate the issues: gather clues from each of them and then close duplicates. Then with the clues gathered, we can look for the cause of the issue. Perhaps there is more than one cause. When the cause is known, it would be best if the bug could be replicated with an automated test. This makes it easier to find out if a proposed fix really fixes the issue or just partly.

dineshw’s picture

This should be straight but somehow it’s not working with feeds:
Can someone summarise steps for testing periodic import

Feed importer configured to import json fields works perfectly when run via import link

Same importer does not run via Cron run even after having option periodic import enabled.

MegaChriz’s picture

Steps for using periodic import:

  1. On the importer basic settings, set "Periodic import" to a value other than "Off", for example "As often as possible".
  2. Go to the import page for the feeds importer (for example: /import/my_importer).
  3. Specify a source and click the "Import" button. If your importer is attached to a content type, hit "Save". If "Import on submission" is turned off, then you also need to go the import tab (for example: node/3/import) and hit the "Import" button there.
  4. Run cron.
dineshw’s picture

FileSize
20.72 KB

Hi Chriz,
See attached screenshot. My feeds are not attached to content type and set to Periodic import of 1 hour and I have crontab configured for every 15 mins.

But if I run cron either manually or via crontab, I dont see feeds getting imported in content.

But, If I simply use below hook in cusotm module to execute feeds import it does work.

<?php
function MODULE_NAME_cron() {
  $name = 'FEED_NAME';
  $source = feeds_source($name);
  $source->import();
}
?>
pianomansam’s picture

Title: Import on cron doesn't work in Feeds 7.x-2.0-alpha8 » [META] Cron import not working on 7.x-2.0-alpha8 and later
Version: 7.x-2.0-alpha8 » 7.x-2.x-dev
Priority: Major » Critical

Updating title to better reflect this as a meta issue.

pianomansam’s picture

Issue summary: View changes

@dineshw's hook_cron() example is helpful, but it only works on feeds not attached to nodes. Here's an example of how to handle node-attached feeds:

function MODULE_NAME_cron() {
    $node_type = 'CONTENT_TYPE';
    $feed_type = 'FEED_NAME';
    $query = new EntityFieldQuery();
    $query->entityCondition('entity_type', 'node');
    $query->entityCondition('bundle', $node_type);
    $result = $query->execute();
    $feed_nids = array_keys($result['node']);
    foreach ($feed_nids as $feed_nid) {
      $source = feeds_source($feed_type, $feed_nid);
      $source->import();
    }
}

I've also updated the issue summary with these details.

daboo’s picture

I'm having this exact same problem in Drupal 8. Feeds were working as expected and then as I added content and functionality to the site it stopped working. I attributed to another module, but have no idea exactly what the root cause is at this point.
Does anyone have a manual code change as listed above that may work for Drupal 8 as a workaround?