I have a Drupal Commerce site which needs to process thousands of feed items each night. Let me try to sketch the structure of the site.

The data is provided in 10 different CSV files. I have 5 different feed importers, each processing specific data.

  • 2 feed importers for importing/updating users (a separate processor for each role)
  • 2 feed importers for importing/updating terms (a separate processor for each role)
  • 1 feed importer for importing/updating Drupal Commerce products

The reason I have 2 separate importers for mainly the same goal (users or terms), is because in my hook_feeds_after_parse and my hook_feeds_presave, I modify the data to fit in my site. Hence the separate importers.

The last importer, the one importing the +9000 Drupal Commerce products, triggers a function which simply does a db_merge from the 5 other CSV-files, directly onto the database. (The data is sanitised, so no worries about that). This last importer (those 5 other files) is written out in a Drupal Batch, so I won't be hitting any timeout.

This whole process should repeat every night (to make sure the webshop contains the most recent product/client data).
The CSV files are generated by our customers server and he uploads those every night into our Drupal (via SFTP). So I have to assign a source to each feeds importer to make sure the importer doesn't use it's last imported file.

I've written a hook_cron: 

    $feeds = array(
       'feed_importer1'  => 'file1',
       'feed_importer2'  => 'file2',
       'feed_importer3'  => 'file3',
       'feed_importer4'  => 'file4',
       'feed_importer5'  => 'file5',
    );
    
    foreach ($feeds as $feed_id => $file_name) {
       $file_path    = 'public://feeds/import/' . $file_name . '.CSV';
       $destination  = 'public://feeds/processed/' . $file_name . '.CSV';
    
       $file = file_get_contents($file_path);
       $file = file_save_data($file, $destination, FILE_EXISTS_REPLACE);
    
       if ($file) {
          if (_check_for_utf_8($file->uri)) {
             $source = feeds_source($feed_id);
             $file   = file_load($file->fid);
    
             // Load the source fetcher config.
             $fetcher_config = $source->getConfigFor($source->importer->fetcher);
    
             // Add the new file object, and it's uri to the fetcher config.
             $fetcher_config['source'] = $file->uri;
             $fetcher_config['file']   = $file;
    
             $config = array(
                'process_in_background' => FALSE,
             );
    
             $source->importer->addConfig($config);
    
             // Set the source fetcher config.
             $source->setConfigFor($source->importer->fetcher, $fetcher_config);
    
             // Save the source.
             $source->save();
    
             // Execute the import.
             $source->import();
          }
       }
    }

For some reason, the code is perfectly working when I trigger this in /devel/php, but when I configure this cron job to run at 3 AM, the importer gets stuck...

I'm rethinking and redesigning this whole import process over and over again, but I cannot seem to figure this out.
One other pitfall is the "Skip Hash" option has to be enabled in order to process all items correctly. And by so, updating +9000 Drupal Commerce entities takes about 15 minutes to complete. Doing so in code doesn't create an error, triggering this in cron does.

I'm also thinking of creating separate cron jobs for each importers and run those each hour, but I don't know if that would make any difference.
I'm looking forward to any advice, suggestion or improvement to make sure all data will be imported each night at 3AM.

Comments

megachriz’s picture

megachriz
Primary language Dutch
commented
Status: Active » Fixed

Ultimate cron can be used to run a task at a specific time

I don't know if this helps, but I use the Ultimate Cron module to run tasks at a specific time, including a task to run multiple Feeds importers programmatically in a specific order. Ultimate cron keeps logs for every task, so you can quickly see which tasks failed and lookup the log for the specific task. In my setup I've also a task that sometimes can take 30 minutes to complete and most of the times that task is completed successfully (this task is not a feeds import task).

Skip hash check

I think you should look for a method which does not require the "skip hash check" option to be turned on. You save a lot of resources if only changed commerce products need to be updated.

Cron vs /devel/php

One of the differences between running a script manually via /devel/php and cron is that cron is usually ran as an anonymous user. It's therefore possible that during cron access checks are performed that did not run when executing the script manually (assuming you did that as user 1, for which most access checks are skipped).
There are more issues about importing on cron in this issue queue. Maybe some of these issues will give you further hints on how to solve your issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

dureaghin’s picture

dureaghin commented

Ultimate Cron is not picking up separate feeds. You can use this sandbox module to run multiple feeds Feeds Ultimate Cron

or 


/*
 * Feed import managed by Ultimate Cron
 */
function MODULE_NAME_cronapi($op, $job = NULL) {
  $items['fetch_feed_name_#1_cron'] = array(
    'description' => 'FETCH_DESCRIPTION',
    'rule' => '* * * * *', // Every 1 minutes
    'callback' => 'MODULE_NAME_fetch_cron',
    'arguments' => array(array('FEED_NAME_#1')),
	'weight' => -10,
  );
  $items['fetch_feed_name_#2_cron'] = array(
    'description' => 'FETCH_DESCRIPTION',
    'rule' => '* * * * *', // Every 1 minutes
    'callback' => 'MODULE_NAME_fetch_cron',
    'arguments' => array(array('FEED_NAME_#2')),
	'weight' => -9,
  );  
  return $items;
}

function MODULE_NAME_fetch_cron($feednames){
  if (function_exists('feeds_source')){
    foreach($feednames as $feedname){
      $source = feeds_source($feedname);
      $source->import();
    }
  }
}