Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Problem/Motivation
Imports work for the first node. But any subsequent nodes don't trigger an import.
I have a content type called 'staff member' and am using feeds to import the staff members' publications from a URL.
I add a URL to the feed node and then run cron. Everything imports as expected. I add a second URL to a different feed node and run cron - the import is not triggered for this new source, even after running cron many times. The import frequency is set to 'as often as possible', so this is not the issue.
In feeds_cron(), an empty array is always returned when calling feeds_schedule, so no importers are being scheduled:
// Find importers that need to be rescheduled.
if (!$importers = feeds_reschedule()) {
return;
}
Proposed Resolution
I noticed that if I change the importer frequency, save, and run cron then the import runs as expected. Looking at the code in FeedsImporter::configFormSubmit(), it runs because the importer is rescheduled:
/**
* Reschedule if import period changes.
*/
public function configFormSubmit(&$values) {
if ($this->config['import_period'] != $values['import_period']) {
feeds_reschedule($this->id);
}
parent::configFormSubmit($values);
}
My proposal is to also reschedule the importer when a feed node is saved in FeedSource::save():
if (db_query_range("SELECT 1 FROM {feeds_source} WHERE id = :id AND feed_nid = :nid", 0, 1, array(':id' => $this->id, ':nid' => $this->feed_nid))->fetchField()) {
drupal_write_record('feeds_source', $object, array('id', 'feed_nid'));
feeds_reschedule($this->importer->id);
}
Unless someone with a more thorough understanding of the scheduling side of feeds has a better way of fixing this?
| Comment | File | Size | Author |
|---|---|---|---|
| #35 | interdiff-2450365-33-35.txt | 4.52 KB | MegaChriz |
| #35 | feeds-schedule-on-update-2450365-35.patch | 9.49 KB | MegaChriz |
| |||
| #33 | interdiff-2450365-31-33.txt | 1.67 KB | MegaChriz |
| #33 | feeds-schedule-on-update-2450365-33.patch | 6.41 KB | MegaChriz |
| #31 | interdiff-2450365-28-31.txt | 2.39 KB | MegaChriz |
Comments
Comment #1
littledynamo CreditAttribution: littledynamo commentedComment #2
littledynamo CreditAttribution: littledynamo commentedPatch to follow
Comment #3
littledynamo CreditAttribution: littledynamo commentedComment #4
littledynamo CreditAttribution: littledynamo commentedI may be off track with my proposed resolution - still not working properly. I need to do more digging.
Comment #5
littledynamo CreditAttribution: littledynamo commentedOk, so I think my original feeling was correct. When the 2nd, 3rd or any subsequent feed node is saved, the importer is not rescheduled and the import doesn't run. When I add feeds_reschedule() to the source::save() function, my importer runs:
P.s. sorry for the confusion from #4, it wasn't working because the staff member had masses of publications and the request was timing out (need to look into batching).
Comment #6
littledynamo CreditAttribution: littledynamo commentedComment #7
littledynamo CreditAttribution: littledynamo commentedComment #8
littledynamo CreditAttribution: littledynamo commentedComment #10
littledynamo CreditAttribution: littledynamo commentedComment #11
littledynamo CreditAttribution: littledynamo commentedComment #12
littledynamo CreditAttribution: littledynamo commentedComment #13
twistor CreditAttribution: twistor as a volunteer commentedI think I understand what you're trying to do here.
Allow me to write up an example.
Let's say we have a site with a bunch of page nodes and no Feeds configured.
We configure an importer, and attach it to the page.
If we go back to those existing nodes, and enter in URLs to import, they are not scheduled.
I can replicate this scenario.
Comment #14
twistor CreditAttribution: twistor as a volunteer commentedThis changes things to use JobScheduler::check() instead of re-scheduling a job on every node save.
Comment #15
twistor CreditAttribution: twistor as a volunteer commentedComment #17
twistor CreditAttribution: twistor as a volunteer commentedComment #18
twistor CreditAttribution: twistor as a volunteer commentedComment #21
twistor CreditAttribution: twistor as a volunteer commentedComment #22
twistor CreditAttribution: twistor as a volunteer commentedComment #26
MegaChriz CreditAttribution: MegaChriz as a volunteer commentedThe patch is hanging on the same error as the test posted in #2624344: Import via pushImport() keeps looping / never completes, which uncovered an endless loop. Now that I committed a fix for that issue, let's see if a retest helps...
Comment #28
MegaChriz CreditAttribution: MegaChriz as a volunteer commentedRan into this issue while working on #2868134: Show next time that the source will be imported.
Here is a reroll with tests.
Comment #31
MegaChriz CreditAttribution: MegaChriz as a volunteer commentedThe tests ran forever because of a
while()inFeedsSchedulerTestCase::testBatching(). Let's fix that. Tests will fail however.Comment #33
MegaChriz CreditAttribution: MegaChriz as a volunteer commentedAlright, figured it out.
When a feeds job gets dispatched by Job Scheduler, the job is set on the queue and then the job is 'reserved', meaning it is put on hold by setting the
$job['scheduled']value. This wil make Job Scheduler ignore the next import time until the job is done (which Feeds never officially reports).When checking for an existing job with
JobScheduler::check(), the job is only rescheduled when the period changed.So what we need to do:
If the job does exist and it is currently put on hold, reschedule it. If the job does not exist, set it.
Attached a new patch.
Comment #35
MegaChriz CreditAttribution: MegaChriz as a volunteer commentedThe test failed because a 'feeds_reschedule' should force a feed to be rescheduled, even if done so before.
As suggested in #2630694-20: Run background jobs directly in queue., this adds a method
ensureSchedule()to the FeedsSource class. This way a 'feeds_reschedule' will force the feed to be rescheduled. When saving a feed node or when hitting "Import" for the standalone form, the feed should only be scheduled if currently not scheduled.Comment #37
MegaChriz CreditAttribution: MegaChriz as a volunteer commentedGreat, tests are passing!
Committed #35.