Closed (duplicate)
Project:
Feeds
Version:
6.x-1.x-dev
Component:
Code
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
29 Jul 2010 at 01:16 UTC
Updated:
12 Sep 2010 at 03:36 UTC
Jump to comment: Most recent file
#867892: PubSubHubbub: slow down import frequency of feeds that are subscribed to hub will require adjusting the schedule of single subscriptions. This is currently not possible as feeds_scheduler does not store a schedule period per task, but rather queries the importer for the schedule period.
| Comment | File | Size | Author |
|---|---|---|---|
| #10 | 867910-10_override_schedule_period.patch | 8.56 KB | alex_b |
| #9 | 867910-9_override_schedule_period.patch | 7.88 KB | alex_b |
| #2 | 867910-2_override_schedule_period.patch | 5.32 KB | alex_b |
Comments
Comment #1
alex_b commentedAnother request that requires this functionality: #662110: Pause single feed nodes.
Comment #2
alex_b commented1. Introduces feeds_scheduler.period_override DB field for overriding the period a feed is scheduled w/. period_override = NULL -> no override. All other values correspond to an override.
2. Tested, functionality looks good.
3. All tests passing.
4. One known problem: the number of items to schedule per cron run is used twice in one cron run now.
4. *Could* be tackled by writing the update period to all jobs and storing a flag whether one is overridden or not. That would allow us to read all jobs with one query. Need to think more about that though.
Comment #3
alex_b commented#2 4.: Storing an update period with every single job has a great advantage: it would do away completely with the weird FeedsImporter::getSchedulePeriod() callbacks used by the scheduler to query the schedule periods from the importer.
HOWEVER: an upgrade path to this approach will be tricky. Here is why: we will need to know present import and expiry periods of a Feeds Importer in order to populate a newly added 'period' column. But we cannot know these periods without using Feeds' (and hence the underlying CTools') API. This will introduce all the risks of using module APIs in an update hook.
*Still thinking*
Comment #4
AntiNSA commentedsubscribe
Comment #5
andrewlevine commentedI can't help noticing that this importer/source issue keeps coming up. It is my opinion that every single setting importer setting should be able to be overridden by a source setting.
I agree that the distinction between the two (especially in the UI) is important. But if we can make the importers overridable by the source on the backend (without necessarily exposing it to the frontend) we would not need to hack around the importer/source problem for every possible override case.
Comment #6
alex_b commented#5: while I agree that there should be better 'override' support and that issue will need to be solved (most likely in the context of #651478: Mapping on import) overrides won't help here at all.
The reason is that we need these schedule periods in their own field in the table in order to do whole sale selects on them. Overrides will always live nested in a serialized array and thus will require per-source, per-importer API calls to query their value.
Comment #7
alex_b commented#3 / upgrade path could be solved with a flag thrown by the install hook (variable_set('feeds_reschedule', TRUE) that is queries on each cron run by Feeds and triggers all feed sources to be rescheduled if set. This piece of functionality could be also used for changing schedule settings on the importer edit page. Rescheduling is a simple whole sale UPDATE query that should perform fine on a page load, much more so on cron.
Comment #8
meatbag commentedsubscribe
Comment #9
alex_b commentedThis is a more comprehensive approach with an upgrade path (see hook_updateN).
I like this approach, this is close to be RTBC. Needs a quick performance test on the
(last_executed_time + period) < %dclause.
Comment #10
alex_b commentedRerolled. Not all tests passing.
Comment #11
alex_b commentedI decided that the job scheduler in Feeds needs a major cleanup. #9 and #10, #721428-12 and #13 show how quickly things get very messy and confusing with the current architecture.
See refactored job scheduler:
#908964: Break out job scheduler