#867892: PubSubHubbub: slow down import frequency of feeds that are subscribed to hub will require adjusting the schedule of single subscriptions. This is currently not possible as feeds_scheduler does not store a schedule period per task, but rather queries the importer for the schedule period.

Comments

alex_b’s picture

Another request that requires this functionality: #662110: Pause single feed nodes.

alex_b’s picture

StatusFileSize
new5.32 KB

1. Introduces feeds_scheduler.period_override DB field for overriding the period a feed is scheduled w/. period_override = NULL -> no override. All other values correspond to an override.
2. Tested, functionality looks good.
3. All tests passing.
4. One known problem: the number of items to schedule per cron run is used twice in one cron run now.

4. *Could* be tackled by writing the update period to all jobs and storing a flag whether one is overridden or not. That would allow us to read all jobs with one query. Need to think more about that though.

alex_b’s picture

Status: Active » Needs review

#2 4.: Storing an update period with every single job has a great advantage: it would do away completely with the weird FeedsImporter::getSchedulePeriod() callbacks used by the scheduler to query the schedule periods from the importer.

HOWEVER: an upgrade path to this approach will be tricky. Here is why: we will need to know present import and expiry periods of a Feeds Importer in order to populate a newly added 'period' column. But we cannot know these periods without using Feeds' (and hence the underlying CTools') API. This will introduce all the risks of using module APIs in an update hook.

*Still thinking*

AntiNSA’s picture

subscribe

andrewlevine’s picture

I can't help noticing that this importer/source issue keeps coming up. It is my opinion that every single setting importer setting should be able to be overridden by a source setting.

I agree that the distinction between the two (especially in the UI) is important. But if we can make the importers overridable by the source on the backend (without necessarily exposing it to the frontend) we would not need to hack around the importer/source problem for every possible override case.

alex_b’s picture

#5: while I agree that there should be better 'override' support and that issue will need to be solved (most likely in the context of #651478: Mapping on import) overrides won't help here at all.

The reason is that we need these schedule periods in their own field in the table in order to do whole sale selects on them. Overrides will always live nested in a serialized array and thus will require per-source, per-importer API calls to query their value.

alex_b’s picture

#3 / upgrade path could be solved with a flag thrown by the install hook (variable_set('feeds_reschedule', TRUE) that is queries on each cron run by Feeds and triggers all feed sources to be rescheduled if set. This piece of functionality could be also used for changing schedule settings on the importer edit page. Rescheduling is a simple whole sale UPDATE query that should perform fine on a page load, much more so on cron.

meatbag’s picture

subscribe

alex_b’s picture

StatusFileSize
new7.88 KB

This is a more comprehensive approach with an upgrade path (see hook_updateN).

I like this approach, this is close to be RTBC. Needs a quick performance test on the

(last_executed_time + period) < %d

clause.

alex_b’s picture

Status: Needs review » Needs work
StatusFileSize
new8.56 KB

Rerolled. Not all tests passing.

alex_b’s picture

Status: Needs work » Closed (duplicate)

I decided that the job scheduler in Feeds needs a major cleanup. #9 and #10, #721428-12 and #13 show how quickly things get very messy and confusing with the current architecture.

See refactored job scheduler:

#908964: Break out job scheduler