Problem/Motivation
Since Feeds 8.x-3.0-alpha7 the expire feature was accidentally broken. This happened when doing some refactoring in #2811429: Switch to feed owner during manual import..
Specifically, FeedsImportHandler used to have the following code:
/**
* Finishes importing, or starts unfinished stages.
*
* @param \Drupal\feeds\FeedInterface $feed
* The feed.
*/
public function batchPostProcess(FeedInterface $feed) {
if ($feed->progressParsing() !== StateInterface::BATCH_COMPLETE) {
$this->startBatchParse($feed);
}
elseif ($feed->progressFetching() !== StateInterface::BATCH_COMPLETE) {
$this->startBatchFetch($feed);
}
elseif ($feed->progressCleaning() !== StateInterface::BATCH_COMPLETE) {
$this->startBatchClean($feed);
}
else {
$feed->finishImport();
$feed->startBatchExpire();
}
}
See one of the last line, where it says $feed->startBatchExpire();. This line got vanished. Something like it should have been added to FeedsExecutable::finish() (or FeedsBatchExecutable::finish(), since the expire feature didn't even work on cron yet).
Steps to reproduce
- Create a feed type.
- On the feed type processor settings, set a value for "Expire content items". For example, set it to "after 1 hour".
- Create a feed and import data.
- Wait until the expire time is over (for example: 1 hour).
- Import an updated feed in which some items are missing.
Expected result: the items that were imported an hour ago and that were not updated, should have been deleted.
Actual result: the expire batch does not happen.
Proposed resolution
It would be cool if the expire feature works both on cron and in the UI. But if that's a lot of work, we can for now probably also restore the functionality by triggering the expiration in FeedsBatchExecutable::finish().
We do need a functional test to ensure that the expire batch gets triggered.
Remaining tasks
- Add test coverage for the expire feature using the UI. See steps to reproduce.
- Investigate how much work it is to also support the expire feature on cron runs.
- If the expire feature is made for cron in this issue, add test coverage for this case.
- Restore the expire feature (for UI and perhaps also for cron).
User interface changes
None.
API changes
Probably none.
Data model changes
None.
Issue fork feeds-3193092
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #4
megachrizI've written a tests for expiring through the UI and cron.
I only wrote a fix for expiring through the UI so far.
Comment #6
megachrizA partly fix for the expire feature has been merged, but expiring items does not happen on cron runs yet. So leaving the issue open for that.
Comment #9
jwilson3Follow. Not only is cron affected, but also
drush feed-imis affected.Comment #12
andyf commentedRebased against
ea64d3b8118d288d77e84f03e8cb1f70f45046f0.Comment #13
megachrizThanks for working on this!
I think that the Expire feature should be able to work without requiring an import. So it should get triggered on cron runs by itself. Do you think that makes sense? Is that doable without loading all feed types on each cron run?
Comment #14
andyf commentedAt the mo I've just rebased existing work and am currently taking a look through.
I've noticed what I think is a problem with the current approach: it locks the feed in the first item and unlocks it in the final item, but as they're cron queue worker items, they might not all get processed in a single run. So I think with sufficiently large workloads this would lead to feeds being locked between cron runs.
Yup!
It's been many years since I last looked at the codebase and I'm just getting acquainted with things so bear with me. IIUC we need to load the feed to get expired IDs, so the only way I can think of to avoid loading all the feeds on every run would be to add a new base field to track the last time it had a cron clear executed on it.
So a rough outline might be
clearedand set it to 0 for existing feeds.entity_update_batch_sizefeeds that haven't been cleared recently (as indicated bycleared). For each of those feeds, if they have any expired IDs, we add the IDs to a per-feed expiration queue;Question: in an approach like this, do you think it's ok to only get a lock when grabbing the expired IDs? Ie actually deleting the items associated with those IDs will happen by processing a queue and the feed won't be locked by the queue workers?
Thanks!
Comment #15
andyf commented