Problem/Motivation
In the D7 version of Feeds there is feature to act on previously imported items that have since been removed from the source. For example, you may want to remove or unpublish previously imported nodes if they are no longer appear on the source to keep the data on your site in sync with the source.
For the old D7 issue, see #1470530: Unpublish/Delete nodes not included in feed.
Proposed resolution
Port the feature from the D7 version. When starting a import, store a list of all previously imported items by that feed. Cross off each item from the list that is seen during the import. At the end of the import unpublish or delete those that were not crossed off.
Remaining tasks
- Implement the proposed solution.
User interface changes
A new option will appear in the processor settings, with options to choose how to act on no longer existing source items.
API changes
To be determined.
Data model changes
The configuration schema of the feed type's processor settings will change.
| Comment | File | Size | Author |
|---|---|---|---|
| #28 | feeds.jpg | 38.14 KB | Dodoto |
| #24 | interdiff-2939503-23-24.txt | 5.81 KB | megachriz |
| #24 | feeds-unpublish-delete-non-existing-2939503-24.patch | 83.46 KB | megachriz |
| #23 | interdiff-2939503-20-23.txt | 1.96 KB | megachriz |
| #23 | feeds-unpublish-delete-non-existing-2939503-23.patch | 83.29 KB | megachriz |
Comments
Comment #2
megachrizI've started with writing a kernel test for this feature. During writing the tests I found out that for the directory fetcher configuration schema is missing (same goes for the upload fetcher). I also added a new
import()method to the Feed class with which you can import a feed in one go, which is in particular useful for the tests. Also noticed that$feed->startCronImport()results into a fatal error because the Feed class tries to call a method with the same name on a FeedImportHandler instance, but that class doesn't have that method. I should open separate issues for these findings, but as the end of my work day approaches, I note them here for the time being.The test that is added by this patch should fail. All other tests should hopefully pass.
Comment #3
Mile3 commentedThanks for the hard work @MegaChriz!
Comment #4
megachrizThis adds an UI for configuring the action to take when items are no longer in the feed. It involves integration with the action plugin API! This means that actions like 'Delete content', 'Unpublish selected content' and 'Make selected content sticky' become available options. The list of actions is currently limited to those that are not configurable as these are harder to integrate in Feeds because they require additional UI and additional config.
The next step is to create methods to actually apply these action plugins.
Comment #5
megachrizFor a part of the patch in #4, I had created a spin-off issue to decrease the complexity of this issue: #2941557: Add a Kernel test for DirectoryFetcher.. Most changes not directly focussing at the core of this issue where handled in that one. Only for calculating dependencies of the processor plugin an other spin-off issue may be worth it.
Since I committed #2941557: Add a Kernel test for DirectoryFetcher., the patch in #4 no longer applies, so here is a reroll that also makes use of the traits that were added in the other issue.
Comment #6
megachrizThis adds the methods
clean()andgetImportedItemIds()to the EntityProcessorBase class. Test coverage is added for these two methods in the EntityProcessorBaseTest class.The whole system of triggering the clean action is not yet implemented. This patch just adds a method to apply the clean action.
Comment #7
megachrizCoding standards + fixes in tests.
Comment #8
megachrizThis patch adds the clean stage. Implementation is a bit experimental and currently works only when doing an import directly (programmatically). It doesn't work under normal conditions yet.
Comment #9
megachrizI had accidentally turned off
UpdateNonExistentTest::testDeleteNonExistentItems().Comment #10
megachrizComment #11
megachrizComment #12
megachrizIntroducing the CleanState concept.
Comment #13
megachrizRemoving implementation from interface.
Comment #14
megachrizCoding standards and test fixes.
Comment #15
megachrizBatch support.
Comment #16
megachrizTest and code style fixes, small refactoring in tests.
Comment #17
megachrizCron support.
Comment #19
megachrizFixing FeedRefreshTest.
Comment #20
megachrizRemoved unused use statement (code style fix).
Comment #21
megachrizSo far I found one small issue: for entity types for which no non-configurable action plugins exist the setting 'update_non_existant' does not show up. This is for example the case for taxonomy. This made sense in one of the earlier patches, but at a certain point I added a built-in delete operation (which is not an action plugin) because the existing actions plugins that perform delete operations don't do the deletions themselves: instead they collect which entities should be deleted and redirect the current user to a confirmation page. That results into nothing when an action is applied during cron runs: for the cron user it doesn't make sense to redirect it to a confirmation page.
Comment #22
ericgsmith commentedCode looks good.
I have tested this via the UI using a CSV parser, and a single feed type.
When set to "Unpublish selected content" - nodes missing from the feed that were previously imported were unpublished as expected
When set to "Delete" - nodes missing from the feed that were previously imported were deleted as expected
I have not done any testing against the other available action - but it appears to be working well for the use case described in the description.
Comment #23
megachrizI have running the patch for about two weeks now on a staging website, and encountered two things:
I fixed the two issues above in the attached patch. Since I do not know the cause of the second issue, I did not write a test for it. I do know however that the error no longer got reported after making the change in
CleanState::nextEntity().Comment #24
megachrizSmall cleanups (mostly in tests).
Comment #26
megachrizGreat, all tests still pass! Committed #24.
Comment #28
Dodoto commentedHello, and thanks for the amazing module, maintainers.

I am writing here because I think this feature is not working properly.
The problem is that I have a feed that creates other feeds, which then create feeds that create nodes. I use the option to delete nodes not included in the feed. This keeps the correct number of elements up to the second level, but it turns out that some of the feeds that create the nodes are orphaned because the second level feeds are deleted without affecting the feeds that create the nodes.
Feed 1 has been changed. As a result, Feed 1.3 has been deleted, but Feeds 1.3.1, Feed 1.3.2, and Feed 1.3.3 remain in existence and are updated at the specified intervals.