Problem/Motivation

In the D7 version of Feeds there is feature to act on previously imported items that have since been removed from the source. For example, you may want to remove or unpublish previously imported nodes if they are no longer appear on the source to keep the data on your site in sync with the source.

For the old D7 issue, see #1470530: Unpublish/Delete nodes not included in feed.

Proposed resolution

Port the feature from the D7 version. When starting a import, store a list of all previously imported items by that feed. Cross off each item from the list that is seen during the import. At the end of the import unpublish or delete those that were not crossed off.

Remaining tasks

  • Implement the proposed solution.

User interface changes

A new option will appear in the processor settings, with options to choose how to act on no longer existing source items.

API changes

To be determined.

Data model changes

The configuration schema of the feed type's processor settings will change.

CommentFileSizeAuthor
#28 feeds.jpg38.14 KBDodoto
#24 interdiff-2939503-23-24.txt5.81 KBmegachriz
#24 feeds-unpublish-delete-non-existing-2939503-24.patch83.46 KBmegachriz
#23 interdiff-2939503-20-23.txt1.96 KBmegachriz
#23 feeds-unpublish-delete-non-existing-2939503-23.patch83.29 KBmegachriz
#20 interdiff-2939503-19-20.txt427 bytesmegachriz
#20 feeds-unpublish-delete-non-existing-2939503-20.patch83.27 KBmegachriz
#19 interdiff-2939503-17-19.txt1004 bytesmegachriz
#19 feeds-unpublish-delete-non-existing-2939503-19.patch83.31 KBmegachriz
#17 interdiff-2939503-16-17.txt12 KBmegachriz
#17 feeds-unpublish-delete-non-existing-2939503-17.patch82.33 KBmegachriz
#16 interdiff-2939503-15-16.txt6.84 KBmegachriz
#16 feeds-unpublish-delete-non-existing-2939503-16.patch75.25 KBmegachriz
#15 interdiff-2939503-14-15.txt16.23 KBmegachriz
#15 feeds-unpublish-delete-non-existing-2939503-15.patch74.58 KBmegachriz
#14 interdiff-2939503-13-14.txt2.02 KBmegachriz
#14 feeds-unpublish-delete-non-existing-2939503-14.patch61.09 KBmegachriz
#13 interdiff-2939503-12-13.txt544 bytesmegachriz
#13 feeds-unpublish-delete-non-existing-2939503-13.patch60.77 KBmegachriz
#12 interdiff-2939503-11-12.txt9.44 KBmegachriz
#12 feeds-unpublish-delete-non-existing-2939503-12.patch60.8 KBmegachriz
#11 interdiff-2939503-10-11.txt619 bytesmegachriz
#11 feeds-unpublish-delete-non-existing-2939503-11.patch54.88 KBmegachriz
#10 interdiff-2939503-09-10.txt1.37 KBmegachriz
#10 feeds-unpublish-delete-non-existing-2939503-10.patch54.88 KBmegachriz
#9 interdiff-2939503-8-9.txt709 bytesmegachriz
#9 feeds-unpublish-delete-non-existing-2939503-9.patch54.55 KBmegachriz
#8 interdiff-2939503-7-8.txt12.2 KBmegachriz
#8 feeds-unpublish-delete-non-existing-2939503-8.patch54.55 KBmegachriz
#7 interdiff-2939503-6-7.txt3.07 KBmegachriz
#7 feeds-unpublish-delete-non-existing-2939503-7.patch44.58 KBmegachriz
#6 interdiff-2939503-5-6.txt14.87 KBmegachriz
#6 feeds-unpublish-delete-non-existing-2939503-6.patch44.48 KBmegachriz
#5 feeds-unpublish-delete-non-existing-2939503-5.patch35.9 KBmegachriz
#4 interdiff-2939503-2-4.txt7.18 KBmegachriz
#4 feeds-unpublish-delete-non-existing-2939503-4.patch105.23 KBmegachriz
#2 feeds-unpublish-delete-non-existing-2939503-2.patch101.71 KBmegachriz

Comments

MegaChriz created an issue. See original summary.

megachriz’s picture

I've started with writing a kernel test for this feature. During writing the tests I found out that for the directory fetcher configuration schema is missing (same goes for the upload fetcher). I also added a new import() method to the Feed class with which you can import a feed in one go, which is in particular useful for the tests. Also noticed that $feed->startCronImport() results into a fatal error because the Feed class tries to call a method with the same name on a FeedImportHandler instance, but that class doesn't have that method. I should open separate issues for these findings, but as the end of my work day approaches, I note them here for the time being.

The test that is added by this patch should fail. All other tests should hopefully pass.

Mile3’s picture

Thanks for the hard work @MegaChriz!

megachriz’s picture

This adds an UI for configuring the action to take when items are no longer in the feed. It involves integration with the action plugin API! This means that actions like 'Delete content', 'Unpublish selected content' and 'Make selected content sticky' become available options. The list of actions is currently limited to those that are not configurable as these are harder to integrate in Feeds because they require additional UI and additional config.

The next step is to create methods to actually apply these action plugins.

megachriz’s picture

For a part of the patch in #4, I had created a spin-off issue to decrease the complexity of this issue: #2941557: Add a Kernel test for DirectoryFetcher.. Most changes not directly focussing at the core of this issue where handled in that one. Only for calculating dependencies of the processor plugin an other spin-off issue may be worth it.

Since I committed #2941557: Add a Kernel test for DirectoryFetcher., the patch in #4 no longer applies, so here is a reroll that also makes use of the traits that were added in the other issue.

megachriz’s picture

This adds the methods clean() and getImportedItemIds() to the EntityProcessorBase class. Test coverage is added for these two methods in the EntityProcessorBaseTest class.

The whole system of triggering the clean action is not yet implemented. This patch just adds a method to apply the clean action.

megachriz’s picture

Coding standards + fixes in tests.

megachriz’s picture

This patch adds the clean stage. Implementation is a bit experimental and currently works only when doing an import directly (programmatically). It doesn't work under normal conditions yet.

megachriz’s picture

I had accidentally turned off UpdateNonExistentTest::testDeleteNonExistentItems().

megachriz’s picture

megachriz’s picture

megachriz’s picture

Introducing the CleanState concept.

megachriz’s picture

Removing implementation from interface.

megachriz’s picture

Coding standards and test fixes.

megachriz’s picture

Batch support.

megachriz’s picture

Status: Active » Needs work
StatusFileSize
new75.25 KB
new6.84 KB

Test and code style fixes, small refactoring in tests.

megachriz’s picture

Status: Needs work » Needs review
StatusFileSize
new82.33 KB
new12 KB

Cron support.

Status: Needs review » Needs work

The last submitted patch, 17: feeds-unpublish-delete-non-existing-2939503-17.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

megachriz’s picture

Status: Needs work » Needs review
StatusFileSize
new83.31 KB
new1004 bytes

Fixing FeedRefreshTest.

megachriz’s picture

Removed unused use statement (code style fix).

megachriz’s picture

So far I found one small issue: for entity types for which no non-configurable action plugins exist the setting 'update_non_existant' does not show up. This is for example the case for taxonomy. This made sense in one of the earlier patches, but at a certain point I added a built-in delete operation (which is not an action plugin) because the existing actions plugins that perform delete operations don't do the deletions themselves: instead they collect which entities should be deleted and redirect the current user to a confirmation page. That results into nothing when an action is applied during cron runs: for the cron user it doesn't make sense to redirect it to a confirmation page.

ericgsmith’s picture

Code looks good.

I have tested this via the UI using a CSV parser, and a single feed type.

When set to "Unpublish selected content" - nodes missing from the feed that were previously imported were unpublished as expected
When set to "Delete" - nodes missing from the feed that were previously imported were deleted as expected

I have not done any testing against the other available action - but it appears to be working well for the use case described in the description.

megachriz’s picture

I have running the patch for about two weeks now on a staging website, and encountered two things:

  1. For entity types without non-configurable action plugins, the option didn't show up (as I already reported in #21). This is for example the case for taxonomy terms.
  2. Under unknown circumstances the clean state doesn't get an entity type assigned. I couldn't figure out yet when, only noticed a lot of error messages in the log: 'The clean state does not have an entity type assigned.'.

I fixed the two issues above in the attached patch. Since I do not know the cause of the second issue, I did not write a test for it. I do know however that the error no longer got reported after making the change in CleanState::nextEntity().

megachriz’s picture

Small cleanups (mostly in tests).

  • MegaChriz committed a1290e7 on 8.x-3.x
    Issue #2939503 by MegaChriz, ericgsmith: Added feature to unpublish/...
megachriz’s picture

Status: Needs review » Fixed

Great, all tests still pass! Committed #24.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Dodoto’s picture

StatusFileSize
new38.14 KB

Hello, and thanks for the amazing module, maintainers.
I am writing here because I think this feature is not working properly.
The problem is that I have a feed that creates other feeds, which then create feeds that create nodes. I use the option to delete nodes not included in the feed. This keeps the correct number of elements up to the second level, but it turns out that some of the feeds that create the nodes are orphaned because the second level feeds are deleted without affecting the feeds that create the nodes.
feeds
Feed 1 has been changed. As a result, Feed 1.3 has been deleted, but Feeds 1.3.1, Feed 1.3.2, and Feed 1.3.3 remain in existence and are updated at the specified intervals.