I have D 8.7.4, Feeds 8.x-3.x-dev (from May 17th) installed on a site.
Using together with Feeds_ex XML parser.

I am importing a feed from remote URL, which returns only about under a 1,100 records.

Import goes well, no issues.
I am having problems if I enable cleaning up the nodes that are no longer in the Feed (I select UNPUBLISH action). The feed processing is scheduled with Cron.

After feed is updated with Cron trigger, I am seeing a huge number of nodes being unpublished. It can be random number: sometimes 60, sometimes 400. The site currently has 1068 nodes, and the feed contains now only 1057. So, technically, only 11 should be unpublished.

From debugging the process, I see everything go smoothly until all of a sudden the "clean_list" array is not tracking removed items correctly.
In the end, when it gets to "cleaning" phase, it can have more items to clean up in array than what it's supposed to be.

Did anyone experience the same issue?

Comments

veronicaSeveryn created an issue. See original summary.

megachriz’s picture

megachriz
Primary language Dutch
commented

@veronicaSeveryn
I remember having seen something in the logs that could hint something in the cleaning phase is not working correctly. For example: I have seen a few times the logs saying that it had cleaned 15 items, followed by that it created 15 items on the next import.
It doesn't happen often though, so I've assumed so far that it could have something to do with that the source is not available occasionally.

But maybe there's a race condition somewhere? A feed gets locked at the start of the process, so in theory two imports for the same feed cannot run simultaneously. But I can imagine that if there are two feeds updating the same content, that the tracking of what needs to be cleaned can become a bit unpredictable. An imported item can only belong to one feed at the same time.

An other possibility is that the list of items to clean isn't always properly updated in the database during the import process - or not properly reloaded from the database. The list is saved in the "key_value" table. The states are saved with the key "feeds_feed." + Feed ID.

veronicaseveryn’s picture

veronicaseveryn commented

@MegaChriz thanks for the ideas..

1. I only have one feed updating these nodes and it's only triggered once a day, so, definitely, there can be no interference from another feed.
And since it's running once a day - we end up with almost half of the content being unpublished right there..

2. I thought about the same thing. I actually did add some logs for debugging to see how this $clean_list array is behaving after the items are removed from there (if the entity is matched in the feed). And I did see that at some point into import it starts having different count/content...
When running EntityProcessorBase->process() where we check if entity exists 

// If the entity is an existing entity it must be removed from the clean  list.
    if ($existing_entity_id) {
      $clean_state->removeItem($existing_entity_id);
    }

I am logging the result of the array after removeItem() and can see that it starts having incorrect array length and items that have been unset previously re-appear..

I just have no idea why this happens..

megachriz’s picture

megachriz
Primary language Dutch
commented

@veronicaSeveryn
On #3069752: Entities sometimes get removed/unpublished unexpectedly on cron - which probably is the exact same issue - @damondt came up with the theory that two queue tasks may be run at the same time. In theory, that would only be possible if you run cron very often - like every minute - but it's an interesting theory that two queue tasks running at the same time could cause interference. You are sure that cron is only initiated once a day from one source? Or would it perhaps be possible that two cronjobs are triggered at about the same time?

megachriz’s picture

megachriz
Primary language Dutch
commented
Status: Active » Closed (duplicate)

I committed a fix for #3069752: Entities sometimes get removed/unpublished unexpectedly on cron. Since I assumed that issue was the same, I'm closing this one as a duplicate now. Feel free to reopen if you think this is a different issue.