Problem/Motivation

I was testing the new alpha7 release of the feeds module on an existing site. I have my feed set to Unpublish any items not present during an import. I was noticing during testing that the number of results on the front-end kept fluctuating, which was due to my node items being unpublished incorrectly. I say in the title that it can potentially cause data loss, since the feeds can be configured to delete items not present.

I don't know every situation this can occur in, but I have found a reliable way to reproduce the issue on my end. I installed the Queue UI module to aid in debugging.

  1. Setup a feed that has more records then can finish in a single cron run (Mine has about 1,000).
  2. Go to you feed view and click the 'Import in background' button. Queue UI should show one record for the feed queue at this point, which is the 'Fetch' stage.
  3. Manually start the cron run and wait until Queue UI shows multiple entries, so that we know it's at the processing stage.
  4. Go back to your feed view and click the Unlock button
  5. If you look at Queue UI again, you should see that some records are still stuck in the queue.
  6. Click the 'Import in background' button again on your feed view and manually kick off the cron again.

After the last step, if you view your queue job you'll see that a huge number (depending on your total import size) of records queued up. Most of those are in the 'Cleaning' state. What seems to be happening is that when the second job starts running it resets the "feeds_clean_list" table to contain all the records. Meanwhile the orphaned queue items from the first job finish processing in the queue and force the feed into the cleaning state. Since the 'feeds_clean_list' table has been reset, it queues up the majority of your imported items for cleaning since it thinks they were missing from the feed now. If you monitor the 'feeds_clean_list' table in SQL you can see that the reset is occurring about when the second feed job kicks off. Using a database table for the cleaning appears new to alpha7, which is probably why it never cropped up in previous versions.

Proposed resolution

I'm not super familiar with the Feeds architecture, but maybe the solution is to just clean the queue of jobs when a new Feed import process runs?

Issue fork feeds-3132198

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

jacobbell84 created an issue. See original summary.

megachriz’s picture

megachriz
Primary language Dutch
commented

Cleaning the jobs when unlocking a feed is perhaps a good idea. Unlocking should only be done when an import gets stuck, but weird things can happen if you unlock a feed when there are still tasks from the previous import on the table.

So the task of this issue is: find a way to automatically clean up queue tasks when unlocking a feed.

Resetting the clean list

What seems to be happening is that when the second job starts running it resets the "feeds_clean_list" table to contain all the records.

Yes, on each new import, the "feeds_clean_list" table is reset. This happens in EntityProcessorBase::initCleanList(), which get called only when the first item during the processing stage is processed.

feeds_clean_list

Using a database table for the cleaning appears new to alpha7

The 'feeds_clean_list' table is new in alpha7 indeed and was introduced to fix a similar issue like this: in alpha6 and earlier, the clean list was saved on a serialized variable in the key_value table. It turned out that when running queue tasks in multiple threads, the clean list could get corrupted as each queue task running in the processing stage would start with the same list into memory. Modifications made by queue task 1 to the list would get lost when queue task 2 wrote their modifications back. With the switch to a database table this no longer happens as the complete list is no longer loaded into memory each time.
See #3069752: Entities sometimes get removed/unpublished unexpectedly on cron for details.

megachriz’s picture

megachriz
Primary language Dutch
commented
Title: Cron run can potentially cause data loss. » Clean queue tasks when unlocking a feed to prevent potential data loss

MegaChriz opened merge request !87

MegaChriz opened merge request !88

megachriz’s picture

megachriz
Primary language Dutch
commented
Status: Active » Needs review

This is now ready for testing and review. It is useful to test in an environment with many imports going on. Monitor on your site if imports that run on cron keep successfully finishing. The code cleans up queue tasks whenever a feed gets unlocked or finishes an import.

  • MegaChriz committed 3e27fd4 on 8.x-3.x 
    Issue #3132198 by MegaChriz: Clean queue tasks when unlocking a feed to...
megachriz’s picture

megachriz
Primary language Dutch
commented
Status: Needs review » Fixed

I did some A/B testing for some time. At one point I noticed a small difference between two sites, namely a difference in the number of items imported. But now a few days later the amounts have become equal again. Do note that on both sites the import was still running. So I suspect the one site did imports slightly faster than the other one for some reason.

Anyway, I think it's time to commit this so a new release can be made soon.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.