Deleted and edited aggregator feeds are restored from queue

Can't reproduce this on my local D7 and D8 installs. Does it still happen for you? If so, maybe more specific info about php and mysql versions used might help.

Marked #1208912: Aggregator deleted feed magicaly reappears as a duplicate.

I, too, find that a feed that is deleted reappears with no title. It is more of a problem because a field as more than 255 characters so an error message is reproduced. This is in D7.8.

I was able to remove the feed by using phpMyAdmin to search all database tables for the title of the feed, then delete references in aggregator and watchdog tables.

Ok, this is really happening then :)

Yes. If it helps, deleting only the aggregator table references deleted the aggregator items from the site, but running cron brought them back again. It was only after re-deleting the aggregator table references AND deleting the watchdog table references that the problem ceased to recur.

I can confirm that I was able to fix this problem in phpMyAdmin by A) deleting the problem feed from the aggregator_feed table, B) deleting all aggregator% entries from the queue table (since I was unable to distinguish which ones pertained to the problem feed), C) running cron to verify that the problem had gone. I did not need to do anything to the watchdog table. However, I'm not sure how to get the rest of my feeds back into the queue, or whether this is necessary. Editing and saving each of the feeds did not cause them to reappear in the queue.

Same issue in D7.12.

When the feed is corrupted, this leads to an error at each cron run: "The website encountered an unexpected error. Please try again later."

If one replaces the feed address, some feed items form this new source can be fetch before the new address gets eventually replaced by the old one and the issue comes back... In this case, the "aggregator/sources/source_id" page displays a source with feed items which do not correspond.

For both reasons think this bug has at least a priority of "normal", if not "major" (especially in the reduced scope of aggregator.module).

I have test the issue and got a issue that when I delete the feed and confirmed for deletion it is reappeared without any items in particular feed name.

To solve this issue I have attach a patch file. PFA patch file and kindly review it. It might be a solution.

Hi,

I am having the same problem: some «ghost» feed that keeps «coming back» without title, blocking if I understand the cron process.

I have tried to erase the feed directly from phpmyadmin, but I still miss something: the feed keeps coming back. What are exactly the files to erase?

In the «queue» I cannot identify the feed. Is there a way to do so?

Thanks.

Hi,

I have managed to erase all the feeds blocking the Cron job. Cron jobs work well: there are no alerts in admin/reports.

But, if I run cron manually, I get a white page. I have to refresh the browser's page twice to see the cron page again with a message telling me that the cron job went well (no error messages in reports).

What could possibly cause this kind of behavior?

Thanks.

I had same problem, with the feed coming back without a title. Removing the feed from the aggregator_feed table resulted in it reappearing when cron was rerun--clearing the site cache didn't help.

Ben Stalling's approach worked for me:

I can confirm that I was able to fix this problem in phpMyAdmin by A) deleting the problem feed from the aggregator_feed table, B) deleting all aggregator% entries from the queue table (since I was unable to distinguish which ones pertained to the problem feed), C) running cron to verify that the problem had gone. I did not need to do anything to the watchdog table. However, I'm not sure how to get the rest of my feeds back into the queue, or whether this is necessary. Editing and saving each of the feeds did not cause them to reappear in the queue.

I too am having problems with a zombie feed causing errors.

I've just migrated to 7.14.

this feed worked fine in 6.24: http://feeds.feedburner.com/uua/Lxnn?format=xml

however, in 7.14 it's causing cron to fail and putting an error in the log:

PDOException: SQLSTATE[22003]: Numeric value out of range: 1264
Out of range value for column 'timestamp' at row 1: ... [:db_insert_placeholder_5] => -2209143600

I also was having problems just removing the feed via the u.i. and also directly from the aggregator_feed table
until I found this thread. and deleted everything from queue. then cron worked fine and the zombie feed did not reappear.

I then re-added the above feed, with the same PDOException error message.

so, now my 2 good feeds aren't updating (since I removed everything from the queue) -- I didn't see anything above about how to re-enable those.

and my bad feed is updating but causing errors... -- so I'll remove it again...

suggestions?
Steve

ok. I dropped feeds and items from the aggregator_feed and aggregator_item tables.
I re-created my 2 good feeds, ran update items, and they worked properly.

I tried adding the 3rd feed, this time w/o the ?format=xml at end, i.e.,
http://feeds.feedburner.com/uua/Lxnn

but still got PDOException errors. does this imply that the feed has something bad in it, even though it works fine in Drupal 6.26 ?
(or maybe Drupal 7.14 is just processing it 'properly' and getting the error?)

for now, I'll just leave that feed out.

Steve

I think it has something to do with the lenght of the url:
- http://drupal.org/node/218004
- http://drupal.org/node/1622974

Deleting the feed from the DB broke something in my D7 install: I still get a white page when I run cron manually. I have to reload the page many times to make it «work».

Hoping for a 7.15 version with a working feed module.

R.

I've got the same problem here: after deleting a feed, it reappears with empty title. This doesn't let the all the feeds updating when cron runs and makes the aggregator module quite useless.

I could reproduce the problem this morning with Drupal 7.15

I tried most of the suggestions above but missed the recommendation to delete the aggregator rows from the queue table. That fixed my problem. Cron runs fine now.

I am experiencing the same problem with Drupal 7.21. I am not database savvy, so am leery of the database deletion suggestions here. Seems this should be fixed in core so deletions through the aggregator interface actually get deleted...

I have tried the solution in #8 -- Deleting the re-appearing feeds from the aggregator feeds table and deleting all the aggregator feeds from the queue table. Already seeing some progress, as some of my feeds have begun updating upon cron run. Not sure the problems are all gone, but this seems a step in the right direction.

Still, it seems the ability to reliably delete a feed without delving into tables is so basic that this should be a priority fix for core aggregator module.

This issue is also mentioned as part of #1805282: Weird bug with the aggregation module: feed refuse to delete itself .

The above comments are correct. The feeds are reappearing due to leftover data in the queue table. This not only causes deleted feeds to reappear, but will result in edited feeds being overwritten by old settings. Here is the order of events:

1. aggregator_cron() marks a feed as needing to be updated. It saves an object representing the feed with all of the feed's properties in the queue table.
2. An administrator edits/deletes the feed.
3. When a feed is automatically updated by the queue system, aggregator_refresh() is called with the queued feed object passed as an argument.

aggregator_refresh() does not check to see if there has been any change in the status of the feed, whether it has been edited or deleted. If it fetches the feed and finds new items, then it overwrites the settings stored in the aggregator_feed table with the settings of the feed object that was stored in the queue table.

It seems likely that this issue could affect D8. I'll check it next.

Thank you, Thank you, Thank you for this! Deleted all data in aggregator table and all name = aggregator_feeds rows in queue tables and cron now drives aggregator properly again!

Happy New Year!

There are bugs in D8. As I described in #23 there are two issues occurring, though they both have the same root cause. When feeds are queued for update, the feed object is stored in the queue table.

If a feed is edited after being queued, then that feed's original settings will be restored from the queued object when the queue is run. This occurs in D7 and D8.

If a feed is deleted after being queued, then what happens differs between D7 and D8. In D7, the feed is restored from the queued object when the queue is run, although the feed title will be blank. In D8 the error shown in the attached image is produced. This happened during a cron run. Presumably the change to the Entity system prevents the feed from being restored. Instead cron runs are broken.

I wrote D8 tests for the two issue cases. The feed editing case is failing as it should be. Unfortunately, the feed deleting case is passing. I would expect the failed cron run to produce a test failure, but it isn't locally. So, I figured I should post the test for others to see.

hm, sounds nasty and probably major for D7 since it leads to data loss..
What could we do, is, instead of passing the feed to queue worker, just pass the ID?

I'm assuming something like that is how the issue will ultimately be fixed, but I haven't tried to do it yet. I've only tried to write tests thus far.

PSR-4 reroll.

After weeks of working on it, I still can't get the "queued & deleted" feed test to fail. I'm not sure where to go from here.

seems that the delete error is triggered by the edit module (judging by the screenshot)
so the test maybe is not fail, because it is not enabled in the test run?

@ParisLiakos
Thanks for the tip. Now that I see it, I should have realized it sooner.

Changes:
Added Editor module to enabled modules array.
Removed unneeded assertResponse().
Added asserts that check the number of records in queue table.
Removed unneeded drupalGet().

Forgot the status.

Alright! This is what I'm seeing during manual testing: edited feeds being restored and deleted feeds breaking cron runs in D8.

thanks a lot for the tests! :)

now lets see if this change fixes them

eh, of course, i need to adjust the test

well it needs to clear the cache

Needs a reroll. It would also be nice to upload the new test separately to illustrate the current failure.

Yeah, nearly all the code the patch was written for has changed in the last 7 months. I've looked it over and the source of the problem still exists: the entire feed object is stored in the queue. Whether or not D8 still acts on that object in the same way - restoring deleted and edited feeds based on that object - I can't say. So I've rerolled the original tests-only patch just to try and re-verify the failures. I can't test it locally. This could fail spectacularly and not in the right way.

I think I fixed the problem.

I think this one will fail correctly now. I edited the wrong lines in #42 and #44.

Now we need the patch with the actual fix from #40 and the test from #47 :)

edit: Nevermind this whole comment and any previous edits. I really wasn't thinking when I posted any of it. I'll upload the combined patch in a minute.

This is a reroll of #40.

I am still having this problem in Drupal 7.50:

1. In case you want to replicate it, the feed in question is: http://feeds2.feedburner.com/InvestingRss

2. Each time cron is run the watchdog logs errors like:
35919 15/Jul 10:20 cron error PDOException: SQLSTATE[22003]: Numeric value out of range: 7 ERROR: value "-62106566400" is out of range for type integer LINE 14: ... alt=""/>', 'http://www.investors.com/?p=249064', '-6

3. After deleting the feed, it returns after the cron with a blank title.

I wonder, has the fix to patch it been ported to Drupal 7 ?

JA

I also experience this problem in Drupal 7.50. I can verify that the method for removing the zombie feed described by BenStallings in #8 works. However, BenStallings had some reservations about this method, he wrote:

deleting all aggregator% entries from the queue table (since I was unable to distinguish which ones pertained to the problem feed) […] However, I'm not sure how to get the rest of my feeds back into the queue, or whether this is necessary.

The {queue} table is used by the BatchQueue class. From its manpage:

Stale items from failed batches are cleaned from the {queue} table on cron using the 'created' date.

So I don't think we need to worry about deleting the entries that are about aggregator entries. This is junk that cron fails to clean up because the zombie feed makes cron crash before it gets around to cleaning them up, which leads to cron trying to run them again the next time it is run. If your feed is imported without incident during a batch operation, it will not be registered in{queue}.

Out of curiosity, I examined the blob data in {queue} (it is data serialized as text, so you can inspect it in any text editor). In my case, it turned out that all the aggregator entries was about the zombie feed.

It will be nice to see the patch committed and backported to Drupal 7, but in the meantime, I think the procedure outlined in #8 is a safe way to kill a zombie.

Hello all,

I confirm the issue in Drupal 7.51. The procedure described in #8 worked for me.

Cheers

Marek

I checked the code for the aggregator module in Drupal 7, and I was unable to find a deleteItem() call. Could this be the source of the problem? I think unless there's code to delete the aggregator_feeds queue items, they'll linger around forever.

Rerolled the patch

The aggregator module has been removed from Core in 10.0.x-dev and now lives on as a contrib module. Issues in the Core queue about the aggregator module, like this one, have been moved to the contrib module queue.

Needs reroll for contrib

This is still a problem. Old settings are restored to edited feeds. Deleting a queued feed breaks queue runs, although I'm not sure it's broken in the same way. Here's a reroll.

This looks great to me, the only thing I think we're missing is a hook_post_update to truncate the existing queue items that would have the entity stored, which is no longer compatible with the queue processor.
Thanks for working on this!

@larowlan I'm sorry for all the test spam above. I'm not sure why, but it kept being weird. It queued up two identical tests at the same time for one branch. It tested the wrong branch repeatedly (and just did it again). I feel bad for taking up resources on the test server, but it wasn't something I did intentionally.

Thanks for working on this!

No problem. The other day I looked through my old unclosed tickets and saw these. It's disappointing that this had tests and a fix, but never got a proper review. I guess it just shows how little attention Aggregator got in Core.

No worries, if you're interested I'd be keen for a co-maintainer here - let me know

Here's an attempt at a post update function and a test for it. I'm positive the test isn't going to work, but I can't test it locally. I've never written a post update function before, let alone a test for one.

And yes, @larowlan, if you would like some help with the module, then I'm willing to co-maintain it. Thank you for the offer!

edit: The problem with this one is that I forgot to update the fixture file path.

The test just looks like the path to the dump file isn't right

Other than that it looks good to me

Added #3358993: Add dcam as a co-maintainer

I've added you as a maintainer - thanks!

We don't actually need a fixture anyway. It's sufficient to create a queue item and then let it be deleted by the update. The update system just has to be tricked into running the update right after installing the module in the test setup(). This one is passing for me locally.

1. +++ b/tests/src/Functional/AggregatorCronTest.php
   @@ -18,6 +19,13 @@ class AggregatorCronTest extends AggregatorTestBase {
   +  protected static $modules = array('editor');
   

   What's the significance of editor here?

2. +++ b/tests/src/Functional/AggregatorCronTest.php
   @@ -45,6 +53,32 @@ class AggregatorCronTest extends AggregatorTestBase {
   +    $this->resetAll();
   

   is \Drupal::entityTypeManager()->getStorage('aggregator_feed')->loadUnchanged($feed->id()) enough here? ::resetAll is a bit of a sledgehammer approach

3. +++ b/tests/src/Functional/AggregatorCronTest.php
   @@ -45,6 +53,32 @@ class AggregatorCronTest extends AggregatorTestBase {
   +    $this->assertEquals(0, $database->query('SELECT COUNT(*) FROM {aggregator_feed}')->fetchField(), 'There are no feeds in the database.');
   

   Can we use assertCount(0, Feed::loadMultiple()) instead of a DB query?

4. +++ b/tests/src/Functional/AggregatorCronTest.php
   @@ -45,6 +53,32 @@ class AggregatorCronTest extends AggregatorTestBase {
   +    $this->assertEquals(1, $database->query("SELECT COUNT(*) FROM {queue} WHERE name = 'aggregator_feeds'")->fetchField(), 'There is one queued feed in the database.');
   ...
   +    $this->assertEquals(0, $database->query("SELECT COUNT(*) FROM {queue} WHERE name = 'aggregator_feeds'")->fetchField(), 'There are no queued feeds in the database.');
   

   Can we use $query->numberOfItems() instead?

5. +++ b/tests/src/Functional/AggregatorCronTest.php
   @@ -45,6 +53,32 @@ class AggregatorCronTest extends AggregatorTestBase {
   +    $this->assertEquals(0, $database->query('SELECT COUNT(*) FROM {aggregator_feed}')->fetchField(), 'There are still no feeds in the database.');
   

   here too

6. +++ b/tests/src/Functional/Update/AggregatorUpdateDeleteQueueTest.php
   @@ -0,0 +1,69 @@
   +    parent::setUp();
   +
   +    // Since Aggregator was just installed Drupal thinks that it doesn't need to
   +    // run the update. Remove it from the update list so it can be run by the
   +    // test.
   +    $key_value = \Drupal::service('keyvalue');
   +    $existing_updates = $key_value->get('post_update')->get('existing_updates', []);
   +    if ($key = array_search('aggregator_post_update_delete_queue_items', $existing_updates)) {
   +      unset($existing_updates[$key]);
   +    }
   

   Neat trick!

What's the significance of editor here?

See comment #31. Years ago the test simply failed unless the Editor module was installed. We weren't sure why. But I just re-ran the test without it and it passed. So maybe it's no longer a problem outside of Core or it was a symptom of a separate problem that got fixed. I'll upload a new patch without it.

is \Drupal::entityTypeManager()->getStorage('aggregator_feed')->loadUnchanged($feed->id()) enough here?

Works for me.

Can we use assertCount(0, Feed::loadMultiple()) instead of a DB query?

That's a good idea.

Can we use $query->numberOfItems() instead?

I meant to change that, but forgot. Thanks for catching it. The only reason that I can think of that I might have avoided it years ago was because the docblock for numberOfItems() says 'This is intended to provide a "best guess" count of the number of items in the queue' and it isn't clear why. So maybe I didn't trust it.

Neat trick!

Thanks!

This is fixed for contrib Aggregator. The patch in #82 needs to be rerolled for Core.

9.5 only gets security fixes now, moving back to fixed.