update module fails due to queue.expire & key_value records

I had this exact same issue in a D8.2.6 installation. I'm also not sure when this issue started so I can't remember what could have caused it. I don't recall having issues during some updates but I'm certainly not sure so don't take that last statement as a fact.

The fix carteriii talks about did work for me too.

Same problem with 8.2.5. The fix of carteriii worked for me.

Today I had another symptom because of this problem.

Instead of a working page overview with "No available releases found" messages I got a PHP Warning:

Warning: Invalid argument supplied for foreach() in template_preprocess_update_report() (line 44 of core/modules/update/update.report.inc).

Besides the error and the available updates header no modules were shown on that page, just a little sentence: "No results". At that point I had to rely on Drush to see if there were available updates which was no good to deliver this project to the client.

It took me a while before I realized this was the same issue. The fix (deleting all the "update_fetch_tasks" rows in the key_value table) worked again and got everything back to normal.

As to why I got a Warning this time, I'm not sure but one difference was that there were 2 "update_fetch_tasks" rows for modules that were no longer installed.

Searching for a solution also brought me to this forum post: https://www.drupal.org/node/2842673. So apparently other people are experiencing this bug too.

This still exists as of 8.3.5.

The immediate problem can be fixed by running:

delete * from key_value where collection = 'update_fetch_task'

Tagging with related issue #2876583: Maintain update status cache until new information is successfully updated

I'm getting the same error as hatznie after updating core to 8.5.0

Warning: Invalid argument supplied for foreach() in template_preprocess_update_report() (line 44 of core/modules/update/update.report.inc).

I've tried following the recommendations here to wipe out the contents of key_value and not having a lot of luck. It looks like the current codebase uses \Drupal::keyValueExpirable('update_available_releases')->getAll(); to try and get the release info. I can confirm that the database has said release info, but the code never receives it, $data comes back null, and the report barfs as a result.

For starters, we should probably add some basic error handling to ensure there IS $data before trying to iterate through it (clear up the error)... then the question becomes, why are we never getting any update information back from the code that is, in fact, supposed to be getting it (especially when it does exist in the database).

Digging a bit deeper with @EclipseGc, the getAll() method in the docroot/core/lib/Drupal/Core/KeyValueStore/DatabaseStorageExpirable.php class is acting a bit funny.

When this query executes:

/**
   * {@inheritdoc}
   */
  public function getAll() {
    $values = $this->connection->query(
      'SELECT name, value FROM {' . $this->connection->escapeTable($this->table) . '} WHERE collection = :collection AND expire > :now',
      [
        ':collection' => $this->collection,
        ':now' => REQUEST_TIME
      ])->fetchAllKeyed();
    return array_map([$this->serializer, 'decode'], $values);
  }

The "expire > :now" components seems to be not working properly. I can confirm that there is data in the key_value_expire table, and that the expire column has timestamps that are "in the future."

As soon as we modify the above query to remove the "now" component, we immediately get data back and the report populates.

So, this works:

/**
   * {@inheritdoc}
   */
  public function getAll() {
    $values = $this->connection->query(
      'SELECT name, value FROM {' . $this->connection->escapeTable($this->table) . '} WHERE collection = :collection',
      [
        ':collection' => $this->collection
      ])->fetchAllKeyed();
    return array_map([$this->serializer, 'decode'], $values);
  }

However, this is a core database service that the update module is calling so modifying that query wouldn't be feasible.

Backing up from here, it seems like updating the update.module update_get_available function to call $available = \Drupal::keyValue('update_fetch_task')->getAll(); however does return data, but it shows security updates are available for all projects.

Changing the expire option to a < option also seems to work (and returns valid data). Some very strange activity here for sure...

Further digging shows that the fetchData method in the docroot/core/modules/update/src/UpdateProcessor.php class is not actually getting a value for $this->updateSettings->get('fetch.timeout'). It looks like in the database for my project, Update is missing a default update timeframe from the settings tab (daily, weekly).

I am submitting a patch that sets a default value for this as 1 day if it returns Null. This appears to immediately resolve all issues in my case.

Final notes on this:

when fetchData runs, it sets the expiration at time() plus the value of $this->updateSettings->get('fetch.timeout'). Since that was returning null, the expiration time is *always* in the past by the time that the update task is completed, and as a result the report never has anything to show.

My proposed fix provides a default of a day, so that even if the result comes back null, the report continues to function. I'm not sure how to get a site into a state where that update settings value is null, so reproducibility may be a challenge here.

I experienced both problems ("no available releases found" and "invalid argument supplied for foreach") on two drupal 8.5.3 envirnoments. For "no available releases found" I can confirm running "delete from key_value where collection = 'update_fetch_task'" fixes the problem.
For "invalid argument supplied for foreach" the problem is missing settings. Going to /admin/reports/updates/settings and saving the settings fixed the problem.

+++ b/core/modules/update/update.report.inc
@@ -40,43 +40,44 @@ function template_preprocess_update_report(&$variables) {
+  if (is_array($data)) {
+    foreach ($data as $project) {

Instead of doing this, and thus making the patch hard to read, can we do the reverse here?

if (!is_array($data) { 
  return; 
}

That way we can see what the actual changes to this file are.

updated the patch per @borisson_'s request!

This makes the patch a lot easier to grok! We need to add tests for this bug as well, so we can prove that this fixes the bug and that it does not reappear in the future.

Refactored for 8.9.10 (#20 did not apply)

Not sure related.. but seems to be.

I have a D9.0.11 site I am starting to put together. Just noticed that the Available Updates page shows "No Results". Very odd. Came across this post.

- deleting values from key_value did nothing
- numerous cr, cron, updb did nothing.
- checking manually and i get "checked update data for 78 projects" (and then 153 after deleting all the update_fetch_tasks rows) - but still no results.

- re-installed update module - no luck

then i got to the bottom of this thread and saw comments on expiry, thought to go check site time settings. Saw that nothing set in Regional Settings. Set this. Another cr.. problem fixed.

pretty surprised if this was the actual fix. Possibly some combination of the removing those rows from key_value and cr was what really fixed it.. but didn't seem to be the case.

- but do have dev/stage/prod version and db backups to test this with.

more testing and suspect my regional setting does nothing. applied the patch here and it also does nothing. Is it supposed to fix the issue or just prevent it from happening again?

the patch is just to set default update timing setting, so doesnt fix the issue; just prevents it from happening again.

i removed the patches, set the update settings, removed the key_value rows - problem fixed.

perhaps there is a db update hook required to go with this patch to delete the key_value data?

We have had this issue popping up a few times for some sites and I only now debugged it and found pretty much almost the same issue as other people have commented previously on this issue.

I have created an issue fork and an MR with the proposal I have to fix this issue, I am not 100% sure how the project gets to this state where it does not fetch project data and it does not clean the update_fetch_task KV storage either. My guess it is a combination of not having the fetch timeout configuration set up and running the updates manually or something like that, although I cannot replicate it.

The first issue is the missing fetch.timeout configuration which has a patch but I'm not really keen on the solution used on the patch from #26 and other comments because it is using 1 day which I believe is too much time, the update module comes with 30 as the default value so I guess it makes sense to use that instead of 1 day, also, the patch does not work correctly, as strtotime returns a timestamp already and doing time() + strtotime() will result in a huge timestamp from year 4000? My approach is using 30 seconds as a fallback value.

The other issue is the fact that we need to keep cleaning the update_fetch_task KV storage and I guess that comes from the fact that the site did not have a fetch.timeout, but I'm not 100% sure. My approach is to check if the queue has any items before assuming the KV storage is correct, if the queue has items on it, then it's likely the KV storage is correct, if the queue is empty, then we should not rely on the KV storage as it is probably incorrect and not mapping the queue items.

I have not written tests for this because like I was saying, I cannot 100% replicate the issue yet, I will see if I can do that at some point but nothing yet.

Also, I believe this also happens/might happen on Drupal 9? Asking that because then I believe we should target 9.3.x instead of 8.9.x

Attaching a patch just in case people want it easily here

Haven't had time to read the whole issue but this seems like it might be related or the same issue as #2920285: Update module can get 'stuck' with 'no releases available'

This is indeed the exact same issue as 2920285.

I proposed a solution (patch) to fix core there which clears away all of the expired data before running either a manual check, or the main scheduled updates check. Patched into core it works quite well to keep the NARFs at bay: #2920285-50: Update module can get 'stuck' with 'no releases available'.

Until this is all fixed, I also created a module which also clears out all of the dud state before running so I don't need another core patch: Update NARF!.

The other option is to re-write the update manager to not use multiple state data sources to manage the update process. The contents of the keyvalue:update_fetch_task is never actually referenced, so this could be used for storing useful information like when it was created so that if the queue runner item for it disappears, it can easily be re-created to complete the job. At present it's only used as a semaphore which blocks the fetch task from being re-triggered.

Updating referenced core to 11.x because this bug (same as 2920285) still exists there too.

thank you #38 works great !

A problem that appears similar the problem described throughout this thread occurred after updating a site to D10.2.4.

After the 10.2.4 update, The "available updates" report displays only about 5 out of 60 or more installed contrib modules and themes.

None of the suggestions provided in this thread have solve the problem.
-
1. When the "available updates" report broke, it continues to list a few modules, but not all.
2. Modules installed after this problem began appear correctly in the "available updates" report.
3. executing the following command via drush does not repair the problem:

php -c ./php.ini vendor/bin/drush php:eval "\Drupal::keyValue('update_fetch_task')->deleteAll();"

4. installing the https://www.drupal.org/project/update_narf module does not repair the problem.

5. the following drush command displays all the contrib modules. Obviously drush is doing something different from the "available updates" report.

php -c ./php.ini vendor/bin/drush pm-list --type=Module --no-core --format=list

5. Does anyone have any additional suggestions or insights about this problem and about how to repair it?

6. applying the patch below does not repair the problem. However, the patch does not appear to break anything.

diff --git a/core/modules/update/src/UpdateProcessor.php b/core/modules/update/src/UpdateProcessor.php
index a07a47c8f5..c4de5df0a3 100644
--- a/core/modules/update/src/UpdateProcessor.php
+++ b/core/modules/update/src/UpdateProcessor.php
@@ -121,7 +121,7 @@ public function createFetchTask($project) {
     if (empty($this->fetchTasks)) {
       $this->fetchTasks = $this->fetchTaskStore->getAll();
     }
-    if (empty($this->fetchTasks[$project['name']])) {
+    if (empty($this->fetchTasks[$project['name']]) || $this->fetchTasks[$project['name']] !== $project) {
       $this->fetchQueue->createItem($project);
       $this->fetchTaskStore->set($project['name'], $project);
       $this->fetchTasks[$project['name']] = REQUEST_TIME;
diff --git a/core/modules/update/tests/src/Functional/UpdateSemverCoreTest.php b/core/modules/update/tests/src/Functional/UpdateSemverCoreTest.php
index 4573cff6c0..87fc2f03a9 100644
--- a/core/modules/update/tests/src/Functional/UpdateSemverCoreTest.php
+++ b/core/modules/update/tests/src/Functional/UpdateSemverCoreTest.php
@@ -193,12 +193,12 @@ public function testFetchTasks() {
     $this->assertEquals(2, $queue->numberOfItems(), 'Queue contains two items');
     // Try to add a project again.
     update_create_fetch_task($project_a);
-    $this->assertEquals(2, $queue->numberOfItems(), 'Queue still contains two items');
+    $this->assertEquals(3, $queue->numberOfItems(), 'Queue still contains three items');

     // Clear storage and try again.
     update_storage_clear();
     update_create_fetch_task($project_a);
-    $this->assertEquals(2, $queue->numberOfItems(), 'Queue contains two items');
+    $this->assertEquals(4, $queue->numberOfItems(), 'Queue contains four items');
   }

   /**
diff --git a/core/modules/update/update.install b/core/modules/update/update.install
index b77654d622..b067b788dd 100644
--- a/core/modules/update/update.install
+++ b/core/modules/update/update.install
@@ -87,6 +87,8 @@ function update_uninstall() {

   $queue = \Drupal::queue('update_fetch_tasks');
   $queue->deleteQueue();
+
+  \Drupal::keyValue('update_fetch_task')->deleteAll();
 }

 /*