"No available releases found" - fetching update information exceeds timeout

I'm having the exact same issue with 7.x-rc3.

And yeah, the strangest part is how it shows "No available releases found," on a completely different set of modules and themes each time. On one run, it might show that for core and, say, Mollom and a handful of other modules. Not "no updates," but no releases period. Yet when I immediately check again without making any changes, it shows core as being up to date, but with a completely different, and seemingly random, set of modules and themes saying "No available releases found."

Tried to clear all caches?
Never seen this before...

Strange, I know. Unlike the original poster, though, I didn't even have caching turned on, so unless Drupal isn't obeying my cache preferences, it doesn't seem cache-related. Sorry I didn't mention that in my first reply. I didn't have that issue with the earlier iterations of 7.x, and will see if I get the same result with a fresh install of RC 4 to see if one of the other under-the-hood fixes to upgrade path made it all better and tucked it in for good.

Considering the post announcing RC 4 mentioned numerous fixes to the upgrade path, that seems likely. I also should have mentioned that it was only after upgrading from RC 2 to RC 3 that I noticed this. I blame the pain medication for my omission of these salient points. That might not have been the reason I left that out, but it's a handy excuse, so I'm sticking to it.

I get No available releases found on many, but not all of the contrib modules. If I do a refresh it gets more or all of them. So something is not waiting long enough or timing out on responses.

Here's what I got today:

1. First try: See attached screenshot no. 1. Many modules are marked as they're missing any release.
2. After maybe 10 minutes: Screenshot 2. Look's better now. There's an update for the Date-module.
3. Immediately after that I asked drush to update my site. This is what I got:
   

   Update information last refreshed: Di, 01/25/2011 - 18:46
   
   Update status information on all installed and enabled Drupal projects:
    Name                  Installierte Version  Proposed version  Status                                   
    Administration menu   7.x-3.0-rc1           7.x-3.0-rc1       Aktuell                                  
    Automatic Nodetitles  7.x-1.0-alpha1        7.x-1.0-alpha1    Aktuell                                  
    Drupal core           7.0                   7.0               Aktuell                                  
    Chaos tool suite      7.x-1.0-alpha2        7.x-1.0-alpha2    Aktuell                                  
    Field group           7.x-1.0-rc1           7.x-1.0-rc1       Aktuell                                  
    Global Redirect       7.x-1.3               7.x-1.3           Aktuell                                  
    Localization client   7.x-1.0-beta3         7.x-1.0-beta3     Aktuell                                  
    Localization update   7.x-1.0-alpha2        7.x-1.0-alpha2    Aktuell                                  
    Taxonomy Manager      7.x-1.0-alpha4        7.x-1.0-beta1     Aktualisierung verfügbar                 
    Token                 7.x-1.0-beta1         7.x-1.0-beta1     Aktuell                                  
    Views                 7.x-3.0-alpha1        7.x-3.0-alpha1    Aktuell                                  
    Wysiwyg               7.x-2.0               7.x-2.0           Aktuell                                  
   

   The date-module is not even listed, but now there is an update for Taxonomy Manager!

4. I reloaded the update status-page. The Taxonomy Manager-module is still marked as not having any release (and others changed back to that status as well).

I get this problem as well, exactly the same thing.

dww: This is reproducible on a site of mine as well. What I think is going on is we fetch release data for a fixed amount of time (5 seconds?) and then give up and display whatever we have. In this case, however, we aren't able to empty the fetch queue in time so a lot of gray boxes are displayed.

Have an identical issue with Drupal 6.20 where modules would randomly report "No available releases found"! See attached images 3 and 4. I must state that I am running on a VMWare virtual machine with PHP 5.2.11 with output_buffering=on in php.ini. I traced the issue down to includes/common.inc lines 551 which is:

$response = '';
while (!feof($fp) && $chunk = fread($fp, 1024)) {
$response .= $chunk;
}
fclose($fp);

In my particular instance, the XML file being built up by $response .= $chunk was getting truncated. Testing showed that if the last fread() returned less than 1024 bytes, the status of the module showed correctly. If the last fread() was more than 1024 bytes, $response gets truncated and is unable to be parsed correctly by _update_refresh() in /modules/update/update.fetch.inc as it is not well formed. Also it would appear from the code that if the stream/socket $fp is closed feof(), $response never gets the last $chunk appended.

After referring to the PHP documentation for fread() where for PHP 5 and up it suggests using stream_get_contents(), I replaced the while loop with:

$response = stream_get_contents($fp);

Now my Drupal 6.20 update page reports all module statuses correctly! See attached image 5.

Out of curiosity, I'm looking at the Drupal 7 common.inc file and it appears to still be using fread().

$chunk = fread($fp, 1024);
$response .= $chunk;

The good news is that $response always gets $chunk appended now so I can't help but wonder if it must be a bug related to fread(). If someone running Drupal 7 experiencing the issue is brave enough to test by replacing the while ($alive) loop with $response = stream_get_contents($fp), I'd be interested in seeing your results.

I'm having this problem with 7.0, as well.

Also having this problem with 7.0, it definitely has something to do with timeouts. Refreshing admin/reports/update several times seems to work, but this is painfull. Also, admin/reports/updates/check seems to work out fine, probably since it uses the batch api to check updates one by one.

This seems to be related to http://drupal.org/node/1002658 and the introduction of the 'update_max_fetch_time' variable, which is by default 5. Projects status checks are queued and this queue is only processed for 5 seconds long, so on slower connections or an unresponsive drupal.org, the queue will not be completely processed in one run.

A workaround for this problem seems to be setting the update_max_fetch_time high enough. How to set it to 30 seconds with Drush:

drush vset update_max_fetch_time 30

An additional warning when you are using the admin_menu module: the release info of projects of which the status was checked gets compared with the local project info in update_calculate_project_data() and this comparison info gets cached for at least 1 hour. On subsequent requests to admin/reports/updates/list the queue of project status checks is processed further, but as long as the cached comparison result did not expire will be served. There seems to be an exception for some paths in update_project_cache(), admin/reports/updates is one of them. But admin/reports/updates/list is not one of them. This one is listed in the hierarchical menu structure of admin_menu though. So if you access admin/reports/updates/list through the admin menu you will get the cached comparison result for at least an hour, even if the status check queue wasn't completely processed.

I just went to my database to change the value, and can't. All variables are now BLOBs, so I can't edit it. That change is a royal pain.

I'd sure like to know how to do this via DB, too.
I've only had this on one site; after switching servers.

Ideas?

Have this same issue on all 6 instances of 7.0 that I have.

Same issue on two fresh 7.2 installs.

A little more. After I do a "check manually" I get the proper versions. Upon page reload though, I start to loose some. Doesn't seem to have any rhyme or reason to it....but the big problem is, that it messes up `drush pm-update`

I'm surprised more people are not reporting this bug. Must be something obscure.

I've always had this. But thought it was because I was using xampp locally. On windows 7.

Same issue here, confirm that check manually gets the proper versions

This may very well be a bug on d.o instead. Anyway, 1) not major, and 2) needs to be fixed in D8 first, if there's anything to fix.

FWIW I think this issue is pretty serious: in addition to not displaying the correct module status, Drush seems to fall over when trying to run a module/core update.

Manually checking for updates via the UI does work for me (ie, all green), but it still seems to impact upon Drush to run site updates. Using the D7 UI to download updates partially works, but it doesn't download all contrib modules.

So I seem to have slipped back a couple of years in productivity as I've had to manually download the code from d.o then copy to my dev site... just like in those bad old [pre-Drush/pre-D7] days.

Increasing the timeout to 20 seconds, as per #13 works here without any adverse effects. Both for drush and the admin page.

Subscribing.

Subscribing

subscribing. Experiencing the same problem with Drupal 6 using drush 4.4 but seeing the list of modules with drush up is consistently missing some modules, namely, date, views, gmap

I don't want to be hijacking this issue and I may have to look elsewhere for the problem I described in #27.

The problem I am experiencing is on a particular server with the same sites installed. I am now wondering if it is a question of available resources on that setup.

I can replicate this issue as well, in Drupal 7.2 and 7.4. However, it seems to only happen to me when I update via Drush. If I update via the "update now" link in core, it takes forever but it does find everything. Unfortunately Drush tries to refetch data every time it sneezes so it renders Drush updates essentially useless.

Has anyone else had this issue only with Drush, or does it happen to you through the UI as well?

Update: It looks like the change in #13 is working, too. So... is d.o throttling people? Or do we need to increase the default timeout time in update module?

I don't use Drush and I have seen this in 7.0, 7.2, and 7.4. I have not seen it in 6.22.

[EDIT]
I stand corrected, I just saw this in 6.22 on a localhost implementation. #13 had no effect.

[EDIT]
On 7.4, putting #13 in $conf in settings.php fixed it.

Really surprised that there aren't more people reporting the same thing here. I have experienced this problem on every single D7 installation I've ever come across, no matter the environment. No such problem on any D6 sites.

Subscribing. Same here on D7.

Subscribing. Exact same issue. Random grey out. It was in my 7.0 and my 7.4. Vanilla installs.

Subscribing. Also started having this problem on some 6.21 and 6.22 sites, only on my local WAMP 2.1 server.

If it's starting to show up on D6 sites as well, then it feels like a timeout or flood control issue on d.o. Extending the timeout length on the sites themselves is just a workaround. :-(

Subscribing. Same Issue on 7.7

After some refreshing of the page and clearing caches some of them turned to green or yellow.

#13 resolved the issue for me. I had to set it to 45 seconds though.

I still think this is important to resolve though as it messed up `drush pm-update`

PS. Drupal7.7

Same issue.

Is there a way to set the update_max_fetch_time to 30 without using Drush ?

Thanks

@gagoo you can do with with a UPDATE query on the variable table WHERE name = 'update_max_fetch_time'.

If it's not in there though, you're gonna need to insert it properly with serialize() perhaps.

See #30

Thanks for your answers.

I added the variable "update_max_fetch_time" with the variable_set('update_max_fetch_time',30) funciont in a custom module and it seems to have solved the issue, all my modules update status are up to date.
However, if I click on the "check manually" link in admin/reports/updates I have this error message : Failed to get available update data for 65 projects

After that all my modules status in the update status list are back in grey, even if I refresh the page.

Thank you

Been seeing this problem on all my D7 sites as well; haven't noticed it in D6 yet. #13 seems to be helping so far.

I've changed the define('UPDATE_MAX_FETCH_TIME', 5) in update.module file to 60 seconds and problem continues. I've clicked the "check manually" option and still nothing -- the "Last checked" lists "0 sec ago". Can anyone help me figure out why my site is not able to get the available updates needed? Please....

When visiting the modules page, a warning is display:
There was a problem checking available updates for Drupal. See the available updates page for more information and to install your missing updates.
There was a problem checking available updates for your modules or themes. See the available updates page for more information and to install your missing updates

and running "Available Updates" takes me to a report where listed beside core and each module
Failed to get available update data

Clicking "check manually" returns the same report with a warning box at the top stating:
Failed to get available update data for 10 projects.

however:
On the Update tab it says all my projects are up to date until I click "check manually", then I get
the warning:
Failed to get available update data for 10 projects. and
All of your projects are up to date.

This has been normal operations on D7.7 running on a fully up-to-date lamp server. I have performed the drush "fix" up to 45 seconds with no change.

In response to hvang on post #10 and except for desperation, I messed with core code in common.inc:
changing
$response .= $chunk;
to
$response = stream_get_contents($fp);

with no effect. Right now the only way to update d7.7 is by manual comparison. I'm willing to test if someone has any ideas.

In settings.php try

ini_set('update_max_fetch_time', 60);

In php.ini try

update_max_fetch_time = 60

I have made the settings.php in D7, looks that this works.
I am not sure, please test it and give feedback.

Note sept-19: after testing: above settings don't solve this issue!!!!

Same / duplicate issue "Module updates of new modules in core are not executed" on http://drupal.org/node/1239370 ?

This is now approaching critical as I cannot get around it in 6.22 at all.

Subscribed. I'm having the same issue Drupal 7.7 on RedHat 5.
#46 did NOT work.

I found this link:
http://programmingbulls.com/d7-unable-get-update-info-after-uploading-we...
Would love to see some ideas on what to check to verify the network, system and applications can access what needs to be accessed, as well as a suggestion of what log files to check, etc.

Subscribing.

Had the issue described above for 8 to 10 modules in 6.22. I picked one (webform), disabled and re-enabled it (and its child, ND Webform), and all the modules' statuses showed properly after that.

I just did the same. As I expected, you just got lucky. Sometimes I can just refresh the page.

When this going to be fixed in D7?
Why not put update_max_fetch_time = 60 at default settings.php?
Why let some people have this problem?
Do other people need update_max_fetch_time = 5? Does it create problem to some if it's 60?

I was able to fix some other problems with the update module by adjusting my .htaccess file to point to either the www or non-www version of my url. I think it might have fixed the module checking issue as well, though I'm still not 100% sure.

You do that by uncommenting lines in .htaccess. Read it - it has a ton of instructions.

I could be wrong though.

htaccess already points to one of them.

Tried all the tweaks to .htaccess, php.ini, settings.php and the variables table as mentioned in this thread. Have also played around with PHP versions too and tried a couple of different D7 sites. Still a sea of grey, and Drush is neither able to update core nor contrib.

Only thing that I'm uneasy about is the effect of MAMP - as I'm only seeing this problem when running locally on my OSX 10.7.2 running MAMP 1.9.4. All (so far) is fine on my externally hosted solutions, ie, a site where updates fail on my local [MAMP] system has an identical mirror on a hosted system that works fine. Is MAMP (or WAMP/XAMPP) common to others reporting this problem? Or is it that my local internet connection is the problem (which is only about 3Mb/s), compared to the blazing fast connection my external hosting providers use?

Something that seemed to have a positive effect (not a cure alas), was to delete the entire contents of the cache_update table. Rather than a sea of grey, the first 15 (or so) modules have a green or pink background (ie, the status has been compared correctly) post-deletion of the cache_update table. Looking through the MySQL table I'm still seeing mostly 'fetch_task' as the cid (which seems to correspond to 'No available releases..."), rather than the desired 'available_releases.'. (NB: If you delete all rows in cache_update, this table is repopulated when you run the updates check.) Caching has been mentioned earlier in this thread, so this might be worth a bit more follow up if you know more about caching than me.

Just a thought though, should the update history site: http://updates.drupal.org/release-history be a complete list of all updates [for all contrib modules], regardless of whether the project has defined its own? I tried http://updates.drupal.org/release-history/views/7 and that came up with 'No release history available' error. This lookup is supposed to be fallback behaviour for the Update module to check the status of each module, but I'm not sure where else the Update module looks for contrib release history or what logic it uses to decide how to find out a project's current status. Maybe this is connected?

#57 (arrrgh): You need to put the URL as http://updates.drupal.org/release-history/views/7.x

Thanks @matthewv789

I'm relatively new in drupal and I have been noticing this issue in drupal 7.x. Now I would like to help in solving bugs in drupal. So I have decided to start from here. I have been looking the code and I think that more or less I've understood all, but I would like if someone can give me a summary how the update module works in case I've missed something.

I implemented the change in #10 and am having no other issues.

I'm not sure what else uses that routine (it is in "common.inc" so I'm guessing it is used quite often?)
At this point, my drupal_http_request() looks like this:

/* ADFMOD
   * Remove the while() loop per: http://drupal.org/node/952394#comment-form #10
  while ($alive) {
    // Calculate how much time is left of the original timeout value.
    $timeout = $options['timeout'] - timer_read(__FUNCTION__) / 1000;
    if ($timeout <= 0) {
      $info['timed_out'] = TRUE;
      break;
    }
    stream_set_timeout($fp, floor($timeout), floor(1000000 * fmod($timeout, 1)));
    $chunk = fread($fp, 1024);
    $response .= $chunk;
    $info = stream_get_meta_data($fp);
    $alive = !$info['eof'] && !$info['timed_out'] && $chunk;
  }*/
  $response = stream_get_contents($fp);  

Here is the fread() information from the PHP manual:
http://us.php.net/manual/en/function.fread.php
and the related text:

Example #3 Remote fread() examples
Warning

When reading from anything that is not a regular local file, such as streams returned when reading remote files or from popen() and fsockopen(), reading will stop after a packet is available. This means that you should collect the data together in chunks as shown in the examples below.

// For PHP 5 and up
$handle = fopen("http://www.example.com/", "rb");
$contents = stream_get_contents($handle);
fclose($handle);

$handle = fopen("http://www.example.com/", "rb");
$contents = '';
while (!feof($handle)) {
  $contents .= fread($handle, 8192);
}
fclose($handle);

From my testing, I really think the issue is related to timeouts on the fread().

If someone else has any other thoughts, I"m all ears, but for now, I"m using the above mod until something else works.

Andrew.

i added this to a module's .install (D7)

function mymodule_enable() {
  // fix core bug that causes update to not find modules: http://drupal.org/node/952394
  variable_set('update_max_fetch_time', 60);
}

and i have not seen grey boxes anymore. if i ever see one again i'll try out the suggestion in #10 (although that's a kitten killer and this solution isn't.

also, obviously, setting a $conf['update_max_fetch_time'] = 60; in settings.php will do the same thing (likely what the guy in #46 was attempting to do).

When I was taking a look on this a while ago, I discovered that update_max_fetch_time is not used for each single request, but for all the modules that get checked in total. This is why 5 seconds is too short most of the time and 60 seconds will work better.

(If I remember correctly.)

I think we can reasonably set that to higher than 5, here's a patch setting it to 30.

#63 Oh, wow.

Why don't we just set the default fetch time higher then?

...Doh. This is what I get for opening all my updated issues at once in a zillion tabs.

Alright, anyway. Can we have some folks who are encountering this issue test #64 (with the more conservative 30s) and see if it resolves the issue for them?

It's difficult to determine the right value because it always depends on how many modules are being checked in one call. If one has 100 modules installed then even 60 seconds can end up too short. So the whole concept has a flaw.

Well, I think setting some maximum time for the whole process is still sensible, and the fix above should hopefully resolve the issue in the vast majority of cases. 30 or 60 seconds is reasonable; 5 is not. Since this is a major bug in an active version of Drupal, it would be nice to get that in.

After that, if we wanted to make it "smarter," I see two possibilities:

1. Add a UI at admin/config to allow the user to set the config variable, and add a message directing the user to this form if/when the max time is exceeded.
2. Add a per-project max time and count the number of projects. That seems sort of hackish to me. (A similar solution for a related problem was discussed in: #1125662-21: UpdateTestContribCase fails on sites with contrib modules/themes in the filesystem).

I've been running all my sites with 30 seconds for several months now without issue.

@Crell: what is the maximum number of modules installed on your sites?

How about an equivalent patch for 6.x?

I tried

/**
 * Maximum number of attempts to fetch available update data from a given host.
 */
define('UPDATE_MAX_FETCH_ATTEMPTS', 4);

and it got me through.

#71: No idea off hand. Probably in the 60 range is my wild stab in the dark.

Patch in #65 (60 seconds) was not enough for me. Going to 120, and using the change in #72 still didn't get it. 900 wasn't even enough.

#74 Are you sure you don't have the configuration variable set? If you do, it will override the default provided by the constant.

@NancyDru: Seems like xjm is right. This sounds like you have the variable ('update_max_fetch_time') set. The changes you did (#65, #72) only affect the default values.

Maybe try

> drush vget update_max_fetch_time

to verify.

And then

> drush vset update_max_fetch_time 60

to set it.

Or

> drush vdel update_max_fetch_time

to remove it, to use the the default value.

Absolutely sure - and just double checked. BTW, Drush commands don't help me a bit.

[EDIT] So I did put in 60 in the settings.php file and it works.

Ok, sorry. But that's strange then. Anyway, good if it works now.

Attached exposes the fetch time and fetch attempt settings in the UI at admin/reports/updates/settings. Maybe this will make it easier for users to resolve these issues.

Okay, this time without inventing creative new names for FAPI properties.

From the screen shot: The max fetch time doesn't say what the unit of measurement is. Time in milliseconds? Seconds? Days?

Is there really a need to add a UI for configuring this? I'm fine with choosing a more reasonable default .... but isn't this adding unnecessary complexity?

From the screen shot: The max fetch time doesn't say what the unit of measurement is. Time in milliseconds? Seconds? Days?

Yeah, this is fixed in #80. In #79 I decided for some reason that #field_suffix was called #after.

I'm not actually particularly keen on setting the values in the UI, but I'd like to know if it resolves #74/#77. Just a proof of concept, I guess. If anything, I think we'd want to tuck these in a collapsed fieldset at the bottom of the form, and (for it to be useful) add feedback pointing people at those settings when an update attempt fails or times out.

In that collapsed fieldset. UI text could probably use tweaking. "Update fetch settings" is odd, I guess, but I couldn't come up with something better off the cuff. ;)

Collapsed (default):

Expanded:

Way too techy. I'd have absolutely no idea what to configure for that myself.

Edit: Never mind. Answer to my question is "No, read the patch." Edit 2: So, we are budgeting one second per attempt per module, correct?

When testing #85, be sure to remove your hardcoded values or saved 'update_max_fetch_time' first. To get rid of the saved variable, you can use the drush commands above, or devel, or just delete the row from the variable table and clear the cache.

#85

1. budgets 1 second per enabled module * fetch attempts. The keyword "seconds" is indeed missing in the comment.
2. doesn't have any effect when there is a configured or hard-coded value for the variable. (minority?)
3. doesn't account for max_execution_time at all
4. doesn't account for a value of 0 fetch attempts (but that's a hard-coded constant in the first place)
5. ultimately is probably not the proper fix, since this entire issue basically just says: "the built-in partial fetching of available releases is nice, but totally doesn't work in the UI" -- anyway, might be a step forward.

The problem with this is that 200 * 2 would mean potentially 400 seconds spent processing a single queue item. If the queue is processed in an http request (which it definitely will be with stock core), then that's going to time out in PHP on most installs - since max_execution_time defaults to 30, let alone what else might run during that request.

At this point it looks like the only robust solution will be to set up a batch job for the manual update fetch operation.

Summary action would be good here.

@bfroehle #90: There is a batch job (with progress bar). If you're on admin/reports/updates there is a link called 'Check manually'. That does the batch job.

I just tried the following:

1. Set the variable 'update_max_fetch_time' to the stock 5 seconds.
2. clear the 'cache_update' table in the database

TRUNCATE TABLE  `cache_update`

3. go to admin/reports/updates. After 5 seconds I see a couple of modules with update information (maybe 9). The rest is gray.
4. Click on 'Update manually'. Now I see the progress bar finishing (taking about 37 seconds). After that, I get the message "Checked available update data for 49 projects." and all of my modules have update information. Not a single one is gray.

In response to #89, #90 and #92, I can imagine raising the limit can indeed be a problem when using D7's default Drupal cron configuration ("Poor man's cron") which runs the Drupal cron at regular intervals during a user-initiated HTTP request. A really bad idea IMHO. When running Drupal cron only from unix cron (IMHO like it should be for a decently configured Drupal website on a decent hosting infrastructure) AND through the drush cron command which doesn't use a HTTP request, this won't be a problem.

Would it be possible to only process the update check queue when a real cron (not the "Poor man's" one) is initiated?

@Cyberwolf: At first I was thinking, who was talking about cron. But of course, if automatic update notification is turned on with "poor man's cron", then that's indeed an important point.

I just verified it, and the 5 second check for update information (or whatever is set in 'update_max_fetch_time') is in fact called by cron. It is not garanteed, because if all information is fresh enough it's skipped. But it will be called when neccesary.

So, then when the 'Update manually' batch job is working fine, why bother about the 5 seconds at all. It's just one click to check all modules, and you even got a progress bar giving feedback.

After all, the 5 seconds really make sense now (as default value). Nobody was considering the cron thing (and possibly not the 'Update manually').

Considering that "Poor Man's Cron" is in core, and few are willing to copy that humongous key to a "real" cron job, I suspect that better than 90% of D7 sites are using it. Therefore you are risking becoming flame-bait here.

I think that we have pretty well established that 5 seconds is inadequate for a large proportion of people, so can we get that much committed for 7.10 and 6.23? Restructuring the calls could then become another issue.

There was nothing against using "Poor mans cron". And especially because most sites probably use "Poor mans cron", raising 'update_max_fetch_time' seems not to be a good idea. That was the point.

Here is my understanding on how this whole mechanism is designed:

There are two spots where module update information is fetched:

1. A job that is regularly called in background to always complete the update information bit by bit over the time. Each call is lasting 'update_max_fetch_time' (5 per default) seconds. This job is called by cron and also one time if you open the update report page.
2. The 'Update manually' batch job, for the administrator, to check update information for all modules at once, no matter how long this will take.

Please correct me if I got this wrong. It's just my conclusion right now.

Looking at this thread, there might have been other problems, like the system forgetting about update information, or other strange things. But those are different stories to me.

Sorry, to correct myself, Cyberwolf was actually speaking against "Poor mans cron" (#93). And he is right in the fact that using a real cron is generally the better method. But of course you cannot make this a requirement.

And if you consider this "works as designed" (which I do, but like I said, correct me), then allowing update-checks only for real cron is not really necessary anyway.

Here is some more detailed information.

In Drupal 7:

1. There is the method _update_fetch_data(). Called by update_cron() and update_get_available(). Using variable 'update_max_fetch_time'. (see #96, 1.)
2. Then there is update_fetch_data_batch(). Called by update_manual_status(). Not using any timeout. (see #96, 2.)

In Drupal 6.22 it is a different situation:

1. There is only one method _update_refresh() called both by cron and by the 'Update manually' link. I couldn't find any timeout. Also 'update_max_fetch_time' can not be found anywhere in the (whole code of the) site. Only the variable 'update_max_fetch_attempts' is used.

So I consider my "works as designed" as valid for Drupal 7 (and 8). If there is problems in Drupal 6, some solution might me necessary.

Well, I have had "check manually" fail as well. Perhaps I hadn't cleared the cache in advance.

Here are some more insights:

When developing with Drupal (7) I have always wondered why sometimes a single (random) page request is taking (very) much longer than normal. Mostly when opening a page after some time has passed (maybe half an hour or longer).

I was allways thinking it might be some cleaning, caching, rebuilding or whatever Drupal has to do in background. But in fact, it was the call to _update_fetch_data() that "Poor mans cron" initiates regularly.

I put some debug message into the loop of _update_fetch_data() with dd() ('devel' module) and was watching the log file while working. Whenever a page was hanging, I could see the messages in the log showing that update information is fetched.

So for me this has two consequences:

1. Using a real cron job is a really good idea.
2. Raising 'update_max_fetch_time' can make this even worse. A site with few visitors may take like 30 seconds for the first page to appear that gets requested after there was no visitor for a while. It would practically even make sense to lower the value, especially if a real cron job can not be established.

Marked #1091992: Provide UI element to modify 'update_max_fetch_time' variable as a duplicate of this issue.

@yettyn: I find it a bit difficult follow your description. Specifically "...the value for when cron was last run still gets increased by the second since "poor man" run last. I tried to change it back toas it was before (3h), 1 Day and 1 Hour, but still Status (or the cron config page) restart the counting of last run." isn't very precise.

Are you sure the cron job is configured correctly?

I would recommend to check "Reports -> Recent log messages". There you should see a log entry for every single cron run.

Also check http://drupal.org/node/23714 ("Configuring cron jobs using the cron command") if you haven't done already.

@yettyn: I noticed that your cron line is missing one ">" right before "/dev/null". So it should be

*/5 *  * * * curl --silent --compressed http://dev.domain.com/cron.php?cron_key=vApV_snykQqwaDHZTk6htMvJ63zZOvO2nw >/dev/null 2>&1

Did you try to call the url (http://dev.domain.com/cron.php?cron_key=vApV_snykQqwaDHZTk6htMvJ63zZOvO2nw) from a browser and see if that triggers it?

Hm, would it be possible to put some temporary fix into core?
Have not been following the entire thread. At the moment does not even search for updates the maximum php script time, but about 5-6 seconds if I'm right.
Until a robust and "real" solution is in place, it might be some progress to have it a least run 30 seconds.
As not having followed the thread, this might not be possible.
But if it is...

@eigentor: D6 or D7?

uh oh I am seeing this thread is set to D8.
I mean D7, was presuming to be in the right issue...

D8 is probably the same issue as D7. You should consider using the 'Update manually' button. Read my comments #94 #96 #97 and onwards. I consider D7 (and probably D8) as "works as designed" (if it's a good design is another question). With a proper cron job it should be ok. Without it there is the 'Update manually' button which works well.

One fix that I could imagine is some better documentation/messages on the updates page that informs about the situation and the possibilities.

ah thx.
I think I already knew that, but somehow...
While the issue still being one for not-so-experienced users and thus not a small one, it is solved for me :)

Subscribing

Updated issue summary as part of drupalofficehours.org, with advice from xjm.

$ drush pm-update
Failed to get available update data for 71 projects.

Doesn't matter how many times you clear the cache....
Doesn't matter how many times you manually click "(Check manually)"....

Updates are hosed. Probably shouldn't be, but here is one fix:

Run cron, which has changed since D6, so it no longer works to just /cron.php

Head over to:
/admin/reports/status

Find the section "To run cron from outside the site, go to," and copy that big ugly mess into your crontab.

Or just click it, but you'll still get an

Access denied
You are not authorized to access this page.

message, even though it did work.

Best,
Michael

Downgrading this from major, sites can survive without update status information for a little while and setting the variable is an acceptable workaround unless you have a ridiculous number of modules installed (in which case this isn't your worst problem usually).

Just get rid of automatic checking altogether and provide a "Check for updates" link to fetch current information.

Or make automatic checking optional, for example:

function update_settings() {
...
  $form['update_check_frequency'] = array(
    '#type' => 'radios',
    '#title' => t('Check for updates'),
    '#default_value' => variable_get('update_check_frequency', 1),
    '#options' => array(
      '1' => t('Daily'),
      '7' => t('Weekly'),
      '0' => t('Never (check manually).'

$conf['update_max_fetch_time'] = 180;

worked for me, thanks!

#85: drupal8.update-fetch-releases.82.patch queued for re-testing.

Resetting the status - it doesn't seem there has been a decision to consider this issue fixed.

Questions:

• Should the behaviour be changed for D8?
• Can the current sitation for D7 be considered 'by design' as per #109?
• If this is by design, should the behaviour be documented better?

Updated issue summary.

Updating tags

So what I gather from the issue comments is if you are using your own cron job (not "poor man's cron"), the most reliable setting for update_max_fetch_time is not the default, but 30. Is that correct?

Seems this page should be updated with that information, then:
https://drupal.org/node/23714

That page is referenced by the main instructions for cron:
https://drupal.org/cron

Proper configuration for cron needs to be documented.

On our low-traffic D7 site with system cron job requesting cron.php, we would frequently get "There was an error checking for available updates". Setting update_max_fetch_time to 30 has seemingly cured the problem.

Hit this myself earlier today in D7, and had absolutely no idea wtf was going on (last night it was Diff, today it was Views, on a site with only 5 modules in total). It seems like this is biting enough people that it should be at least major, esp. since it prevents you from learning about and getting security fixes. Maybe d.o's performance has atrophied to the point where 5 seconds doesn't work anymore? At any rate, hacking it to 30 seconds ala patch #64 fixed it for me.

Later patches seemed to get a bit crazy, exposing a UI for this which doesn't really make much sense given how technical of a detail this is (a good patch for Update Status Advanced Settings module, though), and then trying to set it automagically which there were also problems with. So here's a re-roll of #64 for D8 and D7. Take it or leave it.

I'll take it.

I think increasing the timeout is good enough -- it runs in a background process to begin with.

Committed to 8.x.

We'll want to backport this to 7.x and 6.x as it may affect people's security (and the quality of the data on https://drupal.org/project/usage/drupal).

Thanks David!

Here's the 7.x patch from #122 without the -do-not-test extension.

Looks like a testbot fluke?

126: update-max-fetch-952394-122-7.x.patch queued for re-testing.

Filed #2246867: Intermittent "test did not complete due to a fatal error" failures when testing Drupal 7 patches about the plague of intermittent testbot failures.

126: update-max-fetch-952394-122-7.x.patch queued for re-testing.

I committed this to Drupal 7 for now, but the patch has problems.

After testing it on a moderately-large (but not that large) site this is what happens:

1. Very long page load times at admin/reports/updates, sometimes 30+ seconds. Usually they'll be lower the second time you visit the page after a cache clear, but if you're running lots of custom modules I would expect long page loads every time (since it will try to find them on drupal.org every single time and fail, using up as much of the 30 seconds it can to do so). At least the time spent is mostly not actual PHP execution time, so I don't think this will cause problems with reaching the PHP execution limit.
2. The above is for admins; I don't even want to think about what it does to the poor unlucky soul who triggers a cron run on a random page requests (for sites using that feature)... but that poor unlucky soul is in most cases already very unlucky.
3. Site installation will take a long time too, if using a Drupal distribution with many modules.
4. It improves the situation significantly regarding the original bug report, but doesn't actually fix it. I still saw cases where contrib modules at the bottom of my list were grayed out because it didn't have enough time to get the data about them from drupal.org.

Although all the above is bad, I think the status quo may be worse, which is why I committed it (with a big warning in CHANGELOG.txt). I can roll it back in the next couple days if need be. The situation is critical because it seems like at any random time you visit your site (or worse, get the Update Status email sent to you) you're likely to be getting incomplete information about your site's security. I see no evidence that this bug affects the data sent back to drupal.org though.

The real fix here looks like it would involve making the code in Drupal 7 and 8 smart enough to not clear the previous cached data about a module until the new data for that module is actually finished building? Then we could go back to 5 seconds just fine, and it wouldn't matter if it takes more than one cron for the new data to build.

There is nothing in the original patch that looks like it can be backported to Drupal 6, but a fix as proposed in the above paragraph might be possible to backport. It depends what's causing the equivalent issue in Drupal 6 (certainly seems like a much less severe problem there).

I think you are correct that the right fix is to not clear what you already know and let the info collect over several cron runs. Presumably this doesn't change the manual path which one always do if you need the right answer now.

On D7 sites I am getting messages that I have updates and when I look there are none. I don't know if this is related. It only seems to happen if I have just updated something.

Cron runs is #1599622: Run poormanscron via a non-blocking HTTP request.

Update module in 8.x is already using key value + temp store instead of the cache system, so once the initial list of projects is built it shouldn't disappear. However it does do a delete + write. Changing that to write through sounds reasonable.

Seems like we should change the logic to:

- fetch all the records
- compare each against the staleness threshold
- fetch and update as many stale records as possible, ideally in order of most stale first.
- any that aren't updated will be caught next run

When I try to check for available updates manually, the process takes way too long. I actually don't have the patience to wait that long so I can't even say how long. Name of modules appear multiple times on the UI so I guess system is looping...

Looks like this issue is more about the auto check that happens when one just access admin/reports/updates right?

I tried $conf['update_max_fetch_time'] = 60; it works for when I access admin/reports/updates but it does not solve the fact that checking manually never ends...

Maybe this issue is more representative of my problem https://drupal.org/node/1296860 but it was closed as a duplicate of this issue... but this issue does not seem to be focusing on manual checking of updates... so I don't know where to look...

The manual check should probably use the batch API, that could be done in a separate task to this issue.

@David could you confirm your D7 site isn't also suffering from #1296860: Update module (still) creates duplicate queue items?

Also I disagree that this is critical, it's a significant bug but it neither renders the system unusable nor affects every site using update module.

Drupal 7.x - after trying every suggestion is this thread I finally just disabled and uninstalled the Update Manager module. Then reenabled it again and the updates started working again.

@quantumized - thanks your suggestion helped me. I too read through this long thread and tried numerous suggestions with no luck. Your suggestion of uninstalling Update Manager and reinstalling solved the problem for me.

I’m seeing the same issue with 7.32 and 7.35 (just under 60 projects listed.)
Both in our profile build (our live site), and in dev copies.
The live site seemed to be fine, but it was only showing the first several items in the updates list.
(Which I realized after comparing them thoroughly, oh look a scroll bar.)

Using both ‘check manually’ from /admin/reports/updates and drush up
(drush v6.2 and v6.5)

I’ve tried:
truncate cache_update
adjusting update_max_fetch_time (tried values from 30 to 300), with drush vset and removed with vdel
confirmed I can reach/resolve updates.drupal.org
confirmed I can resolve the site locally
server reboot

Sometimes no error, claiming the process went fine, but lots of projects showing: “Failed to get available update data.”

Other times, a few will work, and the rest show in the alert at the top: “Failed to get available update data for 58 projects.”

Saw the note about cron (real vs. default), and I have apparently been doing both. The built-in was set to 3 hours, and I’ve been pinging the URI+key at 13 past the hour. I’ve resolved that, but see no affect in the updates.

I’ve also tried the most recent suggestion, uninstall and reinstall the update module.
drush dis update
drush pm-uninstall update
drush en update
(this will wipe the cache_update table, BTW.)

no change.

On one dev copy, that was 7.32, I managed to get drupal updated to 7.35, but the issue persists.

Short of hacking some ad hoc logging into the routine responsible, I’ve tried everything I can think of.

For the love of atoms, what electron isn’t spinning the correct direction!?!

This is getting a bit ridiculous. Time for the hachet; I’m off to chop at some code (some ad hoc logging.)

So far... watching the URLs hit common.inc (@793), I'm not seeing them all. Just a handful.

Once during the cycle: clear cache table, do something, click 'manually check'; I received a module error count that was about triple the actual number of 58.
Only once.

Turned 'attempts' up to 11. (drush vset update_max_fetch_attempts 11)
Now about half are loading, where 5 or less were previously.
With a clean cache_update table, using ‘check manually’ from /admin/reports/updates

Realized that 5 custom modules were erroneously reporting a project key. Corrected that.

Trying again. 10 of 53 failed.
(so slow... last time I checked we had about 25Mbps overhead on our broadband.)

No cache clear, still at 11 attempts, and now only 3 failed checks. (The list shows only one 'Failed to get available update data.' — module_filter 7.x)

Going back to no custom setting for attempts. I need to find out why only a few URLs would appear @line 793 of common.inc, prior to this.
Looking at /modules/update/update.fetch.inc next.

…logging (dd) the fetch_url_base variable in /modules/update/update.fetch.inc
What I see is the same data for all modules I'm checking.
This is used for the fail counter; therefore it is the same counter for all modules. (Is this intentional?)

What this appears to do is cause the fail count to be global for all modules at the same base URL (eg. drupal.org)

I've adjusted the assignment order and added the project name to the fail counter:
was:
149: $fetch_url_base = _update_get_fetch_url_base($project);
150: $project_name = $project['name'];

testing:
149: $project_name = $project['name'];
150: $fetch_url_base = _update_get_fetch_url_base($project).'/'.$project_name;

…results are similar to having update_max_fetch_attempts set to 11. 18 failed, 35 succeeded.

Watching my ad hoc log, and I can see that now all modules are being attempted thanks to the new suffix, and the failed items are attempted 1 more time following that.

Next, we'll set update_max_fetch_attempts to a sane number like 3, and see if that does what we think it will.

(If this works, I'll post a patch for ease.)

IT Munger, it is unclear to me reading your comments: Why are your update checks failing? Are they exceeding the 30-second timeout? You could use a packet-capture tool like Wireshark to monitor the requests and responses.

okay, 3 attempts doesn't quite do it. But drush vset update_max_fetch_attempts 5, results in only 3 module update check fails.
Not quite there, but close.

Another benefit from this patch; the module URLs are now attempted without the need to clear cache_update.

So, I'm recommending this patch.

This is our first submission; I hope folks with greater knowledge can improve and / or adjust this as necessary.

Thanks @AohRveTPV. I do have a config related timeout issue; or possibly a hardware problem... I'm still working to confirm; my config appears to be proper.
That's another conversation, but I appreciate the suggestion.

Our patch addresses the failure to continue checking for updates beyond the first failure (a unique fail counter.)
(Hoping someone can confirm this.)

I suspect that others with the same results as myself, also have a configuration or hardware issue.

Setting attempts to 3 or 5 or 7 … perhaps 'sane' wasn't the best term. how about, 'effective' for my case.

IT Munger, I will leave reviewing the function of the patch to others more familiar with the Update module code, but two notes:
1. The patch seems to be backwards. The line that is added should be the line removed, and vice versa.
2. Binary operators (e.g., '.' concatenation operator) should be surrounded by spaces per Drupal coding standards.

Of course, my first patch submission would be a failure to see that. Doh!
Learning in public isn't my favorite place, but it works.

A corrected patch file for unique fail counter per module/project name.
works for 7.32 and 7.35

NOTE: This works but interferes with the failed module count. It displays the fail count, not the module count.

I would suggest this patch only for stress testing a problem server. I was not able to overcome the issue with high retry counts, but it did make the issue very clear in a pcap.

I have confirmed that the same code set, un-patched works fine on an identical CPU. Previous to that, I eliminated the NIC and wiring as the possible culprit.

The short version:
Our local DNS is somehow causing the update batch to fail (probably during initialization, which takes much longer.)

Details:
When our local DNS is resolving, the Drupal build on that server fails. I've a duplicate server to compare against.
(stalls on initialization, about a minute, and then goes through about 5 or 6 modules before the fail counter interrupts the batch.)

When I resolve the sites, using only the hosts files; things work flawlessly.

When I moved the local DNS, the issue moved with it.
I have several DNS replicas... the 2 servers are both replicas of another system.

… still testing, bringing in additional eyes…

After much deliberation and testing. We have returned our site and config to its original state/server.
Here's why…

Everything is working fine, and we can no longer reproduce the issue.
The batch runs with no init lag, 3 seconds with an empty cache table.

We've confirmed the config is the same as during the failures.

…also

The pcaps taken (during the failures) show 2 things that shouldn't be in the conversation …that aren't present when it's working.
Retransmission packets and Reset packets. ('from' server MAC / LAN IP to router MAC / WAN IP of the updates server.)

It's not clear why this did not happen to the site copy on an identical server. The only difference: the identical server IP was not in our router's NAT table and not in our internal DNS for the site domain. That's it. Move the NAT/DNS, and the problem moved with.

Other development sub-domains not in the internal DNS, were fine the whole time; on both servers.
Why would that bit matter? Early, less thorough, tests indicated that all sub-domains on the primary sever were affected, but I couldn't reproduce that after the NAT/DNS moves/testing.

Injecting spurious packets can cause the problem we were seeing. Possibly at any point in between the servers.
…wish we'd performed a trace route during the issue. (forehead += palm)

The tr I just ran, shows all hops belong to our ISP, until the last… the IP of the updates/cdn server.

It is a known tactic of less scrupulous ISPs; to inject packets that will interfere with a conversation that they deem inappropriate.
some idle reading: https://www.eff.org/wp/detecting-packet-injection

The details of my pcap aren't identical to those examples. Some similarities. Could be something/someone else.
Perhaps servers at the cdn were targeted. Perhaps we were targeted. Perhaps gremlins?

Clearly, this all melts into speculation. We would need to pcap traffic at the cdn, while we ran the batch... compare pcap files...
But it's all for naught, without the ability to recreate the scenario.

Apologies for the length of my collected comments here. Hoping they might help.
The patch and a pcap might.
[end transmission]

PS: I've corrected the code style violation... Appreciate the comments from @AohRveTPV.
The patch is only necessary for testing; to force checks to continue when they would otherwise give up. Still not clear about why the counter is not unique in the base code, but in normal operating it doesn't seem to matter.

Hi,

Can anyone help me with what IP address(s) need to be configured in firewall rules so that we get available updates on all the modules and drupal core we have installed on the website.

Any help is appreciated.

Thanks

Closing this out as part of triage at Baltimore2017, patch has been committed to 7.x and 8.3/8.4

What about everything in #133 and subsequent comments?

Those could be moved to new issues, but someone would have to create those before closing this one.

Note that #1548286: API for handling unique queue items and #1492188: Update module creates duplicate queue items also deal with some of the issues with non-unique queue items, which was aggravating things here.

I've opened #2876583: Maintain update status cache until new information is successfully updated with a couple of notes from here and this issue, going to close this again. It's much easier to just create new issues in the first place than leaving committed ones open, which I failed to do responding to this issue at the time.

I've also opened #2876586: Update unique project tracking is too unique for IT Munger's finding, attached their patch and assigned issue credit.

I think that's everything. If there's more, let's please just open the new issues and link from this since it's very confusing at 164 comments with so many different aspects going on.

This worked for me on D7:

1) TRUNCATE TABLE cache_update

2) Add $conf['update_max_fetch_time'] = 200; on settings.php

3) Access this URL: /admin/reports/updates/check?destination=admin/reports/updates

This worked for me on D7:

1) TRUNCATE TABLE cache_update

2) Add $conf['update_max_fetch_time'] = 200; on settings.php

3) Access this URL: /admin/reports/updates/check?destination=admin/reports/updates

Thank you @Fredsvanelli! Simply doing #1 (truncating cache_update table) worked for us in D7. I'm ashamed to admit it, but our admin/reports/updates page hasn't worked properly in our production environment in over year. Apparently clearing drupal cache does not flush this tables cache, so simply truncating it got it working us.

For D8 you can do this
Use devel module. In /devel/php execute:

$query = \Drupal::database()->delete('key_value');
$query->condition('collection', 'update_fetch_task' , '=');
$query->execute();

Answer found

#169 Works for me. Thank you @kellyimagined

For D8: Just got to /admin/reports/updates or /admin/reports/updates/update and click on (Check manually)