Drush pm-update, up and updatedecode commands do not seem to be recognising all the available updates properly. It is quite random. Sometimes it pick up one available update or sometimes the other and most times not any at all even there are updates available. I did several drush rf to get drush recognise the available dev update but was not successful. I had to keep on trying at different intervals and then on one occasion it picked up coincidently when I decided to use --debug flag to try to find out the cause of error. Drush ALL Versions-HEAD also presents this issue. This was not present in All versions-3.3.
Comment | File | Size | Author |
---|---|---|---|
#73 | drush-1002658.patch | 3.88 KB | jonhattan |
#66 | drush-1002658.patch | 2.93 KB | jonhattan |
#64 | drush-1002658.patch | 2.95 KB | jonhattan |
#60 | drush-1002658.patch | 7.87 KB | jonhattan |
#52 | drush-1002658.patch | 1.58 KB | jonhattan |
Comments
Comment #1
aanjaneyam CreditAttribution: aanjaneyam commentedComment #2
greg.1.anderson CreditAttribution: greg.1.anderson commentedA lot has changed in the pm code in drush-4, but updatecode still uses Drush update status to determine whether updates are available. Not sure what to say about further debugging.
Comment #3
Metztli CreditAttribution: Metztli commentedThis is what I get when attempting update (up) using Drush 4.0 RC3 on Drupal 7.0 RC1 (I am passing the --yes option argument so as not to be prompted for input):
Refreshing update status information ...
Done.
Update information last refreshed: Mon, 12/20/2010 - 04:52
Update status information on all installed and enabled Drupal projects:
Name Installed Proposed Status
version version
Drupal 7.0-rc1 7.0-rc2 SECURITY UPDATE available
core
Danland 7.x-1.0-rc1 7.x-1.0-rc2 Update available
NOTE: A security update for the Drupal core is available.
Drupal core will be updated after all of the non-core modules are updated.
Code updates will be made to the following projects: Danland [danland-7.x-1.0-rc2]
Note: Updated projects can potentially break your site. It is NOT recommended to update production sites without prior testing.
Note: A backup of your project will be stored to backups directory if it is not managed by a supported version control system.
Note: If you have made any modifications to any file that belongs to one of these projects, you will have to migrate those modific
ations after updating.
Do you really want to continue with the update process? (y/n): y
Refreshing update status information ...
Done.
Update information last refreshed: Mon, 12/20/2010 - 04:52
Update status information on all installed and enabled Drupal projects:
Name Installed Proposed Status
version version
No code updates available. [ok]
Comment #4
greg.1.anderson CreditAttribution: greg.1.anderson commentedDrush updates Drupal core in a separate pass from the rest of the modules. What you see above is the upgrade of your modules worked, but when drush went back and tried to update core, it "forgot" there was an update.
Did it work after running pm-refresh and pm-update again?
Comment #5
Metztli CreditAttribution: Metztli commentedNo. Drush 4.0 RC3 did not perform an update on the theme nor subsequently on the Drupal 7.0 RC1 core.
Executing pm-refresh:
Refreshing update status information ...
Done.
And pm-update output is the same as previously posted. It does not matter if I specify --uri=http://localhost or --uri=http://127.0.0.1 in my testing environment.
Thank you for your reply.
Comment #6
jonhattanNot sure about danland not being updated (it works for me) but there's an issue in the second call to pm-updatecode to update drupal core.
so two calls to drush_pm_updatecode() are two calls to _pm_get_update_info(). On the second call, the update array for drupal contain no releases:
I guess it's something inside update.module: it is not neccesarily written to be used more than once within the same process.
Comment #7
jonhattanand it fail now and not before because of this recent adition to _pm_get_update_info():
see #836198: pm-updatecode uses obsolete info from cache for reference.
Comment #8
greg.1.anderson CreditAttribution: greg.1.anderson commentedDarn.
The right solution would be to fix up the drupal update so that it did not move core files out of the way per #759906: Update core without moving core files out of the way.. Then we would not need a second call to pm-updatecode. Unfortunately, I don't have time to do that right now, so we need to consider workarounds.
One possibility might be to use drush_invoke_process_args to run pm-updatecode drupal; however, I am a little reluctant to move in this direction at this point.
I think I would favor a fix that effectively "put things back the way they were"; for example, something like this:
Inelegant, but it would get the job done with the minimum amount of change in terms of code changes and code behavior. pm-updatecode drupal would then run more or less as it did prior to #836198: pm-updatecode uses obsolete info from cache, and we could just keep it like this until we were ready to move on with modernizing core updates.
What do you think?
Comment #9
jonhattanI think of using static variables.
Comment #10
jonhattanIt's harder. Unable to fix with static variables nor your suggestion Greg. Now looking deeper into update.module. Don't know if it's also an issue in d6.
Comment #11
jonhattanwell,.... it is easier than all of that. I hope I'm not missing something,...
Comment #12
jonhattannow more cleaner than #11.
Explanation:
* update core after non-core projects have been updated. No need to invoke again pm-updatecode. Just call _pm_update_core() if _pm_update_packages() went ok.
* drop several (now unneeded) messages to the user. pm-updatecode can update non-core and core in the same execution (previously it was only possible with pm-update).
* I realized $module_list passed to _pm_update_core() is not used: we are not disabling non-core modules. Is this really needed?
* So removed $module_list and comments related to the disable-update-enable pattern.
Comment #13
greg.1.anderson CreditAttribution: greg.1.anderson commentedBrilliant.
You are correct about $module_list; an earlier version of core update disabled all non-core modules, did the upgrade, and then re-enabled the modules it disabled, as is recommended in the Drupal upgrade procedures. However, none of the drush maintainers found it ever necessary in practice to disable non-core modules during an upgrade, so that code was removed. Apparently, a bit of it got left behind.
Comment #14
jonhattancommitted.
randomness in OP seems a network problem or like, and I've been unable to reproduce #3... it's almost impossible to not print any message after drush_confirm(). So I'll close the issue. Please reopen if problem persist or better, post new issues with detailed output: follow the bug submission guidelines at http://drupal.org/node/add/project-issue/drush
Comment #15
aanjaneyam CreditAttribution: aanjaneyam commentedCould it be explained in a bit simpler terms as to what went on. I normally use drush up to update modules and core. What is the exact difference between drush up pm_update and pm-updatecode ( and I also see another one in this discussion- pm-update-core). To me the first three seemed to do more or less same thing ( I may be wrong).
Comment #16
jonhattanamsri:
pm-update is pm-updatecode + updatedb. There's nothing more. In previous comments we've mixed command names and internal function names (_pm_update_core()).
Comment #17
aanjaneyam CreditAttribution: aanjaneyam commentedWell I think the problem seem to be still their. Exactly the same problem. Random recognition of available updates. Some of them not getting recognised at all. For example drupal available updates page say that there is an update available for wysiwyg module but drush just doesn't catch it despited repeated refresh with drush rf. I am using the latest All-Versions-HEAD dated 23 December 2010.
Also I don't know if it is related- running of drush makes the site unresponsive for a while. I mean after the drush command has ended executing.
Comment #18
aanjaneyam CreditAttribution: aanjaneyam commentedAnd now after several retries it recognises wysiwyg -
Comment #19
jonhattanyou mean there was a updated version of wysiwyg-7.x-2.x-dev and drush wasn't showing this but drupal was? Perhaps you're in the edge case of trying to update to a release when actually update service is not updated yet? Does it also happen if you download an old version of a module and try to update?
The update from rc2 to rc3 in drupal core is also randomly detected? or is it shown always?
Comment #20
jonhattanComment #21
aanjaneyam CreditAttribution: aanjaneyam commentedYes there was a updated version of wysiwyg-7.x-2.x-dev and drush wasn't showing this but drupal was. You may be right in saying that I may be in edge case. Well I will try again. I will retest and post. But there were two occasions (that's what I remember) when D7 rc2 to r3 was not recognised. I haven't tried downloading and updating an older version of a module.
Comment #22
kscheirerWhat if I want to update all contrib modules *without* updating core? That doesn't seem possible anymore. Core updates should be a different command IMO. Or at least allow me to use a --no-core flag.
Comment #23
greg.1.anderson CreditAttribution: greg.1.anderson commentedDoes --lock=drupal work? I haven't tried that...
Comment #24
daniorama CreditAttribution: daniorama commentedI also got the same problem with 'drush up'. It seems the number of modules detected is totally random and it has nothing to do with the update service not being updated yet. I have to pass 'drush up' 4-5 times to make sure all modules get upgraded.
Comment #25
aanjaneyam CreditAttribution: aanjaneyam commentedYes this is a problem which did not exit in drush 3.1. It is very much there and very frustrating.
Comment #26
DamienMcKennaHave been seeing this with Drush 4.3-dev too. I've closed #1081182: drush upc doesn't list themes as a duplicate of this.
Comment #27
DamienMcKennaI added print_r($update_info); to updatecode.pm.inc on line 27, you can see the full output attached, but here's one example:
In reality there was a beta2 that should have been identified, which the core Update Manager displays.
Comment #28
DamienMcKennaI added print_r($info); to line 62 in drupal_7.inc, which outputs the results of update_get_available(TRUE); and it still only lists a few updates rather than all of them, so this might be a core bug.
Comment #29
DamienMcKennaIt does appear to be a bug. I made the following changes to update.module:
The first print_r() is shown in drush-n1002658_log2.txt, the second is shown in drush-n1002658_log3.txt. One example of the discrepancy is that Views is listed in log2 from line 817 while it isn't in log3 at all.
Comment #30
DamienMcKennaHere's log3, I had to compress it as it wouldn't upload as a bare text file.
Comment #31
DamienMcKennaHere are two more log files, log4 contains the output of update_get_projects() while log5 contains the output from _update_get_cached_available_releases(). In these examples the Views update is listed in log5 but the Context update is not.
Comment #32
DamienMcKennaHave moved this bug to Drupal Core as the inconsistency stems from _update_get_cached_available_releases().
Comment #33
DamienMcKennaI added kpr($cache_items); to _update_get_cached_available_releases(). Starting with an empty {cache_update} table, each time I load admin/reports/updates it provides different results:
With each page load it also only displayed the status updates for the projects it had obtained, so e.g. the first time it only displayed the status for addressfield, cnr, commerce, context, date, devel, drupal, entity, field_group, link, media, media_browser_plus, media_flickr, styles. In these cases, modules that didn't have a valid update status would display the message "No available releases found".
Comment #34
DamienMcKennaReviewing update_get_available() further, it first loads the currently cached data, identifies whether the cache is either incomplete (i.e. it didn't finish loading all of the updates last time), out of date ($project['info']['_info_file_ctime'] > $available[$key]['last_fetch']) or missing release data, it assigns a "fetch task" for the plugins. update_fetch_data() then executes and limits itself to a brief period of time from the variable 'update_max_fetch_time'; 'update_max_fetch_time' defaults to the constant UPDATE_MAX_FETCH_TIME which has the value 5, i.e. by default it will only process the "fetch task" queue for five seconds before ending, so clearly it will only obtain a few updates with each batch.
Comment #35
DamienMcKennaJust to clarify, what appears as a Drush bug is really an undocumented feature in Drupal 7.
Comment #36
DamienMcKennaI added a core issue to add a field to admin/reports/updates/settings to allow the 'update_max_fetch_time' variable be edited: #1091992: Provide UI element to modify 'update_max_fetch_time' variable
Comment #37
DamienMcKennaAnother related core issue: #1092000: No validation to ensure that 'update_max_fetch_time' is numeric
Comment #38
DamienMcKennaAm reverting this back to being a drush issue as I believe it needs to possibly operate differently to Drupal core on this.
The issue boils down to Drupal core's functionality only polling a subset of the available projects for a status update at each call of update_get_available(), the specific subset depending on a) the 'update_max_fetch_time' variable, b) how long each status update takes. Due to the different environment dynamics of running updates via the shell (Drush) vs a web interface (Drupal admin UI) it may be worthwhile looping update_get_available() until all update statuses are obtained, so end users are provided what they actually want - status updates for all projects.
Comment #39
DamienMcKennaHere's a patch that makes how D7's project status update polling run on a loop until they are all updated.
Comment #40
DamienMcKennaComment #41
jonhattanAlternative solution in attached patch. It is based in http://api.drupal.org/api/drupal/modules--update--update.fetch.inc/funct...
Comment #42
jonhattanCleaner one.
Comment #43
DamienMcKennaHere's an updated version of your patch using the new Git patch workflow.
Comment #44
jonhattanso it works for you?
Comment #45
aanjaneyam CreditAttribution: aanjaneyam commentedNow which patch should I try #42 or #43 because I am also suffering from this annoying problem from the very beginning which led me to open this issue.
Comment #46
DamienMcKenna@jonhattan: Yes, the changes in #42 worked for me but I had to re-create it as it didn't apply cleanly.
Comment #47
jonhattan@amsri use the patch in #43.
Comment #48
aanjaneyam CreditAttribution: aanjaneyam commentedafter applying the patch and running drush up I get error:
Comment #49
DamienMcKenna@amsri: the patch should not have changed anything to cause it to appear broken like that, does the same problem happen without the patch? Try this:
Comment #50
mcjim CreditAttribution: mcjim commentedThe patch in #43 does work… but the first run takes ages. Initially I though the process had hung, but I kept my twitchy fingers off the keyboard and sat as patiently as I could. I didn't time it, but it took over a couple of minutes.
So, I tried it again, with timer, and it reported back in 22 seconds.
So I cleared the caches and tried again. It took 22 seconds, which is fine.
So, apart from that initial really long run, this seems to be OK (PHP 5.2.14, OS X 10.6.6).
Comment #51
Mac Ryan CreditAttribution: Mac Ryan commentedABOUT THE PATCH @ #43: I can confirm that after applying the patch it takes ages (for me not only the first run, but also the second, third, fourth one...) to get the update process started. The process takes a long break after "Done." (which in turns comes after "Refreshing update status information ..."). However it does fixes the problem. :)
Comment #52
jonhattanUpdated patch on current master. Sorry it is a lazy
git diff
.About the time for the first run: as you figure out it is the batch process running without a progress output. I have been reading the code to try and print a live message but I don't know if it is even possible,.. other command based in a batch process also lacks such a progress message, for example l10n-update. Will do some more tests soon.
Comment #53
zilverdistel CreditAttribution: zilverdistel commentedSubscribing.
I think this might be related to an issue in core: http://drupal.org/node/952394 (No available releases found - cache issue?) ...
Comment #54
Cyberwolf CreditAttribution: Cyberwolf commentedSubscribing.
Comment #55
laura s CreditAttribution: laura s commentedDrush 4.4: I manually applied patch to commands/pm/update_info/drupal_7.inc. Direct match on code. Result: Delay was up towards a minute to get available updates, but list is comprehensive, patch works wonderfully. Full results for the first time since working on D7.
Comment #56
andrewyager CreditAttribution: andrewyager commentedA relatively simple workaround for those experiencing this bug is to modify the update_max_fetch_time variable to 30 seconds (or higher if needed) in drush as per http://drupal.org/node/952394#comment-4321800.
drush vset update_max_fetch_time 30
Comment #57
HnLn CreditAttribution: HnLn commentedsub
Comment #58
mlncn CreditAttribution: mlncn commented#43 works after running
drush rf
, thendrush up
.Bumping to critical (Drush is giving significantly incorrect info saying everything is up to date when it is not) and positing that we need another change:
If you name the module to update, it should check just that module and not ever time out and tell you there are no updates available when you've named a specific module that does need an update.
checks all projects and has this problem (or with the patch, an unnecessarily long run), when there's no reason to do so, correct?
Comment #59
jonhattanI'm working on a better solution.
Comment #60
jonhattanWith attached patch drush creates a fetch task for any project needing fresh update status information
a) if cache has expired
b) if it has changed on disk (ie. manual download)
c) or we don't have any information cached (recently downloaded project or pm-refresh)
... and all pending fetch tasks are run by the batch process. It also prints what projects it is fetching information of, in real time.
@mlncn the usage for
drush up xyz
has always been to just update xyz after computing the whole update status. Feel free to open a new issue for that, as there's no straightforward fix for this.Internal changes:
* drush_backend_batch_process() now accepts a 2nd argument: $interactive. It allows to pass the #interactive flag to the backend call.
* new hidden command pm-updatestatus-batch-process to allow running a custom batch operation function instead of the update.module provided one. Only purpose: drush_log() of the progress messages.
Comment #61
DamienMcKenna@jonhattan: I tested your patch above (#60) against the latest Drush master branch, manually truncated all of the cache tables on a local install on my OSX 10.6 system (MAMP 1.9, PHP 5.2) and had the following results:
The per-project checking & display is a definite improvement. I like it! Good work!
Comment #62
Anonymous (not verified) CreditAttribution: Anonymous commentedsubscribing
Comment #63
jonhattansorry for the delay in commiting this but master has changed again and I'm trying to merge both.
Conflicting changes are: http://drupalcode.org/project/drush.git/commitdiff/2048de5 related to #766080: Windows support for drush: escaping the path to drush in backend invoke and elsewhere
Comment #64
jonhattanFinally commited it without displaying a message per project. I think the proper solution to allow for this is to address #1058874: drush_backend_invoke buffers output till end of command. Show progress. and then implement the rest here.
Comment #65
jonhattanComment #66
jonhattanI did introduce a regression, already fixed. Attached is the good patch.
Comment #67
jeffschulersubscribing.
the
drush vset update_max_fetch_time 30
workaround is working for me in the meantime.Comment #68
zilverdistel CreditAttribution: zilverdistel commentedOn one site, I had to set
drush vset update_max_fetch_time 60
.Comment #69
dwwUgh, sorry for all the trouble in here, folks! I only happened to find this issue via #1091992: Provide UI element to modify 'update_max_fetch_time' variable (I missed it while it was considered a core update bug). All of your troubles stem from #597484: Use the Queue API to fetch available update data. And yeah, I forget that the Update API (such as it is) is shared by drush. So, when I made changes in D7 to make the Update manager interface in core (try to be) better, I accidentally made drush's life much more difficult. :( Mea culpa! I'll hopefully remember this in future work on update manager.
If the API is inconsistent and buggy, I'm all in favor of new bugs filed in core's queue to clean up the interface to be more suitable for drush (and hopefully, therefore, other needs).
Thanks/sorry,
-Derek
Comment #70
dwwp.s. Another option here is to add some code to drush that knows how to drain the fetch updates queue via a batch or some other more interactive means...
Comment #71
jonhattan@dww patch in #66 launchs a batch process, I think this is what you mean in #70.
OTOH I'm working on #1032626: Allow drush pm-update w/o update.module.
Comment #72
msonnabaum CreditAttribution: msonnabaum commentedBackported.
Comment #73
jonhattanin #64 I discarded the code to allow displaying a message per project. Attached patch contains that piece of code. To be tested in conjunction with #1058874: drush_backend_invoke buffers output till end of command. Show progress.
Comment #74
jonhattan#73 discarded in favour of #1186480: Make the batch api benefit of backend show progress automatically..