Here is an update script which can be run from the shell to process the updates.

I have used this here at Lonely Planet to run any updates once we have committed any changes to our local repository. We have a script which will checkout the new commits and run any updates that are required.

If the updates fail then the site is left in maintenance mode to work out why the updates failed.

We are also using part of this as a part of out nightly build testing in conjunction with simple test.

Files: 
CommentFileSizeAuthor
#90 233091-90.update_inc.patch25.7 KBdww
Passed: 11876 passes, 0 fails, 0 exceptions
[ View ]
#89 233091_update_inc.patch25.69 KBanarcat
Passed: 11905 passes, 0 fails, 0 exceptions
[ View ]
#84 233091_update_inc.patch25.73 KBanarcat
Passed: 11876 passes, 0 fails, 0 exceptions
[ View ]
#81 233091_update_inc.patch25.74 KBanarcat
Failed: Failed to apply patch.
[ View ]
#76 233091-update-sh.patch10.76 KBgordon
Failed to run tests on MySQL 5.0 ISAM: failed during invocation of run-tests.sh.
[ View ]
#59 update.inc_.patch40.99 KBanarcat
Failed: Failed to apply patch.
[ View ]
#37 test_update.tar_.gz445 bytesgordon
#33 update.sh_.patch10.76 KBgordon
Passed on all environments.
[ View ]
#31 update.sh_.txt - Drupal 6 version10.21 KBgordon
#31 update.sh_.txt - Drupal 7 version10.25 KBgordon
#22 update.sh_.txt8.34 KBgordon
#21 update.sh_.txt8.44 KBgordon
#20 site-fix-update.patch1018 bytesdwarner
Failed: Failed to apply patch.
[ View ]
#16 update.sh_.txt8.27 KBgordon
#12 update.sh_.txt8.25 KBgordon
#11 update.sh_.txt8.26 KBgordon
update.sh_.txt7.97 KBgordon

Comments

moshe weitzman’s picture

FYI, this duplicates a lot of what drush offers (verbose/dry-run). It does run pending updates which is very cool and not yet in drush ... We really need to decide if drush is core worthy because it makes little sense to use drupal.sh when we have such a nice framework in drush.

gordon’s picture

I had a quick look at drush and didn't see the update.php integration.

But I found that really it couldn't see that we couldn't get maximum control over the update if it was being done via drupal.sh

Also from what I could see with drush is that it was a big utility that tried to be everything, when I much prefer a set of small utilities that do 1 thing really well.

I also want to add some additional functionality in being able to execute a list of particular updates from the shell.

Here at LP we currently have this running live.

moshe weitzman’s picture

i don't really agree that drush is a big utility or tries to be everything. what should it not do, for example? if you do one off scripts like this, you have to code most of the command line options every time (root, user, verbose, dry-run, ...). further, you don't have a directory of available commands with help. drush is quite drupalish in how it lets commands plug themselves in and provides services that those commands can use.

anyway, this is a terrific script. i would be happy if it were committed to core.

gordon’s picture

Thanks, This does still requires a little work, but I felt this was at a point where I could put it out to the community.

As for the command line switches I have been thinking about that, and being able to create something like getopts() to handle this so we can basically pass an array which will pull out the switches that were needed.

There are a couple of other things I would like to do as well, so I will keep plodding along.

Frando’s picture

I actually intended to do some work to bring Drush (or at least its API and drush.php) into core. I even started some basic work for Drupal 6, there's a preliminary patch somewhere in the queue (might not be of much use now, Drush has also advanced), but then flew off to south america, where I still am.

I still think bringing the Drush API into core would be a great idea, even if in a simplified version (e.g. drush not being a module anymore, just an include). Drush in core would also mean that contributed modules could simply provide additional drush shell commands/services, which would be very cool.

I won't be able to work on this until I'm back home, though, which will be in May.

Dries’s picture

Status:Needs review» Needs work

I like -- and I certainly consider a script like this to be core worthy. This looks very convenient to me. There are a couple of typos in the help text, and a couple of code style violations though.

PHP has a getopt() function (http://www.php.net/getopt) that you might be able to use, and that might counter some of Moshe's concerns.

I haven't look at the drush code recently so I don't know how bulky it is or has become.

gordon’s picture

I just investigated getopt() and it doesn't support long options on all platforms until 5.3. They have written their own version

I was thinking that I write our own version of getopt() which supports long arguments. but getopt() would be the better choice.

I will look at it at it some more, and post a new version.

Also I think that we need to move update_sql() to update.inc since I did have to duplicate this into update.sh

Dries’s picture

gordon, actually, I'd like to get rid of update_sql() and use the default database functions moving forward. If it isn't already possible, I think this is something http://drupal.org/node/225450 should enable us to do. Either way, the question is: how can we get rid of update_sql()?

gordon’s picture

Would anyone object to using the Pear getopt() Module, http://pear.php.net/package/Console_Getopt

The reason that I ask is that because getopt() is not really going to be available for all platforms until PHP 5.3, and writting a drupal_getopt() means that we need to still handle the --root switch by it self so that we could find the shell.inc which has the reusable function, or include this function into the script.

If I was to use the Console::GetOpt from pear I can just put up an error if it is not installed and then run

pear install console_getopt

to install it.

Also this means that all scripts that run from the shell can easily implement the standard switches for drupal shell commands with not much work.

moshe weitzman’s picture

@gordon - getopt is one more thing that drush just handles for you - fyi.

i will be turning this into a drush command soon in case people are looking for this.

gordon’s picture

StatusFileSize
new8.26 KB

Here is an updated version which uses getopt(). I have had to add short options which need to be used before php 5.3 on most systems, but after php 5.3 you can use the long options.

I have been using the script live on sites to automate the upgrade when checking out new versions of a site into dev/test. Also at Lonely Planet we used this to checkout and automatically run tests after every commit.

gordon’s picture

Status:Needs work» Needs review
StatusFileSize
new8.25 KB

I found a few problems with the change to getopt() which have been fixed now.

moshe weitzman’s picture

I just tried to run --versions and got "ERROR: sions not found.". No idea where it comes from.

gordon’s picture

The problem is that on versions of php before 5.3 not all versions (in fact all versions I have tried) do not support long options in getopt().

--versions is the only one that I could not think of a short code for. Suggestions are welcome.

moshe weitzman’s picture

I guess -e will work.

gordon’s picture

StatusFileSize
new8.27 KB

I have added 'e' as the short argument for the versions switch.

Once this is getting closer I will need to create a patch for drupal.sh which starts to use getopt as well.

moshe weitzman’s picture

Status:Needs review» Needs work

This works well for now. A couple minor nits:

* -e is not returning anything for me
* found a typo - search for "sprecify"
* "If there are any updates are not processed successfully then the script with exit with a return flag of 1.". This is missing a "that". Also, I'd like to inform the user what he will see at the command line in this case.

moshe weitzman’s picture

Do you have any test code for testing a batch API style update like system_update_6021()?

moshe weitzman’s picture

* create_function() looks a bit odd. Perhaps add a comment why thats needed. I don't know.
* Also, it looks like there is code in global scope at top and bottom of the script. Please consolidate it all to one of those places.

Sorry for multiple posts.

dwarner’s picture

StatusFileSize
new1018 bytes
Failed: Failed to apply patch.
[ View ]

Attached is a patch to remove "detection" of the site so that Drupal can appropriately fall-back in multi-site situations (ie, from 'www.example.com' to 'example.com' to 'default' instead of detecting only 'www.example.com' as a valid site).

This also switches "/usr/bin/php" at the top to "/usr/bin/env php" which should work in more situations.

gordon’s picture

StatusFileSize
new8.44 KB

I have made the changes specified except the change to the site, as the system that you are specifying is the directory and not a host name.

I need to do some testing on the batch updates.

gordon’s picture

StatusFileSize
new8.34 KB

Here is a version that I fixed up to run on Drupal 7

gordon’s picture

Status:Needs work» Needs review

Patch failed to apply. More information can be found at http://testing.drupal.org/node/14025. If you need help with creating patches please look at http://drupal.org/patch/create

yched’s picture

Status:Needs review» Needs work

The current code doesn't support multipass upgrades.
system_update_6021 mentioned in #18 is not a good example, because it is 'old style' (pre batch API) : uses $SESSION to store persistent data.

Look at system_update_7004 instead (or http://cvs.drupal.org/viewvc.py/drupal/contributions/docs/developer/exam... for simpler code)
multipass updates take a $sandbox argument, which is persisted between iterations (safer than $SESSION)

I guess the script needs to support both $SESSION and $sandbox styles.

Also, thare have been a few changes in D6 update.php :
- $ret['#abort'],
- run updates for all modules, including the disabled ones

dlhubler’s picture

Title:Update drupal sites from shell.» drastically modified version

gordon, I took your script, modified it quite a bit and put it together in a set of utilities I've collected.

http://acquia.com/blog/drupal-cli-utils

I didn't integrate all your functionality because I wasn't sure I could test it appropriately but we can work on bringing them back in along with yched's fix if you're interested.

It may not be as straightforward as your single-file, shell script version, but it's clear that many people need this functionality so my hopes was to try to put it in a bundle that could support many different developers needs and modifications by having accompanying unit tests and an extend-able design.

dlhubler’s picture

Title:drastically modified version» Update drupal sites from shell.

revert change to issue title

moshe weitzman’s picture

At first blush, It looks like dhubler's script does not support progressive updates either. I think Gordon's script is based on an older version of update.php. We are going to have to look at current update.php and work back from there. See yched's comments.

gordon’s picture

Yes it was based upon an older version, (D5) but I am in the process of updating it to work with D7.

The last version I did was from D7

moshe weitzman’s picture

i'm happy to test when a D7 or D6 version is ready. Let us know, gordon. This is tantalizingly close.

gordon’s picture

Status:Needs work» Needs review
StatusFileSize
new10.25 KB
new10.21 KB

I have done some work and updated this to use the batchapi so it should handle everything the same as update.php

Dries’s picture

Status:Needs review» Needs work

- t('No Updates available') should use a lowercase 'u'.
- t('Modules Requiring Updates:') should not be camel-case.
- t('Pending Updates:') should not be camel-case.
- I'm not sure I understand the value of the 'dry-run' option in this case. AFAIS, it wouldn't detect errors.
- I think putting the site in maintenance mode is something we should always do -- just like we do for update.php nowadays.
- String concatenations use more spaces nowadays.
- Would be great if we could reuse more code with update.php but haven't studied this to verify if this would make sense. I don't want this script to be a maintenance bottleneck.
- Still somewhat undecided about the fact that this is core-worthy.

gordon’s picture

Status:Needs work» Needs review
StatusFileSize
new10.76 KB
Passed on all environments.
[ View ]

I have made the following changes are per Dries' recommendations.

* Fixed up capitalization of strings
* Removed dry run function
* Fixed up coding standards
* Changed to always put into maintenance mode. Message is optional.

I looked into sharing more code between update.php and update.sh and if we move update_do_one() out of update.php to something like install.inc I could just reuse it. Also with a bit of work I could also move update_batch, but that will require having some duplicate named functions which I think is not a good idea, but I am cool either way.

I do think that this is something that is missing from core. With update.sh in core it means that anyone can create easily create a much better deployment process without too many issues. I have started using this on all my client sites to automatically run update processes as soon as I deploy a new version or any updates to a site.

So with Git I add the following to my post-update script in the hooks directory.

cd ../..
./scripts/update.sh -l
status=$i
if [ "$status" != 0 ]; then
  ./scripts/update.sh -u
fi

This basically means that all the building blocks are there to create a much more effective deployment system without the need for a heap of investiment in searching for solutions and then developing something.

As this is a simple building block it should be able to fit into just about anyones deployment process.

I would also like to develop some tests, but I am not sure if it would be possible. If anyone has any ideas please let me know.

Gordon.

moshe weitzman’s picture

I would like to see this in core too.

I actually don't like unconditionally putting the site into maintenence mode. I am fine if it is the default behavior. If one uses boost module or file based cache, your site continues to serve pages happily even while updating. But not a big deal.

Dries’s picture

Alright, let's get it in core then. :)

My key requirement for inclusion in core is hat we try to keep this script small by reusing as much as possible from core and by making the necessary changes to provide a well-designed and well-defined API. The reason I'm asking is because this could open the door for tools like apt-get, rpm, yumm to integrate with our update/upgrade system.

moshe weitzman’s picture

What is the best way to test this? Do you have a dummy update function that does batch stuff?

gordon’s picture

StatusFileSize
new445 bytes

I was using this, but it doesn't have a batch processed update but it using a batch processing api so it should work.

What I would like is to work out if we can create a test script for it. Maybe I need to talk to webchick and see if she can think of a method.

Gordon.

Damien Tournoud’s picture

Moshe wondered on the IRC how we could test this.

It should be possible in our testing framework, but we will have to make update.sh simpletest-aware (ie. add a --simpletest parameter that sets $_SERVER['USER_AGENT']).

The plan I outlined on the IRC is this one:

(1) we create a mock module with update function(s),
(2) we use it in a test (passing it as a parameter to the setUp() function),
(3) we forcibly downgrade its schema to 1 by tampering with the system table,
(4) we call update.sh with the --simpletest parameter,
(5) and finally, we assert that the update functions were run.

Status:Needs review» Needs work

The last submitted patch failed testing.

lilou’s picture

Status:Needs work» Needs review
SeanBannister’s picture

sub

adrian’s picture

I wrote my own scripts for this in d5 / d6 for aegir.

http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/provision/p...

catch’s picture

Status:Needs review» Needs work

Subscribing. Drush is really handy, having something like this in core would be handy too.

There's a few abbreviated variable names in there - $a, sh should probably be shell. Didn't try the patch yet.

chrisschaub’s picture

This is good stuff, and I like the focused nature of the script. But the missing ingredient is first updating the files. Why not add the ability to checkout changes (patch the site) via CVS for core and contrib? This wouldn't add much to the code and would make it a complete upgrade/update script. Right now, it only does the last part of an upgrade. You only need to support CVS (not wget or curl etc.) since mostly advanced users will be using this type of script anyway. Also, CVS is a much friendlier, lower bandwidth, way to download updates from drupal.org.

Drush can sort-of do this for contrib but not core. And now the CVS functions have been moved out of the main Drush module which makes it not as useful as a standalone update/install script. I was hoping Drush would be able to do this, but it seems to be moving in a different direction, namely drush_extras.

moshe weitzman’s picture

FYI, all in one scripts are not the unix way. Gluing together small, focused scripts is more common.

chrisschaub’s picture

There are sometimes very good reasons to keep a functional process (like updating Drupal, for example) in one script. Error handling, rollback and recovery come to mind. Also, you don't usually update the file system without next running update.php -- and you don't usually run update.php without first updating the file system. These are not disconnected processes that should be separated. I could see if the processes were more standalone, but they're usually not. I only suggested cvs because it is a standard for the Drupal project.

Adding cvs "update" support is not much more code, not an "all-in-one" monster script, but a much simpler, more robust, approach for site maintainers. Just consider the "glue" and error checking a wrapper script will have to contain to ensure that multiple update scripts work well together, especially scripts that run via cron. This becomes a very serious consideration when updating hundreds of sites.

nk_’s picture

subscribing

blit32’s picture

I think having one script to update the files and another to run update.php is better then combining them because some people already have or will have a process to update the modules files, including there custom modules (I am one of those people). The problem is there is no good way to run update.php via a script.

I wouldn't mind seeing another script to update the module files, I just don't think both should be one set process. If a script were to do both update the files and the database then IMO there should at least be a flag where a user can choose to do one or the other.

gordon’s picture

update.sh does only what update.php does. but because it is a script it can be called from script after all the files have been updated on the server.

gordon’s picture

Also I have been doing some work for this. See #383342: Getting access to the test details which adds test scripts for update.php so that I don't break it when I make the changes for update.sh

drewish’s picture

subscribing.

neofactor’s picture

subscribing

sepeck’s picture

I just want to check. This is a Linux only solution isn't it?

egfrith’s picture

@sepeck: The script should work under Windows with a small modification. From the documentation at the top of the script:

* Check for your PHP interpreter - on Windows you'll probably have to
* replace line 1 with
*   #!c:/program files/php/php.exe
sepeck’s picture

@egfrith - thanks, I missed that line.

gordon’s picture

I am not sure changing the first line will work in the standard command.exe enviroment. However you can just do a php update.sh in the command line to run it.

anarcat’s picture

For me, this should be done through Drush.

We're working hard on this in the Aegir project, and currently have to reimplement the functions in update.php right now, which is fairly annoying and error-prone.

The best way to fix this would be to have an update.inc API that would be sourced by update.php (which would be the Web-based interface) and an eventual update.sh (or drush scripts).

This would help us a lot. The patch is trivial (take every function in update.php and throw it in a file in includes/) so I will not include it, unless asked for it.

swentel’s picture

Subscribing, need this functionality badly, will review later!

anarcat’s picture

Status:Needs work» Needs review
StatusFileSize
new40.99 KB
Failed: Failed to apply patch.
[ View ]

So here's my stab at it: it keeps the update.sh script mostly untouched (although I do address #43), but I factor out of update.php the functions we are using in drush/aegir into a new includes/update.inc file.

I noticed that none of the functions used in update.php are used in update.sh, so I guess that means that update.sh isn't designed for major upgrades (d6->d7, for example). Anyways, it assumes it will be able to bootstrap a full drupal, which will not work if you are sitting in a D7 codebase with a D6 database, for example.

That can be fine for an initial update.sh, i guess.

gordon’s picture

#57 The problem with Drush is that it is something that is installed separately from Drupal, and covers a lot more areas. This script will be something that is included with Drupal and will always be there. Drush will not.

As for the API, I still disagree as with the Aegir project you should be able to execute a shell script to do the updates. Using this will mean you don't have to worry about how things are done, and you will not gain or lose any additional functionality by doing this.

anarcat’s picture

#60

I'm not saying drush should be the only way to update Drupal modules. I'm saying Drush and third parties shouldn't have to bend over backwards and basically rewrite update.php (and maintain multiple branches of it for the multiple drupal versions) to be able to do updates.

As for Aegir, yes, we can execute shell scripts, but PHP functions are much more flexible and I fail to see the advantage of the shell layer where we can (trivially, as demonstrated with the patch) provide a simple API to reduce the code size everywhere.

Note that my proposed patch doesn't remove update.sh or any functionality from it, but enables drush and Aegir to do updates much more easily.

adrian’s picture

Status:Needs review» Needs work

Compare :
index.php : http://api.drupal.org/api/file/index.php/6/source
update.php : http://api.drupal.org/api/file/update.php/6/source
install.php : http://api.drupal.org/api/file/install.php/6/source

Whereas the first is just the way you get into the system, the last two are the entire system in and of themselves.
You can't just include the file to have access to the functions either, because there is logic directly in the file that makes things
difficult for you too.

You can't use any of the functions, without re-implementing them. The logic code of the entry pages, is written in a way that _SPECIFICALLY_ stops you from just including it.

The install file is the worst, because you can't include it without it triggering an exit(). And the entire thing is written so that it drupal_goto's , which also results in an exit().

it is too much to ask to make the code be includable?

Your update.sh is also php btw, so you would run into almost all of these same issues. Your script completely fails to take into account the major version updates. So it will fail at the most important time a site needs to be updated.

adrian’s picture

This is what your code is missing (for the d6 version) to be able to do d5 -> d6 upgrades :

http://drupalbin.com/8553 <- about 50 lines of code.

You are missing calls to :
http://api.drupal.org/api/function/update_fix_d6_requirements
http://api.drupal.org/api/function/update_fix_compatibility
http://api.drupal.org/api/function/update_create_batch_table
http://api.drupal.org/api/function/update_check_requirements

Which in turn require :
http://api.drupal.org/api/function/db_create_table/6
http://api.drupal.org/api/function/update_check_incompatibility/6

Additionally, some of the updates then require, for them to even work :

http://api.drupal.org/api/function/db_add_column

<?php
Moya
:DRUPAL-6 adrian$ grep -r db_change_column modules/*
modules/statistics/statistics.install:      db_change_column($ret, 'accesslog', 'sid', 'sid', 'varchar(64)', array('not null' => TRUE, 'default' => "''"));
?>

http://api.drupal.org/api/function/db_change_column

<?php
Moya
:DRUPAL-6 adrian$ grep -r db_add_column modules/*
modules/system/system.install:      db_add_column($ret, 'url_alias', 'language', 'varchar(12)', array('default' => "''", 'not null' => TRUE));
modules/system/system.install:      db_add_column($ret, 'node', 'language', 'varchar(12)', array('default' => "''", 'not null' => TRUE));
modules/system/system.install:      db_add_column($ret, 'blocks', 'bid', 'serial');
modules/system/system.install:      db_add_column($ret, 'filters', 'fid', 'serial');
modules/system/system.install:      db_add_column($ret, 'flood', 'fid', 'serial');
modules/system/system.install:      db_add_column($ret, 'permission', 'pid', 'serial');
modules/system/system.install:      db_add_column($ret, 'term_relation', 'trid', 'serial');
modules/system/system.install:      db_add_column($ret, 'term_synonym', 'tsid', 'serial');
?>

And that's only core.

Currently, the only way to actually get access to any of this functionality is to copy / paste it.

Owen Barton’s picture

Subscribe

adrian’s picture

We have decided to port the commands for updating and installing Drupal from Provision to Drush : http://drupal.org/node/415152

This means that drush will have the ability to update and install Drupal 5, Drupal 6 and Drupal 7 sites, both minor and major version upgrades, will full support for modules, themes, install profiles and translations.

We still bitterly need the update.php and install.php files fixed, so they can actually be re-usable, and don't need to be copied nearly verbatim into our own project.

chrisschaub’s picture

This drush news is very welcome. We really need a way to update the file system and then, without much delay, update the database. I know it's more modular to have separate little scripts for each function, fine, but you have to run them one after the other anyway! Update filesystem, then run database updates. Having this functionality in one module like Drush is a very good idea. Users are just looking for a dependable way to upgrade Drupal, especially for many sites. No complete solution exists today. Drush is really good, and it could be a starting point for building an auto-update interface for end-users, not just sysadmins. It might also be able to replace update.php and install.php if it was more full-featured.

adrian’s picture

Lutegrass : check out http://groups.drupal.org/aegir . It provides a Drupal based front end to Drush commands.

Drush already has the pm commands too, which can download packages.

We also already have an api so drush commands can invoke each other.
Once pm install has downloaded the new modules, it can do

<?php
 drush_backend_invoke
('update');
?>

To call the drush update command and run any updates, with full logging, error handling and recovery.

chrisschaub’s picture

Yeah, the missing db functions are a real killer for this script. Can't we all just put our efforts toward Drush now that it's standalone?

chrisschaub’s picture

Can you post the full package, already patched for Drupal 6? It's hard to tell which versions you are patching against at this point. Thanks.

bjaspan’s picture

Adrian asked me to review this issue and the patch in #59. I have read the issue and the patch but have not tried executing any of the code. Comments:

* It's hard to argue against moving useful functions from install.php/update.php into include files so they can be used separately from the hard-coded logic of install.php.

* The patch moves code from update.php into includes/update.inc but does not do that for install.php. Scanning through install.php, perhaps all the code there really is tied to the particular UI workflow and is thus not really useful (perhaps this means it should be refactored?).

* db_add_column() and db_change_column() are obsolete (PostgreSQL-only and pre-Schema API) and need to be removed, but I guess that is a separate topic for #190162: Schema API: rename field to column. It just kills me to see them still around.

* It isn't clear to me why drupal_get_updates() and _update_filter_updates() are in update.sh and not includes/update.inc. The rest of update.sh seems to be tied to the particular UI workflow, but these two functions do not. If they are going to be in update.sh, they should probably not use the name "drupal_get_updates" as that might be wanted in update.inc in the future.

* I'd too would love to see this as part of Drush instead of a stand-alone script, but "Drush in core" is a separate topic that should not hold this one up. If Drush makes update.sh obsolete some day, we can cvs remove it then.

adrian’s picture

I just committed the update command to Drush HEAD. http://drupal.org/cvs?commit=196574

It supports Drupal 5 through Drupal 7, with complete support for minor and major version upgrades and complete logging and error handling.

dww’s picture

I just read this issue, haven't looked at the patches. A few comments:

- Completely agree with keeping the functionality to run the DB updates in a separate script. There's no way to automate the process for updating the code in the filesystem for every site, since there are so many different deployment strategies: direct cvs, (svn|git|bzr) + vendor branches, tarballs, etc, etc. If we tie the DB updates to any one of those, we can't reuse it on the others. Evil.

- Completely agree with moving useful functions into .inc files to be reused where needed.

- I was worried about the potential for user confusion and namespace conflicts when update_status moved into core and people pushed to rename it to just "update.module". I have a template issue reply I use when moving issues out of the "update.module" component into the "update system" component, since people frequently submit things dealing with update.php to "my" component. I'm once again worried (but don't have a solution for) the creation of includes/update.inc. :( Is that DB update functions, filesystem update functions, or both? Ugh.

- If there are parts of install.php that could be reused if moved into includes/install.inc, I'm all in favor, but I don't think that should hold up this patch -- that should be in a separate issue.

raintonr’s picture

Stumbled upon this issue while trying to figure out a way to script (either PHP/Shell/whatever) updates for D5->D6 upgrade for very complex site (loads of modules, theming, etc).

The problem seems to be that Drush and the scripts here seem to have a lot of code duplicated in them that is inside update.php. I'm sure everyone here realises duplicate code is bad, so am curious as to why this is? Are there plans to separate the update.php functions to make them callable from other scripts such as Drush or any custom coded upgrade scripts.

IMHO, the ability to script/automate updates and run again and again and... until it works is critical to many sites. The update script is nice for modules and the like but for major upgrades a web interface is just really cumbersome.

boombatower’s picture

Subscribe.

moshe weitzman’s picture

Anyone up for rerolling #59. Thats just a code reorg that is uncontroversial.

gordon’s picture

StatusFileSize
new10.76 KB
Failed to run tests on MySQL 5.0 ISAM: failed during invocation of run-tests.sh.
[ View ]

Here is a reroll of my version, I am going to be looking at this soon as I really want this in core.

naught101’s picture

subscribe

Why not put drush in core? It's small compared to drupal, if people don't already use it, then it won't affect them...

moshe weitzman’s picture

I'd like to see a patch get committed from this issue. The latest patch duplicates some core code which doesn't make too much sense to me. I would think we want something along the lines of #59 (and maybe some CLI code too).

David_Rothstein’s picture

See #524728: Refactor install.php to allow Drupal to be installed from the command line - which I just posted - for the related issue involving install.php.

Here, I would also agree that it makes sense to do something like #59 before trying to have a shell script in core. (First refactor update.php to be reusable, then actually reuse it...) It would be even better to rework the code so that the UI and API are separated out, but even just moving functions wholesale out of update.php seems like it's a help, so maybe fine to just start with that.

anarcat’s picture

I think this should focus on providing a cleaner API for now. Let's not focus on providing a commandline tool for now (drush does that very well if you really need to), and let's focus on making the core easier for people that do write those tools (which may include a tool shipped with core, but honestly, i think it's complex enough that it could belong to an outside tool).

I'll see if i can reroll that patch from #59.

anarcat’s picture

Status:Needs work» Needs review
StatusFileSize
new25.74 KB
Failed: Failed to apply patch.
[ View ]

Here's another stab at this. I have tried to port the patch to the best of my knowledge. It's getting harder to chase HEAD as I can't just reapply the patch: I need to actually redo the copy/pasting between update.php and update.inc for chunks that fail (since the HEAD version code may change).

So I have purposefully removed the controversial update.sh for now. The reasoning is that the factoring out patch will be useful for an eventual update.sh *and* other projects like drush right away. It's a trivial patch that shouldn't take away any functionality and it's quite hard to maintain/chase head with. So I would just like to get that out the door so that people can actually start improving the commandline version already.

Regarding that, I don't think it's wise to package a .sh in Drupal, especially if it's actually not a shell script. A php command line script should be named .php, not .sh. Furthermore, I think that just running update.php from the commandline should do the right thing: it's perfectly possible to detect commandline vs webserver environment and that should just be dealt with appropriatly. #524728: Refactor install.php to allow Drupal to be installed from the command line apparently takes that approach and I think that's very nice.

So let's treat the .sh and the refactoring issues separatly. I can also open a different issue about the refactoring if that creates conflicts here, but I think both patches can evolve in parallel here, and we should keep a focus on the refactoring until it lands.

Status:Needs review» Needs work

The last submitted patch failed testing.

dww’s picture

Not only does it fail to apply, I was getting WSOD when visiting update.php after (manually) applying the patch. Didn't have a chance to debug. In principle, I totally support the direction of this patch. It just needs to work. ;)

anarcat’s picture

StatusFileSize
new25.73 KB
Passed: 11876 passes, 0 fails, 0 exceptions
[ View ]

Let's try this again: it seems I'm having trouble generating a patch that pfir will appreciate with git... I'll look into the WSOD issue now, so I'm leaving this as needs work.

anarcat’s picture

Status:Needs work» Needs review

Actually, I'm getting a WSOD whether or not the patch is applied here right now... so I'm bumping this back to needs review.

drewish’s picture

anarcat, yeah if you're using git don't forget the --no-prefix option. otherwise you make the test bot cry.

anarcat’s picture

Yep, i'll keep that in mind. It seems my patch passes all tests, can we move on? ;)

I opened a separate issue for the unrelated WSOD bug: #533760: 0.4 - D7 support: set $conf['cache'] properly.

anarcat’s picture

Status:Needs review» Needs work

Alright, it seems the issue was Aegir-specific ($conf['cache'] = 1), and there is still an issue now:

Fatal error: Call to undefined function update_prepare_d7_bootstrap() in /var/hostmaster/drupal-7.x-git/update.php on line 335
anarcat’s picture

Status:Needs work» Needs review
StatusFileSize
new25.69 KB
Passed: 11905 passes, 0 fails, 0 exceptions
[ View ]

So sorry about that, here's a patch reroll that actually works. I naively thought that just copy pasting the functions around would be enough (and it was when I first rolled out this patch) but things are a bit more complex now.

I was able to upgrade from d6 to d7 with this patch.

dww’s picture

Status:Needs review» Reviewed & tested by the community
StatusFileSize
new25.7 KB
Passed: 11876 passes, 0 fails, 0 exceptions
[ View ]

Re-rolled to clean up a bit of the PHPdoc. Otherwise, identical patch. Works fine when I tested running update.php with a real DB update. Trivial patch to just move some functions into a .inc file where they can more easily be reused by anyone who needs them. RTBC.

p.s. Why are db_add_column() and db_change_column() in here and not in the DB API?

gordon’s picture

I have been looking at this quite a bit, but I was having a lot of issues with the update.php on HEAD.

Basically I was going to move this patch out to another issue which is a refactoring patch for move all the functions into update.inc and this set this up as a patch to add the shell version.

But I have been having a lot of issues with session not saving which has been causing issues.

Dries’s picture

Status:Reviewed & tested by the community» Fixed

I committed this to CVS HEAD. Can we create follow-up patches for further clean-up?

If this needs a CHANGELOG.txt entry, feel free to suggest one.

dww’s picture

Title:Update drupal sites from shell.» Move code from update.php into includes/update.inc
Category:feature» task

This change is not end-user visible, so I see no reason for a CHANGELOG.txt entry.

Yes, other clean-up and reorganization can be in other issues. I think this one should be closed at this point. Giving it a more accurate title for posterity. If folks submit follow-up issues, please comment here with a link. Otherwise, there should be no other replies to this issue. Thanks.

int’s picture

@dww, 90% of the CHANGELOG.txt isn't for the end-user.. I think that, all functionality changes, should be described. (like before)
If not, the administrator don't know that functionality exists..

See it here.

dww’s picture

@int: There's no functionality change from this issue. We simply moved code from one file to another to facilitate functionality changes in the future.

dww’s picture

To clarify, by "end-user" I meant "anyone other than a code developer opening up update.php in their text editor".

int’s picture

OK, this issue have changed..

"Here is an update script which can be run from the shell to process the updates."

gordon’s picture

I have created a new issue #536150: Move more functions to update.inc in which I have moved a few more functions over to the new include/update.inc which make it a little more API'ish in that you have a little more control over how the update works.

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

moshe weitzman’s picture

Well done - we now have a bunch of refactor patches in, including #555762: Changes to batch API.

@gordon - any chance you can redo update.sh? we'll try to get into into core. if unsuccessful, i'll put the code straight into drush. let us know if you open a new issue for it.

gordon’s picture

I have it there and now i have everything in core that I require I should be able to put up a new version in the next couple of days.

Also because of these changes that I have contributed to Drupal 7 the only part left is the scripts/update.sh