The current implementation of updatecode drupal should work just fine with svn or bzr version control even if the core directory is not moved out of the way. The 'backup' version control could also be brought into line with code such as this:

+++ commands/pm/version_control/backup.inc	2 Apr 2010 00:23:12 -0000
@@ -10,7 +10,7 @@ class drush_pm_version_control_backup im
   /**
    * Implementation of pre_update().
    */
-  public function pre_update(&$project) {
+  public function pre_update(&$project, $items_to_test = array()) {
     $drupal_root = drush_get_context('DRUSH_DRUPAL_ROOT');
 
     // Save the date to be used in the backup directory's path name.
@@ -19,15 +19,34 @@ class drush_pm_version_control_backup im
     $backup_dir = drush_get_option('backup-dir', $drupal_root  . '/backup');
     $backup_dir = rtrim($backup_dir, '/');
     @drush_op('mkdir', $backup_dir, 0777);
-    $backup_dir .= '/modules';
-    @drush_op('mkdir', $backup_dir, 0777);
-    $backup_dir .= "/$date";
-    @drush_op('mkdir', $backup_dir, 0777);
-    $backup_target = $backup_dir . '/'. $project['name'];
-    // Save for rollback or notifications.
-    $project['backup_target'] = $backup_target;
-    if (!drush_op('rename', $project['full_project_path'], $backup_target)) {
-      return drush_set_error('DRUSH_PM_BACKUP_FAILED', dt('Failed to backup project directory !project to !backup_target', array('!project' => $project['full_project_path'], '!backup_target' => $backup_target)));
+    
+    // Special handling for updating core: move only some items to backup
+    if ($project['name'] == 'drupal') {
+      $backup_dir .= '/core';
+      @drush_op('mkdir', $backup_dir, 0777);
+      $backup_dir .= "/$date";
+      @drush_op('mkdir', $backup_dir, 0777);
+      $backup_target = $backup_dir . '/'. $project['name'];
+      @drush_op('mkdir', $backup_target, 0777);
+      // Save for rollback or notifications.
+      $project['backup_target'] = $backup_target;
+      foreach ($items_to_test as $item) {
+        if (!drush_op('rename', $project['full_project_path'] . '/' . $item, $backup_target)) {
+          return drush_set_error('DRUSH_PM_BACKUP_FAILED', dt('Failed to backup project directory !project to !backup_target', array('!project' => $project['full_project_path'], '!backup_target' => $backup_target)));
+        }
+      }
+    }
+    else {
+      $backup_dir .= '/modules';
+      @drush_op('mkdir', $backup_dir, 0777);
+      $backup_dir .= "/$date";
+      @drush_op('mkdir', $backup_dir, 0777);
+      $backup_target = $backup_dir . '/'. $project['name'];
+      // Save for rollback or notifications.
+      $project['backup_target'] = $backup_target;
+      if (!drush_op('rename', $project['full_project_path'], $backup_target)) {
+        return drush_set_error('DRUSH_PM_BACKUP_FAILED', dt('Failed to backup project directory !project to !backup_target', array('!project' => $project['full_project_path'], '!backup_target' => $backup_target)));
+      }
     }
     return TRUE;
   }
@@ -36,12 +55,24 @@ class drush_pm_version_control_backup im
    * Implementation of rollback().
    */
   public function rollback($project) {
-    if (drush_op('rename', $project['backup_target'], $project['full_project_path']) && is_dir($project['full_project_path'])) {
-      return drush_log(dt("Backups were restored successfully."), 'ok');
+    if (isset($project['backup_target'])) {
+      if ($project['name'] == 'drupal') {
+        $items_to_roll_back = drush_scan_directory($project['backup_target'], '/.*/', array('.', '..'), 0, FALSE, 'filename', 0, 0, TRUE);
+        foreach ($items_to_roll_back as $item) {
+          drush_op('rename', $project['backup_target'] . '/' . $item, $project['full_project_path']);
+        }
+        if (drush_op('rmdir', $project['backup_target'])) {
+          return drush_log(dt("Backups were restored successfully."), 'ok');
+        }
+      }
+      elseif (drush_op('rename', $project['backup_target'], $project['full_project_path']) && is_dir($project['full_project_path'])) {
+        return drush_log(dt("Backups were restored successfully."), 'ok');
+      }
+      return drush_set_error('DRUSH_PM_BACKUP_ROLLBACK_FAILED', dt('Could not restore backup and rollback from failed upgrade. You will need to resolve manually.'));
     }
-    return drush_set_error('DRUSH_PM_BACKUP_ROLLBACK_FAILED', dt('Could not restore backup and rollback from failed upgrade. You will need to resolve manually.'));
+    return TRUE;
   }

n.b. untested.

I tend to think that the current implementation works fairly well, and further improvements could wait until we're ready to refactor the pm code to fix this 'for real'. If the general consensus is that it would be an important improvement to avoid the move-out-of-the-way hack with a workaround similar to the above, then I could go ahead and see if I can make that work. Probably not too hard, but I also think there are bigger fish to fry right now...

Comments

greg.1.anderson’s picture

I was just thinking about a couple of things.

First: If we did change pm-updatecode to download to a temporary directory, then how would the backup code back up core in that scenario? I suspect that the answer is the same as above.

Second: What is the best way to handle modified and added files when under vcs control? I propose the following workflow:

1. Start with a clean version of Drupal. Check it in and tag it.
2. Modify / add files are required. Commit.
3. Prior to upgrade, roll back to the tag in (1)
4. Upgrade. (Requires modification to svn.inc to allow updates of non-up-to-date projects)
5. Check in and tag the result as a branch
6. Do an svn merge between (5) and (2)
7. Commit

For the next upgrade, you simply start again at (3), except that you roll back to the branch at (5).

If both of my suppositions are correct, then we could simply declare that drush does not assist with modified-file management except when using svn version control (bzr should work too). In that case, I don't think that pm-updatecode needs rewriting at all -- just some incremental modification.

If I've missed anything, please propose a better alternative.

Owen Barton’s picture

There is an extra step in this that I am not understanding. Here is what I am thinking:

For backup based systems:
1. Copy existing project to backup directory
2. Download clean copy of target project version into ~/.drush/vendor/projectname/2.3
3. rsync from the vendor directory using --delete and --exclude (we could dry-run and confirm first - obviously this will clobber any local changes the user has not specifically excluded, but it will handle deletions correctly and eventually this could be replaced by a directory merge tool).

For VCS based systems I think using a standard vendor drop pattern (described in e.g. http://svnbook.red-bean.com/en/1.1/ch07s05.html) is going to be most reliable (i.e. preventing as many conflicts as possible) and understandable for users:
1. Perform preflight as usual
2. Check if current version exists in repos at /vendor/projectname/2.2, if not download it to ~/.drush/vendor/projectname/2.2, load it to /vendor/projectname/current and tag as /vendor/projectname/2.2
3. Repeat step 2 the same for the target version (e.g. 2.3), if needed
4. svn merge /vendor/projectname/2.2 /vendor/projectname/2.3 /the/project/directory
5. Check output for conflicts, if none then commit, if so then tell user they need to resolve and commit

Owen Barton’s picture

Note that I think we only need to consider the backup based approach for this ticket (the VCS stuff is probably >3), but it seems to me that it should work identically for both core and contrib (we would just need a specific default set of excludes when the project is "drupal").

greg.1.anderson’s picture

For VCS based systems:

Yes, the vendor drop pattern is exactly what I was thinking of, except optimized to avoid the funky rollback steps. Using a separate vendor branch (obvious, really -- those svn guys are smart), it is not necessary to modify svn.inc at all. In fact, users can follow your recipe above by hand today and successfully upgrade core. The process would start with drush dl drupal --destination=/vendor/ --drupal-project-rename and would continue with the user calling svn merge directly. Post drush-3, we could do those steps automatically as you suggested.

Regarding your comments on unifying core and contrib upgrades, yes, I agree that my code in #0 could be adjusted slightly to work the same way for both if there was a way to specify the excludes for a project. My move-then-download-over is, I think, equivalent to your copy-and-download-elsewhere-then-rsync-with-delete. Maybe rsync is better for rollbacks; I don't feel too strongly one way or the other.

So, it all sounds pretty good and overall I agree with you on technique, but I don't know that I have time to get even that much working in time to include it in drush-3.0 stable and still get that out before DrupalCon. If you have time to throw any code at this problem, it would be peachy. I'll help you test, and I'll certainly put together a patch myself if I have time.

Maybe we go to DrupalCon with 3.0-rc2; I dunno. I think I'd favor 3.0-stable for DrupalCon, and fix this in 3.1 soon after if we don't manage to do it before.

greg.1.anderson’s picture

See also #419520: drush pm-update failed using --backup-dir; pm-updatecode seems to have trouble if --backup-dir is on a different volume than the project being updated. Perhaps using rsync both to back up and to copy-over (with --delete) the downloaded update would be better all around.

mariomaric’s picture

Subscribing.

greg.1.anderson’s picture

moshe weitzman’s picture

As mentioned elsewhere, using rsync for basic rename is likely a bad idea for windows. or at least we need a fallback for windows.

I am still expecting 3.0 full release before drupalcon.

greg.1.anderson’s picture

I'll try to get this in (w/out rsync), but drush-3.0 stable should fly without it if I don't have time. I'm out of town on a business trip Sunday - Wednesday. :(

greg.1.anderson’s picture

Status: Active » Postponed

Wait, I was really counting on using the capabilities of rsync for this issue. I've been thrown for a loop. Let's hold off on this until post-3.0.

moshe weitzman’s picture

Title: Update core without moving core files out of the way? » Update core without moving core files out of the way.
Component: Code » PM (dl, en, up ...)
Category: bug » feature
Priority: Normal » Major
Status: Postponed » Active

I'm hip deep into updatecode and VCS engines and this looks pretty appealing. Lets see what we can do for Drush5. Also, I'm not opposed to requiring rsync these days.

moshe weitzman’s picture

I have a feeling that this problem could go away once git.drupal.org gets up and running next month. Then we can mandate that folks use a checkout from git.drupal.org and updating core will be more like a merge/rebase. sdboyer hints at this at http://drupal.org/node/797190#comment-2960916.

We can make a case to drop all version control engines except git in Drush5. CVS package handler can die as well.

greg.1.anderson’s picture

Assigned: greg.1.anderson » Unassigned

That all sounds good. I'm going to release my exclusive claim on this issue, though. :)

betz’s picture

subscribing

DarrellDuane’s picture

+1

greg.1.anderson’s picture

Regarding #12, we should not require that Drupal core be downloaded via git, because some people will grab Drupal and unpack it by hand before they ever use drush, and they should still be supported in pm-update.

What we could do, though, is generate a patch file via git diff, and then apply it over the existing copy of Drupal core. Maybe we use git checkout instead if core was checked out from git, although if you do this, you would have to be careful to check out from the right upstream source (e.g. if using dog; and hopefully no one decides to rename colab to origin, as that would then make the source to pull from ambiguous!)

moshe weitzman’s picture

Version: » 8.x-6.x-dev

Still not satisfactorily resolved, IMO. I'd like to see a solution here for Drush7

moshe weitzman’s picture

Status: Active » Closed (duplicate)