Here's the relevant code from platform/backup.provision.inc :

  if (substr($backup_file, -2) == 'gz') {
    // same as above: some do not support -z
    $command = 'tar cpf - . | gzip -c > %s';
  } else {
    $command = 'tar cpf %s .';
  }
  $result = drush_shell_exec($command,  $backup_file);

...

  if (!$result && !drush_get_option('force', false)) {
    drush_set_error('PROVISION_BACKUP_FAILED', dt("Could not back up sites directory for drupal"));
  }

(The first bit of code was previously untested without gzip, because there was no option not to gzip. See #2169025 for details.)

There is a difference in behavior between gzipped and non-gzipped backups, in that creating a .tar backup will fail the task when any file in the site's directory can't be read by Aegir. It fails because tar outputs (permission denied) errors (and returns an error code iirc).

Whereas as far as I can tell, gzip simply ignores those errors (or they don't get piped in the first place? not sure), and so doesn't relay them to in the result of drush_shell_exec, silently suppressing them. So we can end up just "losing" files Aegir can't access (created by www-data).

I don't really know how this should be fixed, apart from thinking that piping into gzip is meh, but just a bit higher the code says " // we need to do this because some retarded implementations of tar (e.g. SunOS) don't support -C" and " // same as above: some do not support -z". (Is anyone testing Aegir on SunOS?)

All I know is that ideally the behavior should be the same with and without gzipping, and we should probably test for file permissions somehow so the errors are not suppressed silently.

Feedback/ideas welcome.

Comments

gboudrias’s picture

Version: 6.x-2.x-dev » 7.x-3.x-dev

The code's the same in 7.3, so we have the same issue. Maybe we can fix this in 7.3 and backport it.

ergonlogic’s picture

Priority: Normal » Critical

With the recent work to fix broken mysqldumps in backups, it'd be nice to know we aren't suppressing errors later in the process.

  • helmo committed ceaa53c on 7.x-3.x
    Issue #2377819: Gzipping backups suppresses file permissions errors
    
helmo’s picture

Status: Active » Fixed

We're removing the pipe construct for now, just to be safer.

Not sure if it's still a problem on SunOS, but lacking any test opportunity ....

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

omega8cc’s picture

Status: Closed (fixed) » Needs work

This needs more work, because the fix introduced here makes the check just too aggressive and causes task to fail also when the permissions are related to files, which can be safely "lost" (like tmp, unintentionally uploaded .git, leftovers from .sass-cache etc.) -- although it doesn't happen with this patch reverted anyway, so in the effect this fix generates far more problems with failed tasks than it solves.

gboudrias’s picture

Priority: Critical » Major

If there was a way to set it to a simple warning, I think that would be ideal. (Obviously I didn't find a simple way to do this when we wrote the patch.)

I don't think it should be reverted outright, as I don't think files should be "lost" silently. As far as I can tell there's no way to know if the lost files are important or not before the migration, which could be catastrophic or lead to data loss over time. Please correct me if I'm wrong.

omega8cc’s picture

I agree that it could be dangerous on non-BOA Aegir systems. BOA runs permissions self-repair nightly on all sites, so practically always this patch causes false alarms related to tmp files or leftovers, because it never can affect anything older than the last 1-24 hours max, plus, BOA takes care to set correct chmod on everything uploaded via SFTP/FTPS, so the potential problem could be related only to files created by the web server (which runs under separate user) or extracted from archive on command line, if these files are not group writable.

ergonlogic’s picture

Status: Needs work » Closed (fixed)

This issue is fixed. If it has uncovered previously unknown bugs, let's address those in separate issues. There are already a number of them related to file permissions.