Backup and Migrate - with AWS S3

Backup & migrate module comes with built-in support for Amazon Web Services Simple Storage Service (AWS S3).

Putting your backups entirely offsite is probably one of the best things you can do for yourself and your clients. Making' local' backups is OK for development, but really doesn't help for disaster recovery of live websites.

You must have an AWS account set up first! Once you have that, and can log in to the AWS 'console' OK, we can proceed.

Some of these things may change as Amazon updates its API or UI over time - this walkthrough is current for 2015-04, using a personal AWS account.

• Install and enable backup_migrate.module

• Configure Backup Migrate with a new destination.

  • Visit the settings page admin/config/system/backup_migrate/destination and "Add Destination", choosing Amazon S3 Bucket as the type.
  • You will be told you have to download and unpack the required PHP connector library.
    Do it any way you feel like, eg via the cli like this:

      cd sites/all/libraries/
      # OLD stable version
       wget http://amazon-s3-php-class.googlecode.com/files/s3-php5-curl_0.4.0.tar.gz
       tar -xzf s3-php5-curl_0.4.0.tar.gz 
    

    If using PHP5.5+, you must instead use the updated version from github and be sure to unzip it into a folder named s3-php5-curl.. It's likely the new library is backwards-compatible also.

  • You will be asked a set of parameters to use for connecting to S3, so we should set them up now.

• Create your backups bucket on S3

  • From the AWS console, Find the S3 section.
  • "Create Bucket" and name it something like mycompany-websitebackups. This slug will be used in many placed as a resource identifier. Make sure it's lowercase and simple.
  • For best results, use the US-Default region According to some unsubstantiated reports, as of 2015-04, the stable version of the PHP-S3 library does not support any other regions quite as well.

• Create a backup user on IAM

  You MUST NOT use your admin credentials to get your backup script to connect for you. Instead we must set up a limited access account.

  • From the AWS console, find the Identity and Access Management (IAM) section.
  • Under "Users", "Create new Users". I called mine backup_daemon.
  • When it asks to "Generate an access key for each user", do so and be sure to download the credentials it gives you as you can't recover them later and have to regenerate if you lose it.

• Grant the backup user access to the S3 Bucket

  This starts to get tricky now. The most direct method I've found is:

  • Edit your backup user. Under "Inline Policies" : "Create User Policy".
  • You can use the Policy Generator, though it's tricky to grok if you are unfamiliar with AWS. Instead, choose "Custom Policy" and paste the following:

    Policy name: can-store-website-backups
    Policy Document:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowUserToFindBucket",
                "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:ListBucket",
                    "s3:GetBucketLocation",
                    "s3:ListObjects"
                ],
                "Effect": "Allow",
                "Resource": "arn:aws:s3:::*"
            },
            {
                "Sid": "AllowUserToStoreFiles",
                "Action": [
                    "s3:DeleteObject",
                    "s3:GetBucketAcl",
                    "s3:GetObject",
                    "s3:GetObjectAcl",
                    "s3:ListAllMyBuckets",
                    "s3:ListBucket",
                    "s3:ListObjects",
                    "s3:PutObject",
                    "s3:PutObjectAcl"
                ],
                "Effect": "Allow",
                "Resource": "arn:aws:s3:::mycompany-websitebackups/*"
            }
        ]
    }
    

    Replace mycompany-websitebackups with your bucket identifier

  • Validate and save.

  The above may not be a perfect list, but it's one that was found to work, without allowing too much escalation of privileges.

• At this point, you should be able to return to your Drupal screen and fill in the details.

  Destination name (your label, enter anything)
  Host (defaults to s3.amazonaws.com) ... which works best if you are using region "US-standard" If using a different region, you may also need to change this hostname
  S3 Bucket (Your variation of mycompany-websitebackups)
  Subdirectory (Should probably be required and be unique. It does not need to already exist, and may be any depth)
  Access Key ID (From the credentials.csv you downloaded for the backup-daemon)
  Secret Access Key (From the credentials.csv you downloaded for the backup-daemon)

• Try accessing the S3 bucket from Drupal now.

  In the 'Destinations' tab, try to "List Files". This will test your backup-daemons ability to connect to S3 and read. If there are basic access control problems, you should see the error listed now. If not, we are making progress.

• Try making a backup manually.
  Use the "Quick Backup" option and select your new destination to take a snapshot to.
  Again, this will test your backup-daemons ability to write to the bucket.

• Check the 'list' of backups available in the S3 destination.
  Note that the listing may be out of date, and has an extra link used to refresh it.

You should be set to go now, and can configure the backup schedules etc as normal. See the backup_migrate help docs or issue queue for details.

Troubleshooting

Error: "putObject() necessary data rewind wasn't possible"
...was the unhelpful message I got when trying to use a non-US region. Switching to a new bucket in the US-Default zone worked.
Further investigation showed that changing the S3 'Host' to the matching region (eg s3-ap-southeast-2.amazonaws.com) also seemed to solve it. However, the problem seems irregular and hard to reproduce.
It was tricky to debug, as 'read' would work, but 'write' would not, irrespective of permissions.

Gotcha: Note the extra permission in the Security Policy. Even allowing the user to do "everything" ("s3:*") in the designated bucket ( "arn:aws:s3:::mycompany-websitebackups/*") was not sufficient to give them access to it. The account also needed at least 's3:ListBucket' permission in the root context ("arn:aws:s3:::*") before it could work.

Requirements: cURL is required. If you get a WSOD on the config page, you may need to
sudo apt-get install php5-curl

Compatibility: If using PHP 5.5, you can't use the "old stable version" from the project listed in the help, you need to use the updated version of the s3-php library. It can be found on github at https://github.com/tpyo/amazon-s3-php-class .

Beware: As with all other backup_migrate destination folders, do not re-use the same folder for more than one scheduled job!. The rules used to calculate the number of files to keep just counts the number of files in the directory and then trims the old ones. So if you want to keep 20 weekly backups and 5 daily ones and use the same storage location for both then the rule that trims to 5 files will delete your other 20. And that's not what you want. Once it's working, you should probably visit the settings again and add a named subfolder to your bucket, eg 'scheduled/weekly'. AWS 'filepaths' are actually just identifiers which allow / in their name, so you shouldn't need to make a 'directory' before just using it.