Backup & migrate module comes with built-in support for Amazon Web Services Simple Storage Service (AWS S3).
Putting your backups entirely offsite is probably one of the best things you can do for yourself and your clients. Making' local' backups is OK for development, but really doesn't help for disaster recovery of live websites.
You must have an AWS account set up first! Once you have that, and can log in to the AWS 'console' OK, we can proceed.
Some of these things may change as Amazon updates its API or UI over time - this walkthrough is current for 2015-04, using a personal AWS account.
-
Install and enable backup_migrate.module
-
Configure Backup Migrate with a new destination.
-
Visit the settings page
admin/config/system/backup_migrate/destination
and "Add Destination", choosing Amazon S3 Bucket as the type. -
You will be told you have to download and unpack the required PHP connector library.
Do it any way you feel like, eg via the cli like this:cd sites/all/libraries/ # OLD stable version wget http://amazon-s3-php-class.googlecode.com/files/s3-php5-curl_0.4.0.tar.gz tar -xzf s3-php5-curl_0.4.0.tar.gz
If using PHP5.5+, you must instead use the updated version from github and be sure to unzip it into a folder named s3-php5-curl.. It's likely the new library is backwards-compatible also.
- You will be asked a set of parameters to use for connecting to S3, so we should set them up now.
-
Visit the settings page
-
Create your backups bucket on S3
- From the AWS console, Find the S3 section.
-
"Create Bucket" and name it something like
mycompany-websitebackups
. This slug will be used in many placed as a resource identifier. Make sure it's lowercase and simple. - For best results, use the US-Default region According to some unsubstantiated reports, as of 2015-04, the stable version of the PHP-S3 library does not support any other regions quite as well.
-
Create a backup user on IAM
You MUST NOT use your admin credentials to get your backup script to connect for you. Instead we must set up a limited access account.
- From the AWS console, find the Identity and Access Management (IAM) section.
-
Under "Users", "Create new Users". I called mine
backup_daemon
. - When it asks to "Generate an access key for each user", do so and be sure to download the credentials it gives you as you can't recover them later and have to regenerate if you lose it.
-
Grant the backup user access to the S3 Bucket
This starts to get tricky now. The most direct method I've found is:
- Edit your backup user. Under "Inline Policies" : "Create User Policy".
-
You can use the Policy Generator, though it's tricky to grok if you are unfamiliar with AWS. Instead, choose "Custom Policy" and paste the following:
Policy name:
can-store-website-backups
Policy Document:{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowUserToFindBucket", "Action": [ "s3:ListAllMyBuckets", "s3:ListBucket", "s3:GetBucketLocation", "s3:ListObjects" ], "Effect": "Allow", "Resource": "arn:aws:s3:::*" }, { "Sid": "AllowUserToStoreFiles", "Action": [ "s3:DeleteObject", "s3:GetBucketAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:ListAllMyBuckets", "s3:ListBucket", "s3:ListObjects", "s3:PutObject", "s3:PutObjectAcl" ], "Effect": "Allow", "Resource": "arn:aws:s3:::mycompany-websitebackups/*" } ] }
Replace mycompany-websitebackups with your bucket identifier
- Validate and save.
The above may not be a perfect list, but it's one that was found to work, without allowing too much escalation of privileges.
-
At this point, you should be able to return to your Drupal screen and fill in the details.
Destination name (your label, enter anything)
Host (defaults to s3.amazonaws.com) ... which works best if you are using region "US-standard" If using a different region, you may also need to change this hostname
S3 Bucket (Your variation of mycompany-websitebackups)
Subdirectory (Should probably be required and be unique. It does not need to already exist, and may be any depth)
Access Key ID (From the credentials.csv you downloaded for the backup-daemon)
Secret Access Key (From the credentials.csv you downloaded for the backup-daemon) -
Try accessing the S3 bucket from Drupal now.
In the 'Destinations' tab, try to "List Files". This will test your backup-daemons ability to connect to S3 and read. If there are basic access control problems, you should see the error listed now. If not, we are making progress.
-
Try making a backup manually.
Use the "Quick Backup" option and select your new destination to take a snapshot to.
Again, this will test your backup-daemons ability to write to the bucket. -
Check the 'list' of backups available in the S3 destination.
Note that the listing may be out of date, and has an extra link used to refresh it.
You should be set to go now, and can configure the backup schedules etc as normal. See the backup_migrate help docs or issue queue for details.
Troubleshooting
Error: "putObject() necessary data rewind wasn't possible"
...was the unhelpful message I got when trying to use a non-US region. Switching to a new bucket in the US-Default zone worked.
Further investigation showed that changing the S3 'Host' to the matching region (eg s3-ap-southeast-2.amazonaws.com) also seemed to solve it. However, the problem seems irregular and hard to reproduce.
It was tricky to debug, as 'read' would work, but 'write' would not, irrespective of permissions.
Gotcha: Note the extra permission in the Security Policy. Even allowing the user to do "everything" ("s3:*") in the designated bucket ( "arn:aws:s3:::mycompany-websitebackups/*") was not sufficient to give them access to it. The account also needed at least 's3:ListBucket' permission in the root context ("arn:aws:s3:::*") before it could work.
Requirements: cURL is required. If you get a WSOD on the config page, you may need to
sudo apt-get install php5-curl
Compatibility: If using PHP 5.5, you can't use the "old stable version" from the project listed in the help, you need to use the updated version of the s3-php library. It can be found on github at https://github.com/tpyo/amazon-s3-php-class .
Beware: As with all other backup_migrate destination folders, do not re-use the same folder for more than one scheduled job!. The rules used to calculate the number of files to keep just counts the number of files in the directory and then trims the old ones. So if you want to keep 20 weekly backups and 5 daily ones and use the same storage location for both then the rule that trims to 5 files will delete your other 20. And that's not what you want. Once it's working, you should probably visit the settings again and add a named subfolder to your bucket, eg 'scheduled/weekly'. AWS 'filepaths' are actually just identifiers which allow / in their name, so you shouldn't need to make a 'directory' before just using it.
Comments
Good post - also emphasizing DON'T use the account admin keys
The above guide is spot on - thanks Dan
Must reiterate that you shouldn't use active keys for the Amazon Account Admin user - create a new user and give them limited permissions as per the above notes. Unless absolutely needed the admin account shouldn't have keys assigned.
Do not forget to attach the new backup user to the policy
I forgot to do the following: Under Attached Entities -sections you need to attach the newly created user to the policy you just created.
Another glitch was that for some reason domain.com did not work for bucket name, I used instead domaincom.
AWS S3 policy
The sample policy showed previously doesn't work for me.
I generated this one with the AWS Policy Generator.
Getting S3 working in newer AWS Regions
After much investigation and some code fixing, I was able to setup a proper Drupal backup using S3.
There are two methods (that I know of) for getting S3 support. One is the s3-php5-curl library, which the instructions above describe how to get, and the other is the https://www.drupal.org/project/backup_migrate_s3 module (with a tiny installed base), which uses the more official though sizeable aws-sdk-php library from Amazon (it uses only aws-sdk-php version 2.x which works on PHP 5.3.3+, rather than the latest 3.x which requires PHP 5.5; latest 2.x as I write is version 2.8.31, see https://github.com/aws/aws-sdk-php/releases/tag/2.8.31).
Neither method actually works yet for most regions, with released software. They both end up using AWS Signature Version 2 authentication (V2). Only a few older S3 regions support this outdated protocol. All Amazon S3 regions now support AWS Signature Version 4 authentication (V4), and those deployed after January 2014 only support V4 (http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-req...). See also http://stackoverflow.com/questions/26533245/the-authorization-mechanism-... . Trying to authenticate with V2 on newer S3 regions leads to an error such as "InvalidRequest The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256." Also note that V4 requires specifying the region, and you get a different error if you try to use V4 (as explained in http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html) without specifying the region in which the S3 bucket is located.
Although some do get this working by using the "US Standard" region (now named "US East (N. Virginia)"), requiring the use of a specific region is really not a solution -- first, there are various reasons to want storage in (or away from) specific geopolitical regions; and second, relying on an outdated scheme that might go away at any time just seems like a disaster waiting to happen. Not something I want in a solid backup solution.
There appear to be fixes available for both S3 solutions, but none yet officially released. For backup_migrate_s3 there was a patch in https://www.drupal.org/node/2653612, and I added my own version for reasons described there (sorry my first time submitting a patch, and the patch name does not follow usual conventions). For s3-php5-curl, this looks promising (I haven't tried it; comments indicate some have had some success, though there are still bugs): https://github.com/tpyo/amazon-s3-php-class/pull/128 .
I can edit the page if that indeed seems appropriate, though it will be nicer once fixes get officially released.