Add support to allow using Amazon AWS S3 as a backup destination.

Note: if this is possible using existing modules, provide documentation on how to achieve this.

CommentFileSizeAuthor
#17 interdiff_15-17.txt873 bytessnehalgaikwad
#17 backup_migrate-aws_s3_backup_destination-3027843-17.patch33.68 KBsnehalgaikwad
#16 drupal8-backupS3-upload-16.png55.66 KBWillenLeal
#15 interdiff_11-15.txt2.58 KBsnehalgaikwad
#15 backup_migrate-aws_s3_backup_destination-3027843-15.patch33.66 KBsnehalgaikwad
#11 interdiff_8-11.txt4.26 KBsnehalgaikwad
#11 backup_migrate-aws_s3_backup_destination-3027843-11.patch33.59 KBsnehalgaikwad
#8 backup_migrate-aws_s3_backup_destination-3027843-8.patch33.24 KBsnehalgaikwad
#6 backup_migrate-aws_s3_backup_destination-3027843-6.patch16.96 KBsnehalgaikwad

Comments

DamienMcKenna created an issue. See original summary.

gisle’s picture

gisle
he/him
Primary language Norwegian Bokmål
Location Norway
commented
Related issues: +#2845676: New backup destination: NodeSquirrel (D8)

I second this!

Since May 2018, I've used AWS S3 as for offline backups along with Backup and Migrate.

My setup is external to Backup and Migrate and do not rely on other Drupal modules. Basically I do this:

  1. Configure Backup and Migrate to scheduled backups to the local private filesystem
  2. Have a Unix cron-job that run a script built upon s3cmd to sync the local private filesystem with AWS S3 just after Backup and Migrate have created a scheduled backup.

This works well enough for me to rely on it for all the production sites I am in charge of. However, I would welcome an option to set AWS S3 as a backup destination as part of Backup and Migrate.

alangallery’s picture

alangallery commented

We do this too but use the aws cli, https://aws.amazon.com/cli/, for syncing a folder to S3.

Alternatively there is rclone, https://rclone.org/, which apart from supporting many cloud storage providers, also supports FTP/SFTP.
I don't use this for our production but at home and it works well.

snehalgaikwad’s picture

snehalgaikwad commented
Assigned: Unassigned » snehalgaikwad
rodmarasi’s picture

rodmarasi commented

hi snehalgaikwad, any news on the patch for this feature?

snehalgaikwad’s picture

snehalgaikwad commented
Assigned: snehalgaikwad » Unassigned
Status: Active » Needs review
StatusFileSize
new backup_migrate-aws_s3_backup_destination-3027843-6.patch16.96 KB

Added new destination Amazon AWS S3 for backup.

snehalgaikwad’s picture

snehalgaikwad commented
Assigned: Unassigned » snehalgaikwad

Updating patch with few changes.

snehalgaikwad’s picture

snehalgaikwad commented
Assigned: snehalgaikwad » Unassigned
StatusFileSize
new backup_migrate-aws_s3_backup_destination-3027843-8.patch33.24 KB
muaz7731’s picture

muaz7731
Location Malaysia
commented

Hi,

I've tested out the patch

1) Even though I've added $settings[backup_migrate_aws_access_key] and $settings[backup_migrate_aws_key_id] in my settings.php, the notification for "Please store access key id an secret key id" does not disappeared.

2) When trying out quick backup to my s3 the error appeared as below:

Error executing "PutObject" on "https://[mybucket].s3.ap-southeast-1.amazonaws.com/backup-2019-08-08T08-59-22.mysql.gz"; AWS HTTP error: Client error: `PUT https://[mybucket].s3.ap-southeast-1.amazonaws.com/backup-2019-08-08T08-59-22.mysql.gz` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> AuthorizationHeaderMalformedThe authorization header (truncated...) AuthorizationHeaderMalformed (client): The authorization header is malformed; the Credential is mal-formed; expecting "/YYYYMMDD/REGION/SERVICE/aws4_request". - <?xml version="1.0" encoding="UTF-8"?> AuthorizationHeaderMalformedThe authorization header is malformed; the Credential is mal-formed; expecting "<YOUR-AKID>/YYYYMMDD/REGION/SERVICE/aws4_request".2DD8E33313675943SLNdBHyovR2mDwwII5JnVbgVg4/P01Wqe4SvsekDDnzsl1Di1X475y3f1GMfBuMeTB8tyAGxE9g=

edit:

now I am using the latest development for backup and migrate module updated 13 Apr 2019 at 01:38 UTC

I've now received the following error when quick backup to amazon s3

Error executing "PutObject" on "https://[mybucket].s3.ap-souteast-1.amazonaws.com/backup-2019-08-09T03-45-48.mysql.gz"; AWS HTTP error: cURL error 6: (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

the notification for "Please store access key id an secret key id" still does not disappeared as stated before.

muaz7731’s picture

muaz7731
Location Malaysia
commented
Status: Needs review » Needs work
snehalgaikwad’s picture

snehalgaikwad commented
Status: Needs work » Needs review
StatusFileSize
new backup_migrate-aws_s3_backup_destination-3027843-11.patch33.59 KB
new interdiff_8-11.txt4.26 KB

Here is updated patch with few more changes.

snehalgaikwad’s picture

snehalgaikwad commented

Hi @muaz91,
1. I have uploaded a new patch which will fix issue of notification for "Please store access key id an secret key id".
2. For the second point, I guess it is issue with Bucket Policy for the S3 bucket you provided. Will you please confirm once if access key you have provided have permission to PutObject in the bucket?
Thanks.

muaz7731’s picture

muaz7731
Location Malaysia
commented

Hi @snehalgaikwad,

The patch works, however there are problems that I have found:

1. The region field is not required, you can save without putting any region. When leave blank and try to backup, will get notification as below:

Please fill all mandatory fields to create S3 client
Could not upload to S3

2. The main problem is that, the $settings['backup_migrate_aws_key_id'] and $settings['backup_migrate_aws_access_key'] is getting the wrong information. This means that when i put my access key in $settings['backup_migrate_aws_access_key'] and my secret key in $settings['backup_migrate_aws_key_id'], I will get error as below:

Error executing "PutObject" on "https://[mybucket].s3.ap-southeast-1.amazonaws.com/backup-2019-08-15T07-37-17.mysql.gz"; AWS HTTP error: Client error: `PUT https://[mybucket].s3.ap-southeast-1.amazonaws.com/backup-2019-08-15T07-37-17.mysql.gz` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> AuthorizationHeaderMalformedThe authorization header (truncated...) AuthorizationHeaderMalformed (client): The authorization header is malformed; the Credential is mal-formed; expecting "/YYYYMMDD/REGION/SERVICE/aws4_request". - <?xml version="1.0" encoding="UTF-8"?> AuthorizationHeaderMalformedThe authorization header is malformed; the Credential is mal-formed; expecting "<YOUR-AKID>/YYYYMMDD/REGION/SERVICE/aws4_request".39219DAC9BC3E6E5XRQ+kSGXqqRevmSrZjKNidxQKhhfWX++M7yw20cnvCxXR43Ft5hLl8hWOD+tZgYv4ZdaYUXySQw=

However, if I put:

$settings['backup_migrate_aws_key_id'] = 'my-access-key';
$settings['backup_migrate_aws_access_key'] = 'my-secret-key';

the backup process works. the information put in is in the wrong fields. Thank you.

muaz7731’s picture

muaz7731
Location Malaysia
commented
Status: Needs review » Needs work
snehalgaikwad’s picture

snehalgaikwad commented
Status: Needs work » Needs review
StatusFileSize
new backup_migrate-aws_s3_backup_destination-3027843-15.patch33.66 KB
new interdiff_11-15.txt2.58 KB

Made Region field required and changed variable names of secret key and access key to avoid confusion. Here is updated patch.

WillenLeal’s picture

WillenLeal commented
StatusFileSize
new drupal8-backupS3-upload-16.png55.66 KB

Hello @snehalgaikwad,
I tested your patch it works perfectly, the region is now required and the access_key and secret access_key are better labeled. The last item on my S3 is the default Drupal database backup

snehalgaikwad’s picture

snehalgaikwad commented
StatusFileSize
new backup_migrate-aws_s3_backup_destination-3027843-17.patch33.68 KB
new interdiff_15-17.txt873 bytes

Updated notice for storing secret and access key in settings.php.

muaz7731’s picture

muaz7731
Location Malaysia
commented
Status: Needs review » Reviewed & tested by the community

Tested the patch on #17. It works flawlessly

dinesh18’s picture

dinesh18 commented

#17 patch looks good to me. +1 to RTBC

damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Status: Reviewed & tested by the community » Needs work

First off, I really appreciate the great work put into this, thank you all.

That said, it need a bit more work.

+ To upload backup on AWS S3 :
+
+ `1. Add S3 destination : /admin/config/development/backup_migrate/settings/destination. Select Amazon S3`

These lines should follow markdown syntax to create a numbered list rather than code syntax.

+  "require": {
+    "aws/aws-sdk-php": "^3.100"

This module cannot add a dependency on an AWS library. If this dependency needs to be added in order for S3 to work, it might be worth splitting the S3 functionality into a (bundled) submodule that's only available when enabled.

+class S3Destination extends DestinationBase implements

Could this class please be renamed to "AmazonS3Destination" rather than just "S3Destination"?

diff --git a/lib/backup_migrate_core/src/Main/BackupMigrate.php b/lib/backup_migrate_core/src/Main/BackupMigrate.php
index d5b2171..41dabdf 100644
--- a/lib/backup_migrate_core/src/Main/BackupMigrate.php
+++ b/lib/backup_migrate_core/src/Main/BackupMigrate.php
@@ -1,35 +1,31 @@
 <?php
-
 namespace BackupMigrate\Core\Main;
-

There are a bunch of line removals that need to be reverted.

+  const BACKUP_FILE = '/tmp/awss3_backup.gz';

The BackupMigrate\Core\Main\BackupMigrate class should not have a constant that assumes both a specific path and specific service, this should be overridden in a subclass if needed.

diff --git a/src/Controller/BackupController.php b/src/Controller/BackupController.php
index 5d4de65..4a9922c 100644
--- a/src/Controller/BackupController.php
+++ b/src/Controller/BackupController.php
@@ -1,56 +1,52 @@
 <?php
-
 namespace Drupal\backup_migrate\Controller;
-

There are a bunch of line removals that need to be reverted.

+      if ($entity->type == 'S3') {

+    if ($backup_migrate_destination->type == 'S3') {

The BackupController class should not have special handling for S3 hardcoded into it, instead the APIs need to be expanded so that the S3 subclass can handle the extra use cases.

+      return $this->listDestinationBackups($destination,
+          $backup_migrate_destination->id());

+        [
+            'data' => $this->t('Name'),
+            'class' => [RESPONSIVE_PRIORITY_MEDIUM],
+            'field' => 'name',
+        ],

This BackupController class also has a bunch of indentation changes that were not necessary.

+/**
+ * @file
+ * File to define schema for AWS S3.
+ */

Namespaced class files don't need a @file comment.

hussainweb’s picture

hussainweb
Primary language English
commented

This module cannot add a dependency on an AWS library. If this dependency needs to be added in order for S3 to work, it might be worth splitting the S3 functionality into a (bundled) submodule that's only available when enabled.

I agree it is not a good idea to require on the main module level; however, I'm not sure if the submodule approach would work. I don't know if a composer.json inside a submodule is read to build the dependencies. I broadly see two approaches:

  1. Have the functionality in a submodule and only add the PHP dependency in `suggests` section. The functionality is better off in a submodule. Otherwise, things like extending a class from the PHP package might not work as expected. The module shouldn't be able to install without the dependency.
  2. Put the entire thing in Backup and Migrate S3 module. Over there, it might make sense to put the dependency in the root composer.json itself.

Thoughts?

snehalgaikwad’s picture

snehalgaikwad commented

Thanks @DamienMcKenna and @hussainweb for your suggestions. I'll make appropriate changes in code and will put it in Backup and Migrate S3 as a D8 porting.

erwindeclerck’s picture

erwindeclerck commented

Hi

I manage to upload backup files to my S3 bucket with the backup_migrate_aws_s3 module but only to the root folder of my bucket.
When I use bucketname/subfolder I get error because the slash in the folder is rewritten to bucketname%2Fsubfolder.
Slash is url encoded...

Any suggestions how I could solve this problem?

Error executing "PutObject" on "https://s3.eu-west-3.amazonaws.com/bucket.XXXXX.be%2Fikbenpro/backup-202..."; AWS HTTP error: Client error: `PUT https://s3.eu-west-3.amazonaws.com/bucket.XXXXX.be%2Fikbenpro/backup-202...` resulted in a `403 Forbidden` response: .....

Kind regards.

alexgreyhead’s picture

alexgreyhead commented

Hi @ErwinDeClerck (and anyone else looking to save backups into a sub-folder of their S3 bucket) - I've filed issue #3189043 with a patch which may help you?

:)

Best wishes,

Alex

damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Version: 8.x-4.x-dev » 5.0.x-dev

Moving 8.x-4.x issues to 5.0.x because the older branch is no longer supported.

yago elias’s picture

yago elias commented

Hi folks, i have a question that might sound crazy,

Why are we need create this solution for B&M and not just configure https://www.drupal.org/project/s3fs to work as your private system (or public)?

damienmckenna’s picture

damienmckenna
Location TN, USA
commented

I think it'd be worth just going with Backup Migrate Flysystem as the official solution for "not on this server" - I just tested it and it was able to create backups correctly.