Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Problem/Motivation
Current AWS S3 integration is handled by a separate third party library (sparsely documented is README.txt).
Instead, the project should make use of the AmazonS3 project which provides a file stream wrapper for AWS S3 and depends on AWS SDK for PHP which integrates Amazon's official php DSK with Drupal. This allows for one time configuration of amazon credentials and use of Amazon's official SDK.
boombatower have gone ahead and implemented this. However, no stream wrapper function that will return a file listing with stats (filesize, time, etc) is known. That would need to be called directly from SDK which he has not done yet … He simply does a scandir().
It should be noted neither of these projects are up to date, as they rely on the signature known as SigV2. This signature cannot be used with any AWS S3 region launched after 2013, and will be depreciated for new buckets created after June 24, 2020. For details, see SigV2 Deprecation.
They need updating before they shall be are usable.
(For a cleaner way of setting the bucket: #1190608: Support multiple S3 buckets.)
Proposed resolution
Short term: Initiate the process for Dealing with unsupported (abandoned) projects to either get the current maintainers to renew their engagement, or to find new maintainers willing to update to use the most recent AWS S3 SDK.
Long term: Get both projects up to date with use of the latest SDK. Then make use of them for B&M integration with AWS S3.
| Comment | File | Size | Author |
|---|---|---|---|
| #1 | 1714696-amazons3.patch | 7.07 KB | boombatower |
Comments
Comment #1
boombatower CreditAttribution: boombatower commentedComment #2
JordanMagnuson CreditAttribution: JordanMagnuson commentedI agree that this project should make use of Drupal's existing amazon sdk implementation (awssdk), to avoid redundancy.
I'm not sure that the higher level AmazonS3 module should be made a dependency though, unless the benefits are really striking: the fewer modules/dependencies, the better...
Comment #3
boombatower CreditAttribution: boombatower commentedIn this case the amazons3 module provides the file stream wrapper definition which allows for file_get_contents('s3://foo.txt'); ect. If we were to use the sdk directly then call would be more specific to s3 as in the current implementation. Either would work, but it seemed like the file stream wrapper was a much more elegant implementation and in fact could be turned into a generic "remote storage" plugin that simply used stream wrappers.
Select protocol: s3
path: backups/
and away we go.
Comment #4
boombatower CreditAttribution: boombatower commentedLooks like I need to fix the return value when the environment isn't setup. I changed that code near the end and it should return FALSE.
Comment #5
couturier CreditAttribution: couturier as a volunteer commentedIs this still an issue with the 7.x-3.x version? Many users have more recently reported success with AmazonS3 and newer versions of Backup and Migrate..
Comment #6
DamienMcKennaComment #8
fgmI'd be very curious to see how they can report success: S3 now requires signature version 4 (at least in non-US regions) and the code only supports version 2 at this time. Maybe they had success back when signature version 2 was still supported, but right now, I can't think of a reasonable way to make use of the package the module has been using. Looks like it would need a wrapper of the standard SDK to emulate that legacy library, so we could just as well use the official SDK, which be more reliably maintained in the longer term.
Comment #9
couturier CreditAttribution: couturier as a volunteer commented@fgm Yes, I believe the reported success was with earlier versions because those were older posts in the issues queue than now (2018).
Does this mean we should close this issue as outdated?
Comment #10
fgm@couturier: quite the opposite, actually, I think @boombatower's approach is the most reasonable way to approach talking to S3 in the longer term, since AWS is likely to keep the SDK current, unlike the one-man-project used by current versions of the module.
Comment #11
couturier CreditAttribution: couturier as a volunteer commented@fgm Do you have the resources to be able to help with this patch then? @boombatower hasn't been active with this issue in 6 years. See that the last submitted patch failed testing about a month ago.
Comment #12
DamienMcKennaSeeing as S3 just added SFTP support, should we maybe merge in backup_migrate_sftp and recommend people use it instead?
Comment #13
couturier CreditAttribution: couturier as a volunteer commented@DamienMcKenna good idea. A significant number of people are having trouble with S3 backups, so anything to help them is welcome.
Comment #14
DamienMcKennaComment #15
JordanMagnuson CreditAttribution: JordanMagnuson commentedSide note, in case it helps anyone else: Amazon S3 backup has still been working for me with the current version of the module as it stands, but only if my AWS secret key and access key do not contain any special characters (access keys with "+" or "/" or other special characters will not validate with the ancient version of the library that is in use).
Comment #16
couturier CreditAttribution: couturier as a volunteer commented@JordanMagnuson that is a very important insight. Maybe we should update documentation to warn S3 users about this? This could be the secret to many of the problems we have seen.
Comment #17
DamienMcKennaYes, that is a great point, at the very least it might be worth updating the documentation and UX to note the problem.
Comment #18
couturier CreditAttribution: couturier as a volunteer commentedSince so much has changed with Amazon S3 since this issue was opened and since Comment #15 seems to be a fix for now -- which isn't something affected by a patch but rather by better documentation -- I suggest we change this to a documentation update issue. This is filed under 7.x, but I wonder if it is also valid for our 8.x version of the module. Possible wording:
Comment #19
couturier CreditAttribution: couturier as a volunteer commentedComment #20
gisleThe issue summary is request B&M to make use of the AmazonS3 project which provides a file stream wrapper for AWS S3 and depends on AWS SDK for PHP.
I agree that that would be sensible, as we don't want AWS S3 integration spread over several projects duplicating effort. However, both these projects are seriously out of date as they do not support V4 of the signature required by most AWS S3 destinations.
The maintainers are mostly still active on Drupal (with the possible exception of boombatower), but all of them seem to have lost interest in maintaining these projects. IMHO, both projects now look abandoned and need to go through the process for Dealing with unsupported (abandoned) projects to either get the current maintainers to renew their engagement, or to find new maintainers willing to update to use the most recent AWS S3 SDK, before it shall be feasible to use these project with B&M.
Updated summary with a proposed solution.
Please comment to indicate whether you think this is a way forward, or that something else should be done.