S3 File Sync

Module Description

The S3 file sync module uploads any file or image to Amazon Web Services (AWS) Simple Storage Solution (S3). The module can be used to facilitate a Digital Asset Management solution (DAM) with AWS S3 as the storage backend.

Features Overview

• Upload files to multiple S3 buckets
• Generate image derivatives/version based on existing image styles
• Use custom S3 object name for file uploads; integrates with Token module if available
• Uses asynchronous processing for image derivative generation and file uploads as to not delay node updates or saves
• Individual configuration options for each file field or image field

Requirements

• PHP 5.3 or higher
• Composer Manager module for AWS SDK

Installation Instructions

This module requires an IAM user. AWS has more details about IAM User Management

IAM User Setup

In your AWS account, setup an IAM user that will be used to upload files to S3. Add a policy that allows this IAM user upload permissions to the buckets you are planning to use. Here is an example for a bucket called "bucketname":

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1461098061000",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketAcl",
                "s3:GetBucketCORS",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging",
                "s3:GetBucketNotification",
                "s3:GetBucketPolicy",
                "s3:GetBucketRequestPayment",
                "s3:GetBucketTagging",
                "s3:GetBucketVersioning",
                "s3:GetBucketWebsite",
                "s3:GetLifecycleConfiguration",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTorrent",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTorrent",
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:RestoreObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname*"
            ]
        }
    ]
}

Get the access key id and secret access key id from AWS. Edit your Drupal settings file to add the access key id and secret access key for the IAM user that you setup. You can currently only use a single IAM user for your Drupal site, for example:
$conf['s3_file_sync_s3_bucket_access_key_id'] = 'AKIAxxxxxxxxxxxxx';
$conf['s3_file_sync_s3_bucket_secret_access_key'] = 'eyrzXxxxxxxxxxxxxx';

Bucket Configuration

Navigate to the S3 File Sync configuration site at /admin/config/media/s3_file_sync.

• Create a new bucket configuration at /admin/config/media/s3_file_sync/add, which is a target bucket that you will be able to use in a file or image field.
• • Bucket Name: Name of the S3 bucket
  • Region: Select the region of the bucket
  • ACL: Select the visibility for your assets, currently only supports default AWS ACLs
  • URL Settings: Select the URL method and protocol. These settings do not affect file uploads to S3. Instead, they affect how the URLs for the files that were uploaded to S3 are generated for each file field. After a file is successfully uploaded, all URLs get cached.

Configure an image field to be used with S3 File Sync

This example assumes that you have several image styles setup. You can manage image styles at /admin/config/media/image-styles. This example also assumes that you have an image field in use in one of your content types. You can manage your content types and associated fields at /admin/structure/types.

Navigate to the image field you want to setup and scroll down to the section called "S3 File Sync Settings". Configure the following options:

• Bucket: Select one of the buckets you have setup as the target buckets for image uploads
• File Upload: Enable or disable S3 File Sync for this field. If enabled, the original image file will be uploaded under the name "original.[mime-type]"
• Image Styles: Select the image styles that should be generated and uploaded for this field.
• Queue Processing: Select when files should be processed. By default, files are processed instantly, either on file upload or node save. This adds processing time to the saving processes and will delay execution. Alternatively, files can be queued using Drupal queues. The queue will be processed on cron. Optimally, files should be queued and processed instantly through an asynchronous call. This combines the best of both worlds, where files get queued first, followed by an almost-instant processing of the queue. This should allow for images to be generated and files to be uploaded instantly without delaying page execution.
• Custom Directory Name (and object name): Leave this field empty to use the default directory name based on this token: [current-date:custom:Y]/[current-date:custom:n]/[current-date:custom:U]-[file:fid]. The components for each directory will contain the year, month, unix timestamp of upload, file id, and mime type. An example for a parsed version for this token would be 2016/3/1458914029-578.png. Alternatively, you can set a custom directory name using the tokens listed below. The directory name needs to be unique. I strongly recommend using a unique identifier (e.g. file id, unix timestamp) to fulfill this requirement.

Save the page and try out a new upload.

Configure a file field to be used with S3 File Sync

Follow the same instructions as for image fields. File fields will not show configuration options for image styles.