amazon s3 storage service

I'm working on this in contrib, over on Github (https://github.com/GetValkyrie/hosting_s3). The Aegir integration itself is largely done, based on hosting_site_data, as well as some ideas from @cweagans' hosting_storage. All the functional stuff remains: creating/syncing/deleting buckets.

A couple further considerations:

• I'd initially thought to use https://www.drupal.org/project/awssdk, which provides a makefile and libraries integration. It is also used by https://www.drupal.org/project/amazons3. This module uses AWS SDK v1, which is deprecated. AWS recommends migrating to v2. There is a v2 module, but it is little used, and has some questionable dependencies.

  Also, this code needs to be available for our backend operations, and not the front-end. As such, installation in libraries, etc. doesn't help much. I think I'll use composer to just embed the SDK directly in the project. It's licensed under the Apache license, which allows for this usage. We can revisit this decision, if need be, later.

• While we had originally specified that we'd have a set of credentials per client, so far, I've implemented these fields at the site level. There is a DB table to store these for clients, but nothing else is yet implemented along these lines. I have a TODO in the code to basically have the client's credentials be the default, but allow for overrides on a site-by-site basis. For initial prototyping, I'll probably just stick with site-level credentials.

Currently, this is all-or-nothing. That is, Aegir will run the S3 logic for all site. We have _provision_s3_enabled(), which is where we could determine whether to handle s3 provisioning or not.

I'm reconsidering my second point above. I think we should present blank fields for the AWS keys, and skip S3 if they are left empty. A 'use client credentials' checkbox could then disable those fields. This seems cleaner than having a separate checkbox for whether to use s3.

For the sake of simplicity, these settings will be read-only, once the site is installed. Later, we can figure out how to convert an existing site to use S3 or vice-versa.

I'm also reconsidering my first point in #5. While the AWS SDK needs to be available to the back-end, it's also useful to have it in the front-end. For example, we might want to report the status of the bucket in the front-end, and so could call the SDK methods in hook_node_save().

Added the SDK, and now validate that the supplied credentials allow access to S3 when saving the site node form.

We need to save a bucket name in the context, and then inject it (among other things) into settings.php. The S3Client object conveniently provides both isValidBucketName() and doesBucketExist() methods. For now, I'm going to call these from our validation hook, so we can easily save the bucket name to the DB and the context. In the long run, we should probably look at doing this just prior to actually creating the bucket, since there'll be a delay while the task is queued, etc., during which the bucket name could be used by someone else. Not very likely, if we use a pretty unique name, but still a source of potentially difficult to debug issues.

Optionally inheriting credentials from clients complicates things somewhat. To keep the critical code path as simple as possible, I think I'll add submodules to support inheriting creds from clients and users (since that seems like a valid use-case too.)

We now have 2 submodules that allow S3 credentials to be stored on client nodes and user profiles, respectively. Site credentials can then be inherited from either. This inheritance only happens when installing the site. After that the credentials are independently stored associated to the site. So no relationship is kept to the source of their inheritance. Since the use-case for updating credentials on a site remains a bit murky to me, they remain read-only.

We also now share some validation and a factory method between the front-end and back-end code. This was one of the reasons behind #2300537: Merge Provision extensions into Hosting modules. So it's interesting to see how it works. The main challenge was abstracting output, since we use drush_log() and drush_set_error() in the back-end, and drupal_set_message() and form_set_error() in the front-end.

Finally, I moved the bucket name generation and validation into the back-end, where we store it in the site context, and then pass it back to the front-end.

There are a number of S3-compliant storage services out there, and the AmazonS3 module appears to support them. At this stage, I'm implementing all functionality as a Provision service-type, with an empty default implementation for AWS S3. It might be interesting to add other such service implementations, and #2457425: Allow services to be associated with other Hosting entities might help with that.

We're now creating and deleting buckets with sites.

We're now injecting S3 credentials and bucket name into settings.php. This makes enabling amazons3 and setting the filesystem to use for media assets all that is required to use S3.

I considered forcing S3 in for the file-system, and enabling amazons3.module in a post-install hook, but it seems like these are really better left to the site administrator. In particular, we cannot guarantee that the module's dependencies are in place, and site-install is neither the time nor the place to do that.

I think all we can do about that is include a sample makefile and document some best practices, using Features or a custom module.

While probably out-of-scope for this issue, it might also be nice to include sample code for a hook_update_n() function that will push all existing file assets to the S3 bucket. A couple issues already mention this: #2189763: How do you move from Files in sites/default/files -> S3? and #1985962: Option to bulk upload files that were added before the module was installed. The latter, in particular, provides a Drush command to bulk-upload existing files. We might want to integrate that here, possibly as a stand-alone task.

It turns out that backups are more challenging than I'd anticipated. Copying a bucket is fairly straight-forward. But we then need to pass the new bucket name into the settings.php that gets packaged with the backup. We also need to save it to a new table in the database, associated with the backup, to allow for restores and deletion of backups.

Aegir also takes a backup whenever a site is deleted, which just sticks around indefinitely. Presumably we don't want to do the same with buckets. Instead, we might want to stream the bucket contents into the tarball, so we do have a more permanent backup of deleted sites.

I foresee some other possible issues. For instance, both clone and migrate task start with a backup, but this is likely redundant with S3 buckets; at least with the spec as currently written. But if we don't back up the bucket, then we'll have site backups available that cannot be properly restored.

Initial backups of site buckets is in. We still need to change the bucket name when writing the settings.php packaged with the backup.

Thanks in large part to @anarcat, we're now injecting our backup bucket into settings.php in packaged backups. We had to work around how we re-write settings.php for backups (see: #2460867: Write a temporary settings.php for backups and commit cae256eb)

We also now associate the backup bucket to the proper backup bid. Previously, our post task hook was being called before hosting_site's, and so the current backup ID hadn't been generated yet.

After further discussion with @anarcat, we also decided to follow the current sequence for migrate tasks. That is, create a backup bucket, then deploy a new bucket for the migrated site, rather than simply re-using the existing one. We're looking at ways to optimize the migration/update process, so we'll re-visit the null op, we'd originally spec'd here, at that point.

Deleting backups now also works. Moving on to restore/deploy.

Restoring sites now works. It clears the site bucket and copies the contents of the backup bucket. See this issue, where we'll follow-up with finding a better way of clearing stale data.

Clone doesn't work yet, as it'll require setting the site bucket name at some point in the process.

Ran into a nasty bug where we were overwriting our new context with stale data from the front-end, via an etra provision-save. Also, hook_drush_context_import() was never being called. For details, see: #2470297: Site import broken

With those fixed now, everything works for clones, save injecting the backup bucket name into the deploy task. The site backup filename is passed in as an argument to the deploy task, which runs in a subprocess, and so doesn't inherit options. In drush_provision_drupal_provision_clone(), we trigger the deploy task like so:

[...]
  drush_invoke_process('@none', 'provision-save', array($new_name), $options);
  # note that we reset the aliases so they don't conflict with the original site

  provision_backend_invoke($new_name, 'provision-deploy', array(drush_get_option('backup_file')), array('old_uri' => d()->uri));

  if (!drush_get_error()) {
    provision_backend_invoke($new_name, 'provision-verify');
  }

  provision_reload_config('site', d()->site_path . '/drushrc.php');
}

We could perhaps inject the backup bucket name into the options being saved to the site context, but since this is only intended to be temporary, I'm not sure that'd be the best way to proceed.

See: #2471805: Add hook to allow injecting options into clone/migrate deploy tasks

That new hook works beautifully. Clones are now also working. Migrates are next.

Some testing on a fresh install revealed a few issues:

1. backup-delete doesn't appear to remove the entry from hosting_s3_backups table.
2. an 'extra' backup appears to be taken during a backup/restore cycle (other than the usual one before restores).
3. clones may not work... The backup appeared to fail on the site where the failures above had just occurred. But then worked perfectly on another site.
4. migrates don't quite work either. A new bucket is provisioned and sync'd, and the site context points to it. But the site node still points to the original bucket. According to the current Aegir migrate process, the original site bucket ought to be deleted.

It could be that the first couple bugs here interfered with subsequent tests. I'll tackle those first.

There have been some significant developments in the amazons3 project resulting in a 7.x-2.x branch that appears much more capable. However, it required some slight modifications to the hosting_s3 module (mostly just renaming the variables injected into settings.php).

More challenging has been its dependence on composer_manager. This change has meant that amazons3 cannot be installed from a makefile: #2484379: How to deploy from makefile, nor can Composer Manager be run headless in D7. I've submitted a feature request to that effect: #2484875: Support headless operations. In speaking to my fellow Drush maintainers, they've agreed that incorporating such functionality into Drush (and Drush Make) makes sense: https://github.com/drush-ops/drush/issues/1388.

Until I have a chance to work on that, I've basically forked amazons3 (see: https://github.com/GetValkyrie/drupal_amazons3/tree/vendor) and removed the dependency on composer_manager but running 'composer install' within the project, committing the vendor/ dir, and including the generated autoload.php directly.

I've added the hardcoding/overriding of URI schemes to use S3. Eventually, we might want to make this configurable on the site form.

The results of this can be seen over in https://github.com/GetValkyrie/hosting_s3, where development will continue. Issues, feature requests and such should be filed on github.

Note that I'm about to undertake a fairly significant overhaul, to put a site and its backups into a common bucket. S3 sets limits on the number of buckets it will allow. In all likelihood, this re-factoring will break compatibility with the current implementation. I will keep the existing functionality, which we've been using in production for a number of months, in a separate branch.