Add a storage service: Allow configurable file_directory_path per site.

re. (2) I'd say it would make a good candidate for #2180053: Include more "golden" contrib.

Is the intention here mostly to remove files from backups to speed up clones and migrations? That's always been our use-case for doing something like this.

Some documentation would also be nice. Presumably the various scripts are to allow setup and tear-down of NFS mounts or such?

Wow! That is really golden! :) I'm going to test this this weekend.

Ok, well then bear in mind that backups don't follow symlinks. Perhaps we'll also need to expose whether to follow symlinks in backups as a drush option, so it can be overridden.

I strongly feel that support for storing Drupal files in a different location + still allowing backups, etc. is a pretty important thing.

YES. Strongly Agree.

I am constantly apologizing to people for aegir forcing you to store your files in sites/yourcrappyoriginalsitename.com/files.

First step is to stop forcing the site files path, but still save whatever path that is to the Site context.

Next would be improving clusters to setup NFS automatically.

Not sure, but I don't see a need to "symlink them into place" if we can set the 'file_directory_path' variable.

for backups: we could make backup in two stages. first backup the site directory, then add the files on top of that. that way we don't need to follow symlink, which is inherently insecure (i could just `ln -s /var/aegir/platform/foo/sites/victim.com /var/aegir/platform/foo/sites/attacker.com/files/hahaha` to read all files from victim.com).

that said, i don't think we should split files out of the directory, but move the whole directory completely, as in #1205458: Move modules/themes/libraries/files/private directories out of /sites/example.com. that way we don't need to follow symlinks for backups and things are more consistent. ergonlogic did a lot of good work there and we were pretty much done with that.

i do prefer the approach of having a generic storage service. that's a long standing goal of the project. so i would suggest to focus on this issue for now.

i think the remaining steps here would be:

1. make this work for settings.php and friends (ie. move all of sites/example.com out and keep symlinks back
2. make an upgrade path (i would propose to make "verify" detect if there's a filestorage configured and move files into it when relevant)
3. make sure we don't backup and deploy in migrate (that's the whole point of this optimisation - just backup so we can rollback!)
4. (we probably need to fix rollbacks as well in that regard)
5. fix remote server support (i suspect this could be fixed the same way as for backups: just rsync the symlink target)

alright, so first tests on this show that this extension cannot be installed in a site-specific directory, so that's bad, see #2185627: Add a storage service: Allow configurable file_directory_path per site. for what seems to be a bug in the aegir API. we're also trying to port the module to d7 so it works with aegir3.

next I postponed #1205458: Move modules/themes/libraries/files/private directories out of /sites/example.com for now while we shake down this specific tree.

I believe the issues we were seeing with hosting_ and provision_storage were largely due to #2374841: weird bootstrapping problem when defining services in sites. I have an almost complete port of these to Aegir3, and we'll be further testing them this week. I also merged the provision component into the hosting module, as per #2300537: Merge Provision extensions into Hosting modules.

We're considering whether to use this approach to move the entire site out of the platform, as in #1205458: Move modules/themes/libraries/files/private directories out of /sites/example.com.

Just a thought:
Why to synch symlinks?
Would it be easier to exclude necessary folders from synch, and once everything is pushed to remotes - create symlinks to those folders on each remote server?

Cheers!

This is something we've wanted for devshop for a long time. Since we don't use multisite we want to be able to just use sites/default/files, like so many other hosting systems use.
There are a few small steps we should take immediately:

Make file_directory_path a Site context property, so whatever is saved in the drush alias gets written to the settings.php file. (defaults to site/domain/files, but should also be optional).
Create a property on site nodes for this, and default it to site/domain/files.

Those two things could be acceptable changes to provision and hosting.module. We don't need a UI change in aegir core. Any module can use nodeapi to alter the site node property.

No need to overcomplicate it at this point. Let's start by taking these first steps.

Here's a newer issue I used to store the work in progress: #3016995: Make file directories first class context properties.