Problem/Motivation

We are running a pretty big site with a huge number of files (thousands of files, terabytes of data). Since we don't want to replicate all files on every dev/test/acceptance environment we are trying to figure out a way to allow environments to add changes to the files (new image styles, test content, deletes etc) while keeping the production data intact.

Proposed resolution

We thought of the following solution (kind of like stage file proxy):

  • Allow a dev_folder setting to be configured, similar to the root_folder
  • If a dev_folder is set, try to read/write all files from that folder
  • If a file is not available, try to copy from the default folder (or root_folder if configured) to the dev_folder, and serve the file from the dev_folder

This way we can clean up the dev_folders with environment specific changes periodically, while allowing the production data to remain intact.

Remaining tasks

Discuss
Write a patch

User interface changes

API changes

Data model changes

Comments

seanB created an issue. See original summary.

mikeryan’s picture

s/thousands of files/1645530 rows in s3fs_file/

mikeryan’s picture

Status: Active » Closed (duplicate)

@seanB: As I work on a blog post on our experiences with s3fs, I now see there is an existing project for just this purpose, https://www.drupal.org/project/s3fs_file_proxy_to_s3. It's alpha and "minimally maintained", but that would be the place to pursue this further.

seanb’s picture

@mikeryan thanks!