Problem/Motivation
We are running a pretty big site with a huge number of files (thousands of files, terabytes of data). Since we don't want to replicate all files on every dev/test/acceptance environment we are trying to figure out a way to allow environments to add changes to the files (new image styles, test content, deletes etc) while keeping the production data intact.
Proposed resolution
We thought of the following solution (kind of like stage file proxy):
- Allow a dev_folder setting to be configured, similar to the root_folder
- If a dev_folder is set, try to read/write all files from that folder
- If a file is not available, try to copy from the default folder (or root_folder if configured) to the dev_folder, and serve the file from the dev_folder
This way we can clean up the dev_folders with environment specific changes periodically, while allowing the production data to remain intact.
Remaining tasks
Discuss
Write a patch
User interface changes
API changes
Data model changes
Comments
Comment #2
mikeryans/thousands of files/1645530 rows in s3fs_file/
Comment #3
mikeryan@seanB: As I work on a blog post on our experiences with s3fs, I now see there is an existing project for just this purpose, https://www.drupal.org/project/s3fs_file_proxy_to_s3. It's alpha and "minimally maintained", but that would be the place to pursue this further.
Comment #4
seanb@mikeryan thanks!