In my case, for a project with a very large public:// folder, the "s3fs-copy-local" command may needed to be started and stopped.
When the command was stopped and started again, the copy task would re-copy files that had previously been ported to the s3 bucket which would take up a lot of unnecessary time.
I have created a patch to allow a log file to be nominated to store copied file names.
Add the --log option after specifying a log file in 'File Renaming Settings' in /admin/config/media/s3fs. Files in the log file will be excluded from any future copy to s3 when --log option is used.
Note: It will only work for the drush command.
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | s3fs-log-copied-files-3129789-6.patch | 12.72 KB | sprouse_moose |
| #5 | s3fs-log-copied-files-3129789-5.patch | 12.72 KB | sprouse_moose |
| #4 | s3fs-log-copied-files-3129789-2.1.patch | 12.56 KB | sprouse_moose |
| #2 | s3fs-log-copied-files-3129789-2.patch | 10.22 KB | sprouse_moose |
Comments
Comment #2
sprouse_moose commentedComment #4
sprouse_moose commentedAdded ability to exclude paths within current scheme.
For example you may want to exclude all files within public://tmp/
Comment #5
sprouse_moose commentedAdded cache to retrieving log file
Comment #6
sprouse_moose commentedFixed issue with S3fsFileMigrationBatchInterface.php created from comment #5
Comment #7
naveenvalechaMoving to 8.x-3.x branch.
Comment #8
cmlaraI've only quick glanced this patch so I may be missing items or overestimating the concerns, however when I look at this I'm first hit by a concern of the fact this will do
$log = file_get_contents($log_file);. Depending upon filename length and number of files this seems like it could pose a probability of exhausting memory.I believe we will end up with the following usages for 10mb of ram:
50 character average file name every 200,000.
50 character average file name with 5 of those characters being 3 byte UTF8: 166,666 files.
Worst Case Scenarios for 10mb of memory:
250 character file names: 40,000 files.
250 characters all characters 3 byte UTF8: 13,333 files.
My first thought to this is maybe we should use a database table as a log instead of a file. If we go to using a database table as a log we also already have the S3FS" table available to us, I wonder if we might be better moving this patch idea around a little bit and instead make it less about logging that we transferred a file and instead more about "do not upload files that already exist." That question being in that case of what methods we use for choosing "exists" is it simply the file is present, or do we match additional attributes such as file size and check that newer than what we uploaded, etc.
We currently would need to check the s3fs_file table for public:// and s3://public-prefix/ (also same for private://) until I can circle around to #3202909: File migration does not update database entries from s3://prefix
Comment #9
cmlaraWon't Fix in favor of #3208666: copy local based on file stats which works via GUI and command line.