Problem/Motivation
Variant of #2976759: Improve copy local to S3 process: Temporal table with all scaned files for we can restart the process if something it was wrong. and #3129789: Log copied files
Scans all the files to create the batch and when the file is being processed check if we should always upload, or only upload if the file is newer than when we uploaded to s3fs or a different size.
Proposed resolution
Add checks based on file stats (mtime and size) to determine if a file should be copied.
Remaining tasks
User interface changes
Action form add dropbox to select condition. Also added some CSS to place the buttons on a horizontal row as with the addition of the dropdown box they did not look as reasonable spread over two lines.
API changes
S3fsFileMigrationBatch::execute() and S3fsFileMigrationBatch::copyOperation now take an additional parameter of an array.
Current format for the array is
[
'upload_conditions' => [ 'always', 'newer', 'size']
]
Array allows us add additional options such as a 'max_files' key for #2976760: Improve copy local to S3 process: Limit number of files to upload per process. or a 'files_per_batch' key if needed in future.
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | copy_local_based_on-3208666-6.patch | 15.63 KB | cmlara |
| #6 | interdiff-3208666-5-6.txt | 2.34 KB | cmlara |
| #5 | copy_local_based_on-3208666-5.patch | 14.5 KB | cmlara |
| #5 | interdiff-3208666-4-5.txt | 3.27 KB | cmlara |
| #4 | interdiff-3208666-2-4.txt | 3.25 KB | cmlara |
Comments
Comment #2
cmlaraComment #3
cmlaraComment #4
cmlaraAdded newer_size as an option on GUI and CLI for newer OR size differs.
Refactored some of the code to process the options easier.
Comment #5
cmlaraAdded tests.
Comment #6
cmlaraCode style error cleanup.
Includes adding a .csslintrc file because we seem to be hitting #3196980: Coding standard "line 0 Could not read file data. Is the file empty?" on .css file and I want to test a theory on is it because of a missing .csslintrc file.
Comment #8
cmlara