Problem/Motivation

Variant of #2976759: Improve copy local to S3 process: Temporal table with all scaned files for we can restart the process if something it was wrong. and #3129789: Log copied files

Scans all the files to create the batch and when the file is being processed check if we should always upload, or only upload if the file is newer than when we uploaded to s3fs or a different size.

Proposed resolution

Add checks based on file stats (mtime and size) to determine if a file should be copied.

Remaining tasks

User interface changes

Action form add dropbox to select condition. Also added some CSS to place the buttons on a horizontal row as with the addition of the dropdown box they did not look as reasonable spread over two lines.

API changes

S3fsFileMigrationBatch::execute() and S3fsFileMigrationBatch::copyOperation now take an additional parameter of an array.

Current format for the array is

[
  'upload_conditions' => [ 'always', 'newer', 'size']
]

Array allows us add additional options such as a 'max_files' key for #2976760: Improve copy local to S3 process: Limit number of files to upload per process. or a 'files_per_batch' key if needed in future.

Data model changes

Comments

cmlara created an issue. See original summary.

cmlara’s picture

StatusFileSize
new10.83 KB
cmlara’s picture

Status: Active » Needs review
cmlara’s picture

StatusFileSize
new11.38 KB
new3.25 KB

Added newer_size as an option on GUI and CLI for newer OR size differs.
Refactored some of the code to process the options easier.

cmlara’s picture

StatusFileSize
new3.27 KB
new14.5 KB

Added tests.

cmlara’s picture

StatusFileSize
new2.34 KB
new15.63 KB

Code style error cleanup.

Includes adding a .csslintrc file because we seem to be hitting #3196980: Coding standard "line 0 Could not read file data. Is the file empty?" on .css file and I want to test a theory on is it because of a missing .csslintrc file.

  • cmlara committed 680c67e on 8.x-3.x
    Issue #3208666 by cmlara: Copy local based on file stats
    
cmlara’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.