Before a file can be uploaded to S3 storage, a file key needs to be generated (this is usually the full path). However if a file with the same key already exists in the S3 bucket, then a new key is generated by inserting a numeric value just before the the file extension, and the resulting key tested. As each test requires a query to S3, this quickly gets very inefficient as the number of similarly named files are saved to the bucket.
Instead of testing S3 each time, it will be better to assume that the local cache (table 's3fs_file') is up to date and check that. The check can return the max numeric value already defined, and then the function can increment the value and check S3, with the expectation that it will not already exist. If it does exist then continue incrementing as before, and flag that the table needs to be re-built.
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | s3fs_cors-improve-file-key-function-2977660-2.patch | 5.4 KB | jayelless |
Comments
Comment #2
jayelless commentedPatch attached.
Comment #3
jayelless commentedCommitted.