Storage api cron run was generating several hundred of the following messages:
"Amazon S3: Amazon S3 error: A header you provided implies functionality that is not implemented"

Traced it to some erroneous storage records, it's not clear how they GOT there yet (And that is my next task), but as the source files do not actually exist they stay flagged as STORAGE_STATUS_PROCESS_CRON and are processed over and over again. When the number of these records exceeds a few thousand, it begins to affect resources.

As the cron system attempts to resolve every record flagged STORAGE_STATUS_PROCESS_CRON, and does not (guarantee to) process them LIFO, it is possible to have cron exceed timeouts without ever seeing new records. This should also be resolved, but separately.

Using modules:
storage
storage_core_bridge
s3 plugin

To duplicate the issue, create a storage record referring to a non-existent file on filesystem container.

This patch checks if the source file exists during the check phase of cron. If it doesn't, it flags that file for removal and logs an error. It's unclear to me if the remove should be recursive or if I should use something other than file_exists as it's not necessarily a file from local filesystem. I also want to make sure that file_managed is treated appropriately.

Feedback requested.

EDIT: attached wrong patch. I cannot brain today.

Comments

Silicon.Valet’s picture

Issue summary: View changes
StatusFileSize
new1.97 KB
Silicon.Valet’s picture

StatusFileSize
new1.12 KB

Try number the second.

perignon’s picture

Git apply often drives me nuts.

perignon’s picture

Commited! Thanks for the patch!

perignon’s picture

Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.