When calling apachesolr_attachments_extract_text() on Google Object Storage, our (patched) module needs to download the attachment files every time to calculate the SHA256 of the file. This triggers a GET request to download the file and incurs in a performance penalty and associated request costs.
Google Object Storage allows use of hashes both to check uploads and to verify objects by requesting the object's metadatas, be it MD5, CRC32c or ETags:
https://cloud.google.com/storage/docs/hashes-etags
http://stackoverflow.com/questions/25234515/how-to-get-the-md5-of-an-obj...
Both JSON and XML APIs allow getting an object metadatas by sending specific requests, which the storage backend handles faster, without sending the whole file.
BTW, looks like S3 allows it, too: http://stackoverflow.com/questions/1775816/how-to-get-the-md5sum-of-a-fi... .
Since object storage capabilities can be abstracted via Stream Wrappers, PHP native md5_file function could be implemented efficiently on a stream wrapper to take advantage of storage backend available metadatas.
The only needed modification would be at hash writing and retrieval to add the hash algorithm before the actual hash value. Of course, defaulting to SHA256 if none present.
The 'hash' field value would be, for instance 'md5:45c5991be3c9893aadf3e310480bbbf1'.
Then, the hash verification function could call specific PHP functions, such as md5_file or sha1_file to get check the hash. Site admins would be able to select among a few hash functions or leave the default 'sha256' option, as it is now.
Would like to get feedback about possible issues or caveats and if such a patch would be likely to be accepted.
Comments
Comment #2
amontero