Incompatibility on latest AmazonS3 v2

Yep, looks like this module doesn't work with 2.x of Amazon S3. Not sure what it's going to take to get it to work with 2.x, but I'll try to look into it tomorrow.

First issue I've found is that it appears that the database setup is different. The database table used is now "cache_amazons3_metadata" instead of "amazons3_file". See #2470127: Use doctrine/cache for request caching.

Just based on this I think it will be necessary to start a 2.x branch for amazons3_ffs. That will also help people know which version to download. Of course we need to get this working with 2.x first...

It has been suggested that we simply use AWS SDK and call listObjects() to retrieve files from the S3 server. @torgosPizza, do you have any ideas on how this implementation might work? It seems like we would need to change this function:

$s3files = db_select('amazons3_file', 'af')
    ->fields('af', array('uri'))
    ->condition('uri', 's3://%' . db_like($like) . '%', 'LIKE')
    ->execute()
    ->fetchAllKeyed(0, 0);

to something like this:

$s3files = $client->listObjects(array(
    // Bucket is required
    'Bucket' => '$bucket',
    'Marker' => '$filename',
    'MaxKeys' => 50,

Of course there is more that we would need to do and I'm pretty sure what I have above isn't exactly right, but I'm just trying to get an idea if I'm on the right track...

Well, the cache table is used to store the list of files in, and populate the autocomplete with, so using listObjects() is not the best option. An API call is expensive, and even moreso when you have an S3 bucket with 20,000+ files like we have. We could use listObjects() to populate the cache table, but I wouldn't call to the S3 API every time we want to look up a large number of files. But I'm open to suggestion.

Also yes I agree that making a new branch might be a good idea, especially as the AWS SDK library continues to be updated, along with the amazons3 module itself.

@deggertsen I have made you a maintainer for amazons3_ffs, if you'd like to create a new branch, please feel free.

So perhaps it's best that we recreate the "amazons3_file" table for this module as it doesn't seem viable to use the cache used for amazons3 anymore. See #2563455: How to use data in the new cache?.

Thanks for making me a maintainer. I'll see what I can do.

Interesting, thanks for posting that. The cache table {amazons3_file} isn't part of Drupal cache either, which I think is why it probably exists already. So I'm not sure that this is really the main issue. We should investigate the main OP and find out what the root of the problem is.

My main guess is that it's a PHP SDK library problem, first and foremost. If we knew more about the types of changes are from 1.x to 2.x then I think this could be changed to a Meta issue.

But yes, I think recreating the cache table is probably a good place to start. I think the caching that #2470127: Use doctrine/cache for request caching refers to is more of a request-time issue as opposed to caching a large amount of metadata like we are currently doing in the 1.x branch.