S3FS works wrong if there are more than 1000 files in a bucket

If the there are large amount of objects in S3 the query will make the database "go away". This patch will split the iterator into chunks, it is easy to the database.

Hi! I try to review the most of pending patches the next week, greetings!

Hi @jansete - I've run into this issue as well - it's not just when there are more than 1000 files in the bucket, it's also when there are more than 1000 versions of files in the bucket.

On my side what happened was that I moved a folder (in a bucket with versioning turned on) from location A to location B, and the "ListObjectVersions" call returned only ~1000 versions of files - which was only the old versions, before the move. So my s3fs_file table ended up not containing the "current" versions of any of those files. The AWS doco for "ListObjects" mentions that it only returns ~1000 files max and that you need to do the "GetIterator" call to get more than that. There's no such doco on the "ListObjectVersions" method but I suspect it's the same restriction - there's only so much data they can send over HTTP at once, so there must be some limit without the iterator.

I can confirm that the patch in #2 works for me to fix this issue.

I haven't tested the patch in #3 (I have less than 10000 rows in my s3fs_file table so I think it wouldn't do anything on my side anyway), but it looks sensible to me - I'd expect MySQL on lots of sites to have timeout issues if we tried to write more than 10s of thousands of rows to it at once.

HTH.

Nice job guys! Committed in dev branch :)

This fix now is in the new alpha5 release.