Well, our client's bucket has grown to at least 1M files, and drush s3fs-refresh-cache now dies (with PHP memory_limit set to 1G):
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 12132480 bytes) in /Users/mikeryan/Sites/acme/web/core/lib/Drupal/Core/Database/Connection.php on line 403
Maybe we shouldn't hold all the metadata in memory before writing it to the DB? Yeah, let's do the chunking inside the main loop there...
| Comment | File | Size | Author |
|---|---|---|---|
| #13 | s3fs-2986407-save-memory-alpha14-13.patch | 2.07 KB | stevenlafl |
| #12 | s3fs-2986407-save-memory-dev-12.patch | 2.07 KB | stevenlafl |
| #2 | s3fs-2986407-save-memory-2.patch | 1.25 KB | mikeryan |
Comments
Comment #2
mikeryanHere it is. Running in drush:
A bit better than 1G... I ended up with 1,038,283 rows in s3fs_file.
Also, although I neglected to take the timing previously, it felt quite a bit faster this time.
Comment #3
coredumperror commentedAwesome, thanks! I'll see about back-porting that to D7 when I get a chance.
I bet it ran faster due to lack of any need to swap RAM to disk.
Comment #4
jansete commentedNice!! I hope review soon for D8!
Comment #6
jansete commentedNow is in dev branch, thanks a lot mikeryan!
@coredumperror I change version of the issue to review Drupal 7 back-port.
Greetings!
Comment #7
sebas5384 commentedHi,
this patch worked like a charm! but I wasn't able to find it at first since I was looking for 8.x issues.
@jansete what do you think if we create another issue to port back the patch to 7.x and leave this issue with the right version?
thanks!!
Comment #8
stevenlafl commentedI have a very, very, very large S3 bucket. There is probably just a large amount of metadata in the 10K and perhaps the chunk size should be lowered, or at least have it configurable for those cases.
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 4096 bytes) in /var/www/docroot/core/lib/Drupal/Core/Database/Statement.php on line 59
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 4096 bytes) in /var/www/vendor/composer/ClassLoader.php on line 382
Comment #9
stevenlafl commentedThe part about adding folders with the values() query ignored the fact there may be a very large amount of folders on S3.
I chunked it out into 200/ea for the INSERT query.
Comment #10
stevenlafl commentedWrong version, sorry. This is not a dev patch, it is a patch for alpha13 to replace this one.
Comment #11
stevenlafl commentedIgnore this one
Comment #12
stevenlafl commentedBoth versions are in this one so it's accessible to everyone.
Folder query is limited to 200 per insert. Someone should do some testing to see what is sane there. It could be the 10,000 above it is fine here too.
Comment #13
stevenlafl commentedIs no one going to look at this?
Alpha 14.
Comment #14
naveenvalechaMoving to 8.x-3.x branch.
Comment #16
cmlaraCommitted the @stevenlafl additional enhancement for folders writing.