Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Well, our client's bucket has grown to at least 1M files, and drush s3fs-refresh-cache now dies (with PHP memory_limit set to 1G):
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 12132480 bytes) in /Users/mikeryan/Sites/acme/web/core/lib/Drupal/Core/Database/Connection.php on line 403
Maybe we shouldn't hold all the metadata in memory before writing it to the DB? Yeah, let's do the chunking inside the main loop there...
| Comment | File | Size | Author |
|---|---|---|---|
| #13 | s3fs-2986407-save-memory-alpha14-13.patch | 2.07 KB | stevenlafl |
| #12 | s3fs-2986407-save-memory-dev-12.patch | 2.07 KB | stevenlafl |
| #2 | s3fs-2986407-save-memory-2.patch | 1.25 KB | mikeryan |
Comments
Comment #2
mikeryanHere it is. Running in drush:
A bit better than 1G... I ended up with 1,038,283 rows in s3fs_file.
Also, although I neglected to take the timing previously, it felt quite a bit faster this time.
Comment #3
coredumperror CreditAttribution: coredumperror commentedAwesome, thanks! I'll see about back-porting that to D7 when I get a chance.
I bet it ran faster due to lack of any need to swap RAM to disk.
Comment #4
jansete CreditAttribution: jansete commentedNice!! I hope review soon for D8!
Comment #6
jansete CreditAttribution: jansete commentedNow is in dev branch, thanks a lot mikeryan!
@coredumperror I change version of the issue to review Drupal 7 back-port.
Greetings!
Comment #7
sebas5384 CreditAttribution: sebas5384 at Taller commentedHi,
this patch worked like a charm! but I wasn't able to find it at first since I was looking for 8.x issues.
@jansete what do you think if we create another issue to port back the patch to 7.x and leave this issue with the right version?
thanks!!
Comment #8
stevenlafl CreditAttribution: stevenlafl as a volunteer and at Mobomo commentedI have a very, very, very large S3 bucket. There is probably just a large amount of metadata in the 10K and perhaps the chunk size should be lowered, or at least have it configurable for those cases.
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 4096 bytes) in /var/www/docroot/core/lib/Drupal/Core/Database/Statement.php on line 59
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 4096 bytes) in /var/www/vendor/composer/ClassLoader.php on line 382
Comment #9
stevenlafl CreditAttribution: stevenlafl as a volunteer and at Mobomo commentedThe part about adding folders with the values() query ignored the fact there may be a very large amount of folders on S3.
I chunked it out into 200/ea for the INSERT query.
Comment #10
stevenlafl CreditAttribution: stevenlafl as a volunteer and at Mobomo commentedWrong version, sorry. This is not a dev patch, it is a patch for alpha13 to replace this one.
Comment #11
stevenlafl CreditAttribution: stevenlafl as a volunteer and at Mobomo commentedIgnore this one
Comment #12
stevenlafl CreditAttribution: stevenlafl as a volunteer and at Mobomo commentedBoth versions are in this one so it's accessible to everyone.
Folder query is limited to 200 per insert. Someone should do some testing to see what is sane there. It could be the 10,000 above it is fine here too.
Comment #13
stevenlafl CreditAttribution: stevenlafl as a volunteer and at Mobomo commentedIs no one going to look at this?
Alpha 14.
Comment #14
naveenvalechaMoving to 8.x-3.x branch.
Comment #16
cmlaraCommitted the @stevenlafl additional enhancement for folders writing.