Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.File handling in this module assumes that you can get a lock on files... that's not always true. I ran into a case where I"m using this with a remote filesystem (S3). Simple patch attached to make sure the filesystem is SUPPOSED to support locking, before failing.
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | 2609414-7.patch | 1.26 KB | ricovandevin |
| #5 | file_resup-support-non-locking-fs-2609414-5-D7.patch | 2.11 KB | blake.thompson |
| support-non-locking-fs.patch | 639 bytes | ohthehugemanatee |
Issue fork file_resup-2609414
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes
Comments
Comment #2
haysuess commentedThanks for this. I am using Amazon S3 and was having trouble where the upload progress would go up to 0.9%, then seemingly restart and go up to 0.9% again about 5 times before failing.
This patch works and I can successfully upload files now larger than my PHP limits allow. It will go in 0.9% increments (for this particular upload at least), then pause, and continue going up until it's complete.
For some reason, it seems to upload EXTREMELY slow, and the further along it gets, the longer the progress takes to update it seems.
I'm uploading a 220MB file, and it took almost 40 minutes. I can normally easily upload that via FTP or other ways in a couple minutes. Any ideas?
Comment #3
bburg@haysuess
I believe the issue you are running into is described in this hacker news thread. This issue is that when "appending" to objects in S3, the entire object is loaded into memory, then appended to with the next chunk, a new object is written to storage, which becomes a problem for very large files. My co-worker calls is an problem of "exponential time complexity" since the amount of time to load the new version of the file into memory increases with with every append.
The solution would be to upload the file into a temporary directory on the local file system first, then send it to S3 all at once. An alternative might be to upload each chunk as a separate S3 entry, then combine them at the end. My knowledge in this area and this module is limited though, but, I'm currently working to resolve this issue on a site now, so I may be able to turn up something.
Comment #4
haysuess commented@bburg, Thanks for the info.
I've since upgraded to S3FS 2.x and S3FS CORS Upload and everything works perfectly, so I would recommend that for everyone.
Comment #5
blake.thompson commentedI'm attaching an updated patch that writes to the temporary file scheme before using file_unmanaged_move() to place the uploaded file in the destination scheme. This helps with upload speeds when dealing with S3.
Since this swaps out rename() with file_unmanaged_move() it may be important to consider the recommended patch on this projects page.
Comment #6
ricovandevin commentedThis problem is still present in the versions for Drupal 8+. The solution in #5 is still working so I will work on a port of the patch.
Comment #8
ricovandevin commentedA patch of MR !12 for Composer workflows. That patch is against 2.0.x-dev.
The MR and the patch do not include the changes for using the temporary:// stream wrapper for the initial upload. Although related I think that is a different issue. I have created #3481484: Slow file uploads when used with S3FS for that.