This issue was recreated due to it being closed after committing to D8, even though a D7 patch was made available
Problem/Motivation
When drupal moves a file it issues a copy() and then an unlink() this causes a very significant amount of I/O. If the source and destination are on the same filesystem and rename() is issued instead then virtually no I/O occurs becuase filesystems can perform this operation by just changing a pointer on disk.
Proposed resolution
Instead of file_unmanaged_move() calling file_unmanaged_copy(), break out the common code into a new function file_unmanaged_prepare() and have file_unmanaged_move() attempt to issue a rename(), falling back to copy() and unlink() if this does not work.
This optimizes drupal_build_css_cache(), drupal_build_js_cache(). It also prevents a race condition with these functions where other processes or servers may start reading the files before they are fully written.
A further advanatage of issuing a rename() is that some remote schemes have implemented this method, e.g. it would be possible for a file to be moved to a different location on Amazon S3 without having to download it and re-upload it to the correct location.
Remaining tasks
Reviewing.
User interface changes
None.
API changes
None.
Original report by bvanmeurs
Hi, I'm using Drupal to host a huge photo archive (one node per file). I automatically create taxonomy tags for new directories, and files within those directories are automatically inserted as a node.
When a tag name is updated, I want to automatically change the directory name by creating a new directory, and move all of the files to the new directory. I do this using the function file_move.
When I update the root tag, containing all files (about 1000) it takes more than a minute to complete. I cachegrinded the request and found out that 99% of the resources were spent in the file_unmanaged_move function.
Researching the function, I found out that it simply copies the file and deletes it. I am working with hughe photo files (10mb each) so that means that about 10 gig of data is copied. That takes a whopping time of course, whereas a move would be very fast. I do not see any reason not to implement an efficient move function here. Just adding a parameter to the file_unmanaged_copy and using rename instead of copy would do the trick for me!
Any chance for a fix?
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | fast_move-3098526-2-D7.patch | 2.99 KB | nwom |
Comments
Comment #2
nwom commentedThe following patch was created by Anrikun here: https://www.drupal.org/project/drupal/issues/1377740#comment-9264011, specifically for the File Resumable Upload module.
Comment #3
chaseonthewebIsn't this a duplicate of #2752783: [D7] file_unmanaged_move() should issue rename() where possible instead of copy() & unlink()?
Comment #4
poker10 commentedYes, it seems like a duplicate of that issue.