Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
_file_save_upload_single()
just needs to upload a file and create a temporary file entity. However, it currently mixes in form api and drupal messages. We should clean this up and move it to a separate service to be re-used in other places, such as JSON API and REST modules.
Steps to reproduce
Proposed resolution
Move the logic in _file_save_upload_single()
to a re-usable service.
Remaining tasks
User interface changes
API changes
_file_save_upload_single() is deprecated and moved to a service.
Data model changes
$file->source is no longer set. It was erroneously assumed $file was a file entity and setting $file->source had no effect.
Release notes snippet
Comment | File | Size | Author |
---|---|---|---|
#74 | 3232248-74-interdiff.txt | 2.74 KB | kim.pepper |
#74 | 3232248-74.patch | 27.81 KB | kim.pepper |
Comments
Comment #2
andypostGreat! it will help to centralize access to request's files and transition to entity
Comment #3
kim.pepperInitial patch.
Comment #4
daffie CreditAttribution: daffie commentedThe testbot is not happy.
Comment #5
andypostFix linter, and removed string translation dependency and exceptions now using
sprintf()
(for linter)Comment #7
kim.pepperFixes some of the incorrect namespace issues and the service definition.
Comment #9
kim.pepperSet a default for $destination and add a deprecation and test.
Comment #10
kim.pepperFix unused imports.
Comment #11
andypostComment #12
kim.pepperAdds @group annotation.
Comment #14
kim.pepperFixes return type of
loadByUri()
Comment #16
kim.pepperCatch FileException
Comment #18
kim.pepperStruggling a bit to work out the test fails. Looks like an error message has changed, but can't see what it is.
Comment #20
paulocsI was able to figure out why the tests are failing in
RemoteFileSaveUploadTest::testDrupalMovingUploadedFileError
but not inRemoteFileSaveUploadTest::testHandleExtension
andRemoteFileSaveUploadTest::testHandleDangerousFile
.Comment #21
kim.pepperI figured out the hooks fail. We are doing a file load to see if the file exists before create, so we need to add
'load'
to the expected hooks.However, test now failing on 'For security reasons, your upload has been renamed' not being displayed. From what I can tell, when file_validate() is called, it still thinks
'allow_insecure_uploads'
is true. Can't see why that would be the case.Comment #23
kim.pepperFixes call to load hook validation in SaveUploadFormTest too.
Comment #24
andypostNot sure having this is a good idea, it exposes extra API which is not so much handy (only 1 usage to replace)
Could use better docs - it's kinda converter to value object
Comment #25
kim.pepperAddresses feedback for #24. I renamed
FileUploader
toFileUploadHandler
because the file is already uploaded. We just need to handle it ;-pComment #26
kim.pepperForgot to update the service definition with the rename in #25
Comment #28
mstrelan CreditAttribution: mstrelan at PreviousNext commentedNeeds a CR and updated URL.
Should this be
throw new SymfonyFileException();
?Comment #29
kim.pepperAdded a CR and updated deprecation links.
Also split out the utility ErrorUtils with static helper methods for converting error codes to exceptions.
Comment #30
kim.pepperRE: #28.2 we are throwing both exceptions.
Comment #31
kim.pepperAs this is blocking the critical #2940383: Unify file upload logic of REST and JSON:API marking this critical too.
Comment #32
mstrelan CreditAttribution: mstrelan at PreviousNext commentedShould the service be named file.upload_handler to match the class name?
RE: #30 I was thrown off my the use statements in file.module, ignore me.
Comment #33
kim.pepper#32 yeah makes sense.
Comment #34
alexpottThere's a subtle behaviour change. In _file_save_upload_single() we set
$file->source = $form_field_name;
. This is no longer happening in the new code. This line has always kinda bothered me and it would be great to remove it. But we need to be sure that contrib is not depending on it for something. I think this was introduced in https://git.drupalcode.org/project/drupal/-/commit/41d2b2a349564c7a55df9... and seems to be used once in contrib - see http://grep.xnddx.ru/search?text=file-%3Esource&filename=Comment #35
BerdirDidn't review yet (sorry), but FWIW, $file is not actually a file entity, that's a raw DB result that it loaded directly from file_managed, so I have no idea where it thinks ->source is coming from.
Doesn't mean that something else isn't using it, but I don't think that one is something we need to be concerned about.
Comment #36
kim.pepperYeah, I should have pointed out that that was removed. I couldn't find any usages of it in core.
Comment #37
alexpott@Berdir yep the thing from contrib in #34 is bogus. You're right - the $file seems to the result of selecting directly from the file_managed table which does not have a source column and did not in D7 either. Odd.
I think we still need to document this on the CR and issue summary.
I have lots of thoughts about translatable messages being used as exception messages. I don't think this is how we should do that. I think we should have a method that can convert an exception into a translatable message but the exception message should not be translatable.
Comment #38
kim.pepperre: #37 I have converted all exception messages to just using a string, and added an addition static method to get a translatable message from the error code.
Also updated the IS and CR to mention removing the call to
$file->source
.Comment #39
kim.pepperFixed test fail due to switch from TranslatableMarkup to string in exceptions.
Comment #40
kim.pepperComment #41
andypostI find it ready to go except 2 nits, sorry(
ubernit - why not fileName)
as it's a file name would be great to have " or ' around it
Comment #42
kim.pepperFixes for #41
Comment #43
andypostThank you, gtg
Comment #44
mstrelan CreditAttribution: mstrelan at PreviousNext commentednit: this should also be FileUploadHandler
Comment #45
dwwI keep meaning to review this (sorry!). This at least fixes nit #44. ;) I hope to dive in more closely tomorrow...
Comment #46
dwwTime's up for today, only a partial review. Mostly looks great so far. 1 question about consistency in user-facing messages, another ubernit, and a note-to-self about something I noticed while reviewing. ;)
Do we want to be consistent about wrapping filenames in " in these messages? Or do we want to keep the strings as they were since in many cases the translations already exist from identical t() strings in
_file_save_upload_single()
?Slick. This is why I didn't initially see some of the other possible error messages in the new method that
_file_save_upload_single()
can generate. Cool.Nit: "Throws".
I'll plan to continue tomorrow...
Thanks/sorry, -Derek
Comment #47
kim.pepperre: #46.1 Trying to keep BC with user facing messages. The exception messages don't get displayed to the client, so I think they are ok to add quotes to.
Comment #48
dwwRe #47: 👍 makes perfect sense. Thanks!
Comment #49
dwwGot through the rest of the patch. This is all great work!
A ton of fiddly docs nitpicks follow. I'll include a patch that addresses all those (and #46.3). The un-addressed points in this list are:
Full dreditor review follows. Again, attached patch addresses all of this except the 4 points above.
More nit: If we're fileName'ing the param, it's because we consider "file" and "name" separate words. So this should be "The original file name". ;)
Technically, "mime" is an acronym and should be all caps.
Doc doesn't match the param.
Do we want to mention this is expected to be a directory, not a final filename?
Might as well use ===, right?
Why " vs ' for this?
I don't understand why:
a) We do this twice.
b) With different logic of the source of truth of the new filename.
$destinationFilename doesn't change between these two calls. I think one of them needs to be removed.
Missing docs.
Should this list live in a class constant somewhere? I see it's now duplicated both here and in the deprecated method. Perhaps in other spots, too.
This is dead code, and now handled computationally via isRenamed().
I thought `return $this` was better for such docs.
Missing doc 1-liners.
Should start with a verb.
For consistency with the rest of this patch, these should all be file name, fileName, etc.
Ugh, except we have a bunch of `getFilename()` methods everywhere else in core, so it's probably not worth changing this to getFileName().
Comment #50
kim.pepperThanks @dww!
Re: #49
4. Its confusing. The
$destination
parameter is a directory.7. I followed the logic of
_file_save_upload_single()
pretty closely as this is something that gets a lot of attention for security reasons. Doesn't make sense to me to do it twice. I think it's safe to remove one.9. Yeah, we can make it a class constant. I don't think we need to change anything in
_file_save_upload_single()
as its deprecated and no longer used anywhere.15. Yeah, I would stick with whatever naming convention we already have in core. :-(
Comment #51
kim.pepperFixes for #49.
4. Updated docbloc to say it's a directory
9. Added a class constant
Comment #52
dwwI promised I'd look into #49.7. Here's what I found:
Looking at git blame for
_file_save_upload_single()
, the first occurrence ofsetFilename()
is from here:While the 2nd is from here:
It seems plausible that since 3032390 took so long to land, we didn't notice that 3032376 had already landed and already got this point right. ;) I think it's safe to remove the duplicate 2nd call now, but I'd love to get @alexpott to confirm if he agrees.
Comment #53
alexpottI think we want the comment from the second with the code of the first :)
Comment #54
alexpottI'm not a huge fan of the ErrorUtils class. I think we can get rid of it and have easier to follow code with less indirection.
I think the messages should be inlined here. I think the indirection doesn't really buy us anything and adds complexity. If you want to handle file uploads with messages - use file_save_upload() and wait until we've provided that as a service.
I think we can change all of this so that we don't have ErrorUtils::throwException()...
I think this could be:
I think indirection when throwing exceptions makes it harder to rationalise about what is going on.
Comment #55
alexpottAll my reviews are picking at the edges of this change. Just to make it clear - I think this change is fantastic and will help maintain form, jsonapi, rest file uploads and not have to duplicate code and have separate bugs. Thanks @kim.pepper for all the effort here.
I think the message assertion should still be in the test. And there's quite a bit of unrelated change that shouldn't be done here either.
Comment #56
larowlan#54 would be even simpler using match, perhaps we could add a D10 followup to move from switch to match in D10 when php8 is required
Comment #57
andypostfiled #3243450: Replace FileUploadHandler:: handleFileUpload() switch with match when core requires PHP 8
Comment #58
kim.pepperLooking at this now.
Comment #59
kim.pepperThanks for the review. This addresses feedback in #53 #54 and #55
Comment #61
kim.pepperFix test fail
Comment #62
dwwI reviewed the interdiffs from #59 and #61. Definitely looks good to me and that all of #53, #54 and #55 are addressed. #53 takes care of #49.7, so that's the last point from #49, too.
I also updated the title/summary for #3243450: Replace FileUploadHandler:: handleFileUpload() switch with match when core requires PHP 8 since the switch moved in #59. ;) Thanks for opening that follow-up, @andypost.
Although I uploaded a few patches for this, they were only to do trivial doc fixes and the like, so I think I'm still qualified to RTBC this. I'll wait for the bot results to come back green, but I think we're there. ;)
Thanks, everyone!
-Derek
Comment #63
mstrelan CreditAttribution: mstrelan at PreviousNext commentedOne minor question:
Would it be better to set this after the try/catch block to avoid setting it to FALSE for every caught exception?
Comment #64
kim.pepper@mstrelan then there wouldn't have been an exception and the file would exist.
Comment #65
mstrelan CreditAttribution: mstrelan at PreviousNext commented$result->getFile() is called twice here. It's the second one I was referring to, which FWIW should just be adding $file to the array.
EDIT: OK that might be intentional, ignore me :)
Comment #66
kim.pepperNo, it think you're right. We can just use
$files[$i] = $file;
here.Comment #67
alexpottThis strikes me as odd. How come we're doing an extra load. Why is this refactoring changing the logic?
Comment #68
kim.pepperWe are loading a file entity and updating it, rather than creating a new one, then updating it. So as I see it, we are replacing a write with a read.
Comment #69
alexpott@kim.pepper I think the new way will result in additional queries to the database. Also I think in general it's not great to change behaviour in a refactor as it increases the chance of unintended impacts. If we do decide to change the behaviour then it needs to be clearly documented in a change record and in the issue summary. My preference would be to change the behaviour in a follow-up.
Comment #70
kim.pepperLooking back at
_file_save_upload_single()
its doing a load too, so not sure why this is triggered now. I will take a look tomorrow.Comment #71
kim.pepperre: #69 I believe this restores the original behaviour.
Comment #72
alexpott@kim.pepper++ looks much nicer now there is no SaveUploadFormTest SaveUploadTest changes.
Comment #73
alexpottThis can be written with way less logic....
Also I think the docblock for this should note the side effect of adding a validator if there is not one and removing the validator if it is set but is empty. Because it is very confusing and awkward. Plus I think the method name should not be getExtensions() but that implies it is a simpler getter when it is anything but. How about handleExtensionValidation() - I'm not sure.
Comment #74
kim.pepperUpdates the logic as per #73.
Agree
handleExtensionValidation()
is a good fit. Updated the docs too.Comment #75
dwwThanks for all the effort in here, folks!
#65 was a good point, fixed in #66.
#71 addresses @alexpott's good points in #67 and #69.
#73 is a great suggested improvement, implemented and documented nicely in #74.
I opened #3243712: Change behavior of FileUploadHandler::handleFileUpload() to update existing entities instead of always creating new ones as a child issue for the possible follow-up to actually change the behavior and document it appropriately. No "Needs followup" needed. 😉 Might not even be worth doing, but we can discuss there.
Bot is happy. I believe everything has been addressed. I don't see room for improvement (although others still might). The draft CR looks good. Per the end of #62, I think it's okay if I RTBC this, so I will. 😀
Thanks everyone! 🙏🎉
-Derek
Comment #76
alexpottCommitted bfe5274 and pushed to 9.3.x. Thanks!
Time to get on with #2940383: Unify file upload logic of REST and JSON:API at long last!
Also we can open a follow-up to call it directly in _file_save_upload_from_form() instead of file_save_upload and then handle all the messages there.
Comment #78
kim.pepperWoohoo! 🎉
Comment #79
Wim Leers💯
Also: wow, thanks everyone for the epic progress here!
Comment #80
quietone CreditAttribution: quietone as a volunteer commentedSorry folks.
While working on #92944: Display generic message and log detailed message when file upload fails due to PHP error I read the change record for this issue and noticed that it refers to method
createFromUpload
which does not exist. Also, should the change record mention the various file exceptions that might be thrown?Comment #81
alexpottI've fixed the CR and expanded on why custom/contrib should not actually use the new service.
@quietone tbh I'm not sure that we really should have change records for changing a method that is marked as internal. Also I don't think we need to document the exceptions on the CR. They are documented on the method and a CR that repeats the documentation is not that useful. A CR is to tell people how to update their code to use the new functionality. People should not have been calling _file_save_upload_single() directly and the are not really many use-cases for calling the new service outside of core. 99.9% of the time contrib/custom should be using the managed file api and form elements and not this.
Comment #82
kim.pepperAdded a follow up to this issue #3245244: Make FileUploadHandler re-usuable for non-form file uploads
Comment #84
quietone CreditAttribution: quietone as a volunteer commentedRemoving tag per #81.