Problem/Motivation

POSTing files to file fields that write to the root directory of the public file (Leave the "File Directory" setting of the field setting blank) can lead to a 422 Unprocessable Entity response, with no more details describe what's going on.

The exception is: (The full call stack is in attachment)
Symfony\\Component\\HttpKernel\\Exception\\UnprocessableEntityHttpException: Unprocessable Entity: file validation failed. This value should be of the correct primitive type.
We won't get any more detailed error messages like "Only files with the following extensions are allowed".

While uploading to the root directory is allowed in Drupal itself theoretically, and upload with the "Add content" web interface will work with no problems, I think this is a bug that comes from an untested situation of JSON:API. It's also confusing because once encountered, it will not give any useful information before you realize the problem comes from "where the file should be uploaded".

Steps to reproduce

  • Create a content type and a file field inside it, set allowed file extensions.
  • Fire a POST request to the API:
    curl http://192.168.0.134:8080/jsonapi/node/test_type/field_file \
       -H 'Accept: application/vnd.api+json' \
       -H 'Content-Type: application/octet-stream' \
       -H 'Content-Disposition: file; filename="test.jpg"' \
       -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
       --data-binary ~/Desktop/0.jpg

    (Works with no problems)

  • Remove the default [date:custom:Y]-[date:custom:m] from the field's "File directory" setting, and leave it blank.
  • Fire the same POST request again. This time we will get the error above.

Proposed resolution

TBD

Remaining tasks

TBD

User interface changes

TBD

API changes

TBD

Data model changes

TBD

Release notes snippet

TBD

Issue fork drupal-3214675

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

hehongbo created an issue. See original summary.

hehongbo’s picture

Version: 9.2.x-dev » 9.3.x-dev
hehongbo’s picture

Tested with 9.3.x-dev 8742a3a and got same response

el7cosmos’s picture

Assigned: Unassigned » el7cosmos

el7cosmos’s picture

This happens because following line in core/modules/jsonapi/src/Controller/TemporaryJsonapiFileFieldUploader.php

$file_uri = "{$destination}/{$prepared_filename}";

On public root directory, this value will be public:///test.jpg which parse_url will return FALSE, and will give violations when validated.

I created an MR that remove that leading slash on public root directory so it will be public://test.jpg.

el7cosmos’s picture

Assigned: el7cosmos » Unassigned
Status: Active » Needs review
bbrala’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Wouldn't a ltrim('/') make more sense? Or perhaps implode('/', [$destination, $prepared_filesname]);?

Also this needs a test.

el7cosmos’s picture

I don't think ltrim or implode is neccessary, because the slash only become a problem for root directory only.

Another approach to avoid parse_url that I can think of is to move the responsibility of handling the slash to getUploadLocation, so in handleFileUploadForField can be like just:

    $file_uri = "{$destination}{$prepared_filename}";
el7cosmos’s picture

Status: Needs work » Needs review
bbrala’s picture

Status: Needs review » Needs work

Thanks for all the work you are putting in :)

How about checking if the destination is empty since that is the exact usecase we are fixing here? I really dislike the logic in this code right now:

$file_uri = parse_url($destination) ? "{$destination}/{$prepared_filename}" : "{$destination}{$prepared_filename}";

This reads like: "If the destination cannot be parsed as a destination for whatever reason, just append it in from of the filename". This feels like its way to open to bugsa where the destination or something that parse_url doesn't like.

Perhaps an explicit check on this usecase where destination is empty is best.

el7cosmos’s picture

No, the problem isn't an empty $destination, destination will never be empty. There are two possible cases:
1. $destination will have value like: public://subdirectory, and the $file_uri will became public://subdirectory/file.ext which is valid.
2. $destination will have a value: public://, and the $file_uri will became public:///file.ext which is invalid, and here is the problem

bbrala’s picture

Status: Needs work » Reviewed & tested by the community
Issue tags: -json:api, -Needs tests

You are correct, sorry, i'm going to get me some coffee!

This is looking good, thanks for you patience :)

el7cosmos’s picture

Status: Reviewed & tested by the community » Needs work

No problem :)

bbrala’s picture

Status: Needs work » Reviewed & tested by the community

Think that statuschange was unintended ;)

larowlan’s picture

Status: Reviewed & tested by the community » Needs work

I have similar misgivings as bbrala about the use of parse_url, it feels a bit risky.

Left a suggestion on the MR

el7cosmos’s picture

Status: Needs work » Needs review

Changed as @larowlan suggestion. I think that's much better

bbrala’s picture

Status: Needs review » Reviewed & tested by the community

Thanks @el7cosmos, that looks way better.

larowlan’s picture

Status: Reviewed & tested by the community » Needs review

Left one question on the MR

bbrala’s picture

When you look at the tests in that file the rebuild is done every time a setting is changed, so he just followed the code as presented in the tests. I would also assume that it is needed to rebuild then.

bbrala’s picture

Just to see what happens i've removed the rebuild routes from the test. I still think though that since testFileUploadNoExtensionSetting also does this it might be needed. We'll see :)

If this test does not fail, we should make a followup to check if cleanup of the rebuild routes call is possible to optimize the testload a little more.

bbrala’s picture

Is seems you are right. I'll push the minor change and set to RTBC again so you can review @larowlan. I'll also created a follow up issue #3236255: Remove rebuildAll from FileUploadtest where possible.

bbrala’s picture

Status: Needs review » Reviewed & tested by the community
alexpott’s picture

Status: Reviewed & tested by the community » Needs work
el7cosmos’s picture

@alexpott appending trailing slash for $destination will bring us back again to the bug, if the $destination is root directory: public:///file.ext will be invalid, while the valid one should be public://file.ext

larowlan’s picture

@el7cosmos no alex is asking for the converse. I think it will work - if there's no trailing slash add it - but if there is, don't.

That should prevent the /// issue

el7cosmos’s picture

@larowlan yeah, sorry I missed that part. It will work

el7cosmos’s picture

Status: Needs work » Needs review
bbrala’s picture

Status: Needs review » Reviewed & tested by the community

Changes are as requested by @alexpott. Thanks for the help. :)

alexpott’s picture

Version: 9.3.x-dev » 9.2.x-dev
Status: Reviewed & tested by the community » Fixed

Committed and pushed a8d3f780cc to 9.3.x and 7d047b209a to 9.2.x. Thanks!

Backported to 9.2.x as this is a low risk bug fix.

  • alexpott committed a8d3f78 on 9.3.x
    Issue #3214675 by el7cosmos, bbrala, hehongbo, larowlan, alexpott: JSON:...

  • alexpott committed 7d047b2 on 9.2.x
    Issue #3214675 by el7cosmos, bbrala, hehongbo, larowlan, alexpott: JSON:...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.