Problem/Motivation
Discovered while working on #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method.
First there was \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber
, and JSON error responses were brittle. Introduced in #2323759: Modularize kernel exception handling.
Then there was \Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber
, and JSON error responses were even more brittle. Introduced in #2308745: Remove rest.settings.yml, use rest_resource config entities.
\Drupal\Core\EventSubscriber\ExceptionJsonSubscriber
handles:-
- 400
- 403
- 404
- 405
- 406
- 415
\Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber
handles:-
- 400
- 403
- 404
- 405
- 406
- 422
- 429
The concrete problem that this causes: for consistency with other JSON error responses, you would expect this:
{"message":"Unprocessable Entity: validation failed.\ntitle: \u003Cem class=\u0022placeholder\u0022\u003ETitle\u003C\/em\u003E: this field cannot hold more than 1 values.\n"}
But instead you get:
{"message":"Unprocessable Entity: validation failed.\ntitle: <em class=\"placeholder\">Title<\/em>: this field cannot hold more than 1 values.\n"}
Root cause analysis
\Drupal\Core\EventSubscriber\ExceptionJsonSubscriber
runs first, and whatever it doesn't handle, is handled by \Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber
. So, 400/403/404/405/406 is supported by both, but always handled by the first. 415 is handled by the first. 422 and 429 are handled by the latter.
Besides being very inefficient and confusing, this seems harmless. Except that \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber
uses \Symfony\Component\HttpFoundation\JsonResponse
which uses these encoding options: JSON_HEX_TAG | JSON_HEX_APOS | JSON_HEX_AMP | JSON_HEX_QUOT
. Because: Encode <, >, ', &, and " for RFC4627-compliant JSON, which may also be embedded into HTML.
.
But then there's \Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber
, which uses \Symfony\Component\Serializer\Serializer
, which calls \Drupal\serialization\Encoder\JsonEncoder
which extends \Symfony\Component\Serializer\Encoder\JsonEncoder
, which calls \Symfony\Component\Serializer\Encoder\JsonEncode
which … does not have any encoding options set by default. (This can be overridden via the encoding options: $context['json_encode_options']
.)
The end result: our JSON error responses are encoded inconsistently.
Proposed resolution
Make this consistent.
Possible solutions:
Remove\Drupal\Core\EventSubscriber\ExceptionJsonSubscriber
. But this would still mean JSON responses are encoded inconsistently.- Let
\Drupal\serialization\Encoder\JsonEncoder
override the constructor of\Symfony\Component\Serializer\Encoder\JsonEncoder
so it constructs a\Symfony\Component\Serializer\Encoder\JsonEncode
instance with the same default JSON encoding options asJsonResponse
.
Optionally: consider also removing — this is the scope of #2805281: ?_format=hal_json error responses are application/json, yet should be application/hal+json to fix/change, and that file cannot simply be removed, it will need other supporting changes too.\Drupal\Core\EventSubscriber\ExceptionJsonSubscriber
. It makes things unnecessarily complex/confusing/brittle.
Remaining tasks
- Patch
- Agree on approach
- Review
User interface changes
None.
API changes
None.
But there is a behavior change: all JSON responses will now be encoded with the same encoding options. In other words, ensure this is true for all JSON responses in Drupal: Encode <, >, ', &, and " for RFC4627-compliant JSON, which may also be embedded into HTML.
.
Therefore this should only be committed to 8.3.x.
Data model changes
None.
Comment | File | Size | Author |
---|---|---|---|
#35 | 2813755-33-FAIL-without-encoding-changes.patch | 13.89 KB | Wim Leers |
#33 | 2813755-33.patch | 13.88 KB | Wim Leers |
#21 | 2813755-21.patch | 8.81 KB | Wim Leers |
Comments
Comment #2
Wim LeersComment #3
Wim LeersThis will need test coverage, but first let's see whether this breaks any existing tests.
Comment #4
Wim LeersThis problem is actually not even limited to JSON error responses. JSON responses served by Symfony's
JsonResponse
are encoded differently than JSON responses served by Symfony's JSON Serializer. That makes no sense.Comment #5
Wim LeersComment #6
dawehnerIs there a reason we don't just inline this line? This would be more readable, IMHO
Comment #7
Wim Leers#6: I copy/pasted it verbatim from Symfony's
JsonResponse
. Happy to change that though.Comment #8
Wim LeersComment #11
Wim LeersThere we are, three fails: three cases where we're effectively (yet unintentionally) asserting the current JSON encoding.
Comment #12
andypostDoes it makes sense to backport to SF?
I mean to change
JsonResponse
default serializationComment #13
dawehnerOne question though I'd ask in general, shouldn't we strip of HTML out of those errors messages? For me those don't make sense, beside making it harder to read.
\Drupal\Component\Render\PlainTextOutput::renderFromHtml
or a potential different implementation, could be used here.Comment #14
Wim Leers+1000
Comment #15
Wim LeersAddressed #6.
Comment #17
Wim LeersComment #18
Wim LeersFix incomplete sentence in IS.
Comment #19
Wim LeersSo, let's start to add test coverage. #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method introduced all the test coverage we need. So why is the patch green? Because it still includes the work-arounds I put in place, the work-arounds that this issue must remove.
So let's see what happens when we remove the most important work-around.
Comment #20
Wim Leers#3 fixed things for the
json
format (and hence all JSON tests pass), but not yet for thehal_json
format (and hence all HAL+JSON tests fail), because:and:
… which does not reuse
\Drupal\serialization\Encoder\JsonEncoder
, but extendsSymfony\Component\Serializer\Encoder\JsonEncoder
(\Drupal\serialization\Encoder\JsonEncoder
does this too). Why not let HAL module's JSON encoder reuse the Serialization module's JSON encoder, so that it can benefit from the fix made in #3, without needing to duplicate that logic?That's what this reroll does.
Comment #21
Wim LeersWhile awaiting the results for #19 + #20, here's a reroll that removes all work-arounds referencing this issue. Let's see what the test results are for that.
Comment #25
Wim LeersComment #26
Wim LeersFrom the IS:
This would indeed be great to do, but while working on this, I noticed that that pushes this patch to the scope of #2805281: ?_format=hal_json error responses are application/json, yet should be application/hal+json, and would require lots of surrounding changes to make it work. So, indicating that in the IS.
Comment #27
Wim LeersFixing nits.
Comment #30
Wim Leers#21 did not yet remove all work-arounds. This removes the last few.
Comment #31
tedbowRe #13 and #14 are we going strip the html out of these messages? Maybe that is separate issue?
Other than I look back through the issue and it looks good.
Comment #32
dawehnerThis line could use
\Drupal\Component\Serialization\Json::encode
directly. I'm not sure its a good idea, but its the same codeComment #33
Wim Leers#31: That's IMO out of scope. This issue is solely about the JSON encoding. The clean-up of the message therein can be done separately.
#32: Great point! Can't believe I didn't do that the first time.
Comment #34
dawehnerI'm a bit confused by this change. Isn't it just a special case of #2739617: Make it easier to write on4xx() exception subscribers basically?
Comment #35
Wim Leers#34: if we omitted this from the patch:
then there would be no
on4XX()
-related changes. The root problem is the encoding, not the handling ofon4XX()
. It just happens to be the case that all examples here are using 422 responses. So, it just that it's an unfortunate but necessary indirectly related change.I can easily prove that the root problem is encoding. This patch is identical to #33, it just comments the JSON encoding-related changes.
Or perhaps you'd prefer #2805281: ?_format=hal_json error responses are application/json, yet should be application/hal+json to be done first?
Comment #37
Wim LeersI hope #35 proved my point sufficiently convincingly.
Comment #38
tedbowI think #35 demonstrates it is an encoding issue and seems it is important to get this in so clients have consistent encoding on errors.
Regarding stripping html as out of scope that seems right was checking we hadn't forgot something.
Looks good to me.
Comment #39
Wim LeersOpened #2835683: Remove HTML from EntityResource validation 422 exception message for the removal of that HTML in the 422 responses for failed entity validation.
Comment #40
Wim LeersTurns out #2739617 is blocked on this issue, see #2739617-74: Make it easier to write on4xx() exception subscribers. This is now hard-blocking fixes for REST DX WTFs uncovered by #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method.
Comment #41
damiankloip CreditAttribution: damiankloip at Acquia commentedAlso agree this looks good!
Nice, I noticed these were potentially redundant the other day (i.e. we could extend our own encoder like you are here).
This looks good to me, and probably the simplest way. We could have injected a new argument for the JsonEncode class instead with these parameters. I don't think it buys us much/anything though.
Comment #42
Wim LeersThanks for your review!
Comment #43
alexpottI've been mulling the behaviour change for a while trying to think if there are any possible BC impacts. I can't think of any. Am I right? Anything that decodes the JSON encoded with these options is just going to see the same data - only the JSON will be slightly different right?
Comment #44
Wim LeersThat is correct.
Comment #45
alexpottCommitted badbcd3 and pushed to 8.3.x. Thanks!
Fixed on commit.
Comment #47
alexpottI concerned whether or not this was worth a change record but I really don't think so since all JSON parsers should support this encoding.
Comment #48
Wim LeersYep, exactly!
And yay, this unblocked #2739617: Make it easier to write on4xx() exception subscribers!