Problem/Motivation

Discovered while working on #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method.

First there was \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber, and JSON error responses were brittle. Introduced in #2323759: Modularize kernel exception handling.
Then there was \Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber, and JSON error responses were even more brittle. Introduced in #2308745: Remove rest.settings.yml, use rest_resource config entities.

\Drupal\Core\EventSubscriber\ExceptionJsonSubscriber handles:
  • 400
  • 403
  • 404
  • 405
  • 406
  • 415
\Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber handles:
    400
  • 403
  • 404
  • 405
  • 406
  • 422
  • 429

The concrete problem that this causes: for consistency with other JSON error responses, you would expect this:

{"message":"Unprocessable Entity: validation failed.\ntitle: \u003Cem class=\u0022placeholder\u0022\u003ETitle\u003C\/em\u003E: this field cannot hold more than 1 values.\n"}

But instead you get:

{"message":"Unprocessable Entity: validation failed.\ntitle: <em class=\"placeholder\">Title<\/em>: this field cannot hold more than 1 values.\n"}

Root cause analysis

\Drupal\Core\EventSubscriber\ExceptionJsonSubscriber runs first, and whatever it doesn't handle, is handled by \Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber. So, 400/403/404/405/406 is supported by both, but always handled by the first. 415 is handled by the first. 422 and 429 are handled by the latter.

Besides being very inefficient and confusing, this seems harmless. Except that \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber uses \Symfony\Component\HttpFoundation\JsonResponse which uses these encoding options: JSON_HEX_TAG | JSON_HEX_APOS | JSON_HEX_AMP | JSON_HEX_QUOT. Because: Encode <, >, ', &, and " for RFC4627-compliant JSON, which may also be embedded into HTML..

But then there's \Drupal\serialization\EventSubscriber\DefaultExceptionSubscriber, which uses \Symfony\Component\Serializer\Serializer, which calls \Drupal\serialization\Encoder\JsonEncoderwhich extends \Symfony\Component\Serializer\Encoder\JsonEncoder, which calls \Symfony\Component\Serializer\Encoder\JsonEncode which … does not have any encoding options set by default. (This can be overridden via the encoding options: $context['json_encode_options'].)

The end result: our JSON error responses are encoded inconsistently.

Proposed resolution

Make this consistent.

Possible solutions:

  1. Remove \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber. But this would still mean JSON responses are encoded inconsistently.
  2. Let \Drupal\serialization\Encoder\JsonEncoder override the constructor of \Symfony\Component\Serializer\Encoder\JsonEncoder so it constructs a \Symfony\Component\Serializer\Encoder\JsonEncode instance with the same default JSON encoding options as JsonResponse.

Optionally: consider also removing \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber. It makes things unnecessarily complex/confusing/brittle. — this is the scope of #2805281: ?_format=hal_json error responses are application/json, yet should be application/hal+json to fix/change, and that file cannot simply be removed, it will need other supporting changes too.

Remaining tasks

  1. Patch
  2. Agree on approach
  3. Review

User interface changes

None.

API changes

None.

But there is a behavior change: all JSON responses will now be encoded with the same encoding options. In other words, ensure this is true for all JSON responses in Drupal: Encode <, >, ', &, and " for RFC4627-compliant JSON, which may also be embedded into HTML..

Therefore this should only be committed to 8.3.x.

Data model changes

None.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers created an issue. See original summary.

Wim Leers’s picture

Wim Leers’s picture

Assigned: Wim Leers » Unassigned
Status: Active » Needs review
Issue tags: +Needs tests
FileSize
1.43 KB

This will need test coverage, but first let's see whether this breaks any existing tests.

Wim Leers’s picture

Title: JSON error responses encoded inconsistently: make them all RFC4627-compliant » JSON responses encoded inconsistently: make them all RFC4627-compliant

This problem is actually not even limited to JSON error responses. JSON responses served by Symfony's JsonResponse are encoded differently than JSON responses served by Symfony's JSON Serializer. That makes no sense.

Wim Leers’s picture

Issue summary: View changes
dawehner’s picture

+++ b/core/modules/serialization/src/Encoder/JsonEncoder.php
@@ -18,6 +20,18 @@ class JsonEncoder extends BaseJsonEncoder implements EncoderInterface, DecoderIn
+    // 15 === JSON_HEX_TAG | JSON_HEX_APOS | JSON_HEX_AMP | JSON_HEX_QUOT
...
+    $json_encoding_options = 15;

Is there a reason we don't just inline this line? This would be more readable, IMHO

Wim Leers’s picture

#6: I copy/pasted it verbatim from Symfony's JsonResponse. Happy to change that though.

Wim Leers’s picture

Issue summary: View changes

Status: Needs review » Needs work

The last submitted patch, 3: 2813755-3.patch, failed testing.

The last submitted patch, 3: 2813755-3.patch, failed testing.

Wim Leers’s picture

There we are, three fails: three cases where we're effectively (yet unintentionally) asserting the current JSON encoding.

andypost’s picture

Does it makes sense to backport to SF?
I mean to change JsonResponse default serialization

dawehner’s picture

One question though I'd ask in general, shouldn't we strip of HTML out of those errors messages? For me those don't make sense, beside making it harder to read.

\Drupal\Component\Render\PlainTextOutput::renderFromHtml or a potential different implementation, could be used here.

Wim Leers’s picture

One question though I'd ask in general, shouldn't we strip of HTML out of those errors messages? For me those don't make sense, beside making it harder to read.

+1000

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
1.46 KB
1.21 KB

Addressed #6.

Status: Needs review » Needs work

The last submitted patch, 15: 2813755-15.patch, failed testing.

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
2.52 KB
1.09 KB
Wim Leers’s picture

Issue summary: View changes

Fix incomplete sentence in IS.

Wim Leers’s picture

So, let's start to add test coverage. #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method introduced all the test coverage we need. So why is the patch green? Because it still includes the work-arounds I put in place, the work-arounds that this issue must remove.

So let's see what happens when we remove the most important work-around.

Wim Leers’s picture

#3 fixed things for the json format (and hence all JSON tests pass), but not yet for the hal_json format (and hence all HAL+JSON tests fail), because:

  serializer.encoder.hal:
    class: Drupal\hal\Encoder\JsonEncoder
    tags:
      - { name: encoder, priority: 10, format: hal_json }

and:

use Symfony\Component\Serializer\Encoder\JsonEncoder as SymfonyJsonEncoder;

class JsonEncoder extends SymfonyJsonEncoder {
…

… which does not reuse \Drupal\serialization\Encoder\JsonEncoder, but extends Symfony\Component\Serializer\Encoder\JsonEncoder (\Drupal\serialization\Encoder\JsonEncoder does this too). Why not let HAL module's JSON encoder reuse the Serialization module's JSON encoder, so that it can benefit from the fix made in #3, without needing to duplicate that logic?

That's what this reroll does.

Wim Leers’s picture

While awaiting the results for #19 + #20, here's a reroll that removes all work-arounds referencing this issue. Let's see what the test results are for that.

The last submitted patch, 19: 2813755-19.patch, failed testing.

The last submitted patch, 21: 2813755-21.patch, failed testing.

The last submitted patch, 19: 2813755-19.patch, failed testing.

Wim Leers’s picture

FileSize
9.53 KB
783 bytes
Wim Leers’s picture

From the IS:

Optionally: consider also removing \Drupal\Core\EventSubscriber\ExceptionJsonSubscriber. It makes things unnecessarily complex/confusing/brittle.

This would indeed be great to do, but while working on this, I noticed that that pushes this patch to the scope of #2805281: ?_format=hal_json error responses are application/json, yet should be application/hal+json, and would require lots of surrounding changes to make it work. So, indicating that in the IS.

Wim Leers’s picture

FileSize
9.71 KB
960 bytes

Fixing nits.

The last submitted patch, 25: 2813755-25.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 27: 2813755-27.patch, failed testing.

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
4.28 KB
13.94 KB

#21 did not yet remove all work-arounds. This removes the last few.

tedbow’s picture

Status: Needs review » Needs work
+++ b/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php
@@ -577,11 +573,7 @@ public function testPost() {
+    $this->assertResourceErrorResponse(422, "Unprocessable Entity: validation failed.\nuuid.0.value: <em class=\"placeholder\">UUID</em>: may not be longer than 128 characters.\n", $response);

+++ b/core/modules/rest/tests/src/Functional/EntityResource/User/UserResourceTestBase.php
@@ -163,11 +163,7 @@ public function testPatchDxForSecuritySensitiveBaseFields() {
+    $this->assertResourceErrorResponse(422, "Unprocessable Entity: validation failed.\nmail: Your current password is missing or incorrect; it's required to change the <em class=\"placeholder\">Email</em>.\n", $response);

Re #13 and #14 are we going strip the html out of these messages? Maybe that is separate issue?

Other than I look back through the issue and it looks good.

dawehner’s picture

+++ b/core/modules/rest/tests/src/Unit/EventSubscriber/ResourceResponseSubscriberTest.php
@@ -48,7 +48,7 @@ public function testSerialization($data, $expected_response = FALSE) {
-    $this->assertEquals($expected_response !== FALSE ? $expected_response : json_encode($data), $event->getResponse()->getContent());
+    $this->assertEquals($expected_response !== FALSE ? $expected_response : json_encode($data, JSON_HEX_TAG | JSON_HEX_APOS | JSON_HEX_AMP | JSON_HEX_QUOT), $event->getResponse()->getContent());

This line could use \Drupal\Component\Serialization\Json::encode directly. I'm not sure its a good idea, but its the same code

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
1.09 KB
13.88 KB

#31: That's IMO out of scope. This issue is solely about the JSON encoding. The clean-up of the message therein can be done separately.

#32: Great point! Can't believe I didn't do that the first time.

dawehner’s picture

I'm a bit confused by this change. Isn't it just a special case of #2739617: Make it easier to write on4xx() exception subscribers basically?

Wim Leers’s picture

#34: if we omitted this from the patch:

+++ b/core/lib/Drupal/Core/EventSubscriber/ExceptionJsonSubscriber.php
@@ -93,4 +93,15 @@ public function on415(GetResponseForExceptionEvent $event) {
+  public function on422(GetResponseForExceptionEvent $event) {
+    $response = new JsonResponse(['message' => $event->getException()->getMessage()], Response::HTTP_UNPROCESSABLE_ENTITY);
+    $event->setResponse($response);
+  }
+

then there would be no on4XX()-related changes. The root problem is the encoding, not the handling of on4XX(). It just happens to be the case that all examples here are using 422 responses. So, it just that it's an unfortunate but necessary indirectly related change.

I can easily prove that the root problem is encoding. This patch is identical to #33, it just comments the JSON encoding-related changes.


Or perhaps you'd prefer #2805281: ?_format=hal_json error responses are application/json, yet should be application/hal+json to be done first?

Status: Needs review » Needs work

The last submitted patch, 35: 2813755-33-FAIL-without-encoding-changes.patch, failed testing.

Wim Leers’s picture

Status: Needs work » Needs review

I hope #35 proved my point sufficiently convincingly.

tedbow’s picture

Status: Needs review » Reviewed & tested by the community

I think #35 demonstrates it is an encoding issue and seems it is important to get this in so clients have consistent encoding on errors.

Regarding stripping html as out of scope that seems right was checking we hadn't forgot something.

Looks good to me.

Wim Leers’s picture

Opened #2835683: Remove HTML from EntityResource validation 422 exception message for the removal of that HTML in the 422 responses for failed entity validation.

Wim Leers’s picture

damiankloip’s picture

Also agree this looks good!

  1. +++ b/core/modules/hal/src/Encoder/JsonEncoder.php
    @@ -2,34 +2,20 @@
    -  public function supportsEncoding($format) {
    

    Nice, I noticed these were potentially redundant the other day (i.e. we could extend our own encoder like you are here).

  2. +++ b/core/modules/serialization/src/Encoder/JsonEncoder.php
    @@ -21,6 +23,18 @@ class JsonEncoder extends BaseJsonEncoder implements EncoderInterface, DecoderIn
    +  public function __construct(JsonEncode $encodingImpl = null, JsonDecode $decodingImpl = null) {
    

    This looks good to me, and probably the simplest way. We could have injected a new argument for the JsonEncode class instead with these parameters. I don't think it buys us much/anything though.

Wim Leers’s picture

  1. Yay :)
  2. Yep, exactly — it'd have required setting up yet another service. Wouldn't buy us anything.

Thanks for your review!

alexpott’s picture

+++ b/core/modules/serialization/src/Encoder/JsonEncoder.php
@@ -21,6 +23,18 @@ class JsonEncoder extends BaseJsonEncoder implements EncoderInterface, DecoderIn
+  public function __construct(JsonEncode $encodingImpl = null, JsonDecode $decodingImpl = null) {
+    // Encode <, >, ', &, and " for RFC4627-compliant JSON, which may also be
+    // embedded into HTML.
+    // @see \Symfony\Component\HttpFoundation\JsonResponse
+    $json_encoding_options = JSON_HEX_TAG | JSON_HEX_APOS | JSON_HEX_AMP | JSON_HEX_QUOT;
+    $this->encodingImpl = $encodingImpl ?: new JsonEncode($json_encoding_options);
+    $this->decodingImpl = $decodingImpl ?: new JsonDecode(true);
+  }

I've been mulling the behaviour change for a while trying to think if there are any possible BC impacts. I can't think of any. Am I right? Anything that decodes the JSON encoded with these options is just going to see the same data - only the JSON will be slightly different right?

Wim Leers’s picture

That is correct.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed badbcd3 and pushed to 8.3.x. Thanks!

FILE: ...ev/drupal/core/modules/serialization/src/Encoder/JsonEncoder.php
----------------------------------------------------------------------
FOUND 3 ERRORS AFFECTING 2 LINES
----------------------------------------------------------------------
 26 | ERROR | [x] TRUE, FALSE and NULL must be uppercase; expected
    |       |     "NULL" but found "null"
 26 | ERROR | [x] TRUE, FALSE and NULL must be uppercase; expected
    |       |     "NULL" but found "null"
 32 | ERROR | [x] TRUE, FALSE and NULL must be uppercase; expected
    |       |     "TRUE" but found "true"
----------------------------------------------------------------------
PHPCBF CAN FIX THE 3 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------

Fixed on commit.

  • alexpott committed badbcd3 on 8.3.x
    Issue #2813755 by Wim Leers, dawehner, tedbow, damiankloip: JSON...
alexpott’s picture

I concerned whether or not this was worth a change record but I really don't think so since all JSON parsers should support this encoding.

Wim Leers’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.