Problem/Motivation

Since #2920001: Add cacheable HTTP exceptions: Symfony HTTP exceptions + Drupal cacheability metadata, it is possible to make 4xx responses cacheable. This is valuable for e.g. many GET requests to the same JSON API URL with e.g. incorrect filters. Imagine tens of thousands of Rokus or Apple TVs making the same invalid request. Currently, those would all cause Drupal to bootstrap and fully handle the response. Even though Dynamic Page Cache (or even Page Cache in case of anonymous requests) could handle those much faster, leading to better scalability.

#2765959: Make 4xx REST responses cacheable by (Dynamic) Page Cache + comprehensive cacheability test coverage did this for the core REST module.

Proposed resolution

  1. Wait to do this until JSON API requires Drupal 8.5 or later, because #2920001: Add cacheable HTTP exceptions: Symfony HTTP exceptions + Drupal cacheability metadata was only added in Drupal 8.5.
  2. Inspect the JSON API codebase and convert where this makes sense.
  3. Add test coverage.

Remaining tasks

TBD

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

Wim Leers created an issue. See original summary.

wim leers’s picture

Category: Bug report » Task
StatusFileSize
new4.43 KB

Attached is a patch that doesn't work (it's not yet passing the necessary cacheability metadata), but these are examples of places where I think JSON API would benefit. It's not everything for sure.

wim leers’s picture

Title: Convert "throw new *HttpException" into "throw new Cacheable*HttpException" where possible » [>=8.5] Convert "throw new *HttpException" into "throw new Cacheable*HttpException" where possible
Version: 8.x-1.x-dev » 8.x-2.x-dev

Wait to do this until JSON API requires Drupal 8.5

That makes this a v2-only issue.

wim leers’s picture

Status: Postponed » Active
axle_foley00’s picture

Hi @wim-leers, I'll try to tackle this.

wim leers’s picture

👍 Great, thank you! :)

gabesullice’s picture

wim leers’s picture

axle_foley00’s picture

StatusFileSize
new39.4 KB

Attached is my first pass at converting references to "throw new *HttpException" into "throw new Cacheable*HttpException" wherever possible.

wim leers’s picture

Status: Active » Needs review
wim leers’s picture

Just got back from vacation — you posted that the day after I left for vacation. Queued a test! Thanks so much for working on this!

Status: Needs review » Needs work

The last submitted patch, 9: 2929428-9.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new1.4 KB
new39.17 KB
+++ b/src/EventSubscriber/DefaultExceptionSubscriber.php
@@ -36,8 +36,8 @@ class DefaultExceptionSubscriber extends SerializationDefaultExceptionSubscriber
-    if (($exception = $event->getException()) && !$exception instanceof HttpException) {
-      $exception = new HttpException(500, $exception->getMessage(), $exception);
+    if (($exception = $event->getException()) && !$exception instanceof CacheableHttpException) {
+      $exception = new CacheableHttpException(500, $exception->getMessage(), $exception);

This is probably the only place that you cannot convert to be a cacheable HTTP exception!

Reverting this change.

Status: Needs review » Needs work

The last submitted patch, 13: 2929428-13.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new3.9 KB
new35.35 KB
+++ b/src/Normalizer/HttpExceptionNormalizer.php
@@ -25,7 +25,7 @@ class HttpExceptionNormalizer extends NormalizerBase {
    */
-  protected $supportedInterfaceOrClass = HttpException::class;
+  protected $supportedInterfaceOrClass = CacheableHttpException::class;
 
   /**
    * The current user making the request.
@@ -67,13 +67,13 @@ class HttpExceptionNormalizer extends NormalizerBase {

@@ -67,13 +67,13 @@ class HttpExceptionNormalizer extends NormalizerBase {
   /**
    * Builds the normalized JSON API error objects for the response.
    *
-   * @param \Symfony\Component\HttpKernel\Exception\HttpException $exception
+   * @param \Drupal\Core\Http\Exception\CacheableHttpException $exception
    *   The Exception.
    *
    * @return array
    *   The error objects to include in the response.
    */
-  protected function buildErrorObjects(HttpException $exception) {
+  protected function buildErrorObjects(CacheableHttpException $exception) {

Actually, this is another place. Reverting these changes too (plus those in its subclasses).

wim leers’s picture

StatusFileSize
new1.16 KB
new35.54 KB

Here is an example of a concrete change. In this particular instance, we're throwing an exception due to URL query args that are invalid. So this exception depends on the url.query_args cache context. (See https://www.drupal.org/docs/8/api/cache-api/cache-contexts.)

This makes WorkflowTest pass! So we should see a lower number of test failures now :)

Back to you now!

The last submitted patch, 15: 2929428-14.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 16: 2929428-16.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new5.28 KB
new30.55 KB

From >100 to 35! 🎉

I'll do just one more. TermTest::testRelationships() is failing, and the fatal error points to RequestHandler.php on line 137. On that line, cacheability is missing, just like in #16. But in this particular case, a cacheable exception does not even make sense, because it only ever occurs for POST and PATCH requests, and responses for those HTTP methods are never cacheable anyway, because the HTTP spec forbids anybody from caching responses to such requests!
So in this case, we simply can make the exception not cacheable.

Remember, the issue summary says: Inspect the JSON API codebase and convert where this makes sense. — this is a case where it doesn't make sense.

After fixing that, a similar problem comes up in \Drupal\jsonapi\Normalizer\RelationshipNormalizer::massageRelationshipInput(), that code is also only called during denormalization and hence during unsafe requests.

Now, truly, back to you!

Status: Needs review » Needs work

The last submitted patch, 19: 2929428-19.patch, failed testing. View results

wim leers’s picture

Assigned: Unassigned » wim leers
Status: Needs work » Needs review
StatusFileSize
new30.84 KB

Taking this on.

First step: rebase, because #19 no longer applied.

wim leers’s picture

StatusFileSize
new818 bytes
new31.6 KB

Recent changes (a.o. #2949807: Spec Compliance: Error responses are missing the `jsonapi` top-level member) have made error responses cacheable (yay!), and the changes here result in more accurate cacheability (more yay!).

This demands some assertion updates. This interdiff for example makes ::getIndividual() pass tests.

The last submitted patch, 21: 2929428-21.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 22: 2929428-22.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new12.1 KB
new22.4 KB

This removes most if not all "cacheable" exception conversions from unsafe HTTP method controllers in EntityResource. This should fix many failures!

Status: Needs review » Needs work

The last submitted patch, 25: 2929428-25.patch, failed testing. View results

wim leers’s picture

Assigned: wim leers » Unassigned
Status: Needs work » Needs review
StatusFileSize
new8.26 KB
new20.32 KB

Green? :)

wim leers’s picture

StatusFileSize
new25.9 KB

Oops.

The last submitted patch, 27: 2929428-27.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 28: 2929428-27.patch, failed testing. View results

wim leers’s picture

Assigned: Unassigned » wim leers
Status: Needs work » Needs review
StatusFileSize
new16.9 KB
new23.7 KB

Apparently there's quite a bit more here.

wim leers’s picture

Assigned: wim leers » Unassigned
StatusFileSize
new1.5 KB
new23.17 KB

Fix CS violations.

gabesullice’s picture

StatusFileSize
new1.16 KB

Rerolled.

gabesullice’s picture

StatusFileSize
new22.4 KB

Whoops.

gabesullice’s picture

StatusFileSize
new5.55 KB
new22.35 KB
  1. +++ b/src/Context/FieldResolver.php
    @@ -152,8 +153,9 @@ class FieldResolver {
    +    $cacheability = (new CacheableMetadata())->addCacheContexts(['url.query_args:include']);
    ...
    -      throw new BadRequestHttpException('Empty include path.');
    +      throw new CacheableBadRequestHttpException($cacheability, 'Empty include path.');
    

    ❤️

  2. +++ b/src/Context/FieldResolver.php
    @@ -177,17 +179,17 @@ class FieldResolver {
             : $previous_messages
    
    @@ -522,7 +525,7 @@ class FieldResolver {
    +      throw new CacheableBadRequestHttpException($message);
    

    Missing the first argument of a CacheableDependencyInterface.

  3. +++ b/src/Context/FieldResolver.php
    @@ -233,12 +235,13 @@ class FieldResolver {
    +    $cacheability = (new CacheableMetadata())->addCacheContexts(['url.query_args']);
    

    This could be even narrower: ['url.query_args:filter', 'url.query_args:sort']

  4. +++ b/src/Controller/EntityResource.php
    @@ -353,7 +354,7 @@ class EntityResource {
    +        throw new CacheableBadRequestHttpException(new CacheableMetadata(), sprintf("Filtering on config entities is not supported by Drupal's entity API. You tried to filter on a %s config entity.", $config_entity_type_id));
    

    We can at least cache on the URL's path and filter I think.

  5. +++ b/src/EventSubscriber/JsonApiRequestValidator.php
    @@ -43,6 +44,8 @@ class JsonApiRequestValidator implements EventSubscriberInterface {
    +    $cacheability = (new CacheableMetadata())->addCacheContexts(['url.query_args']);
    
    @@ -60,7 +63,7 @@ class JsonApiRequestValidator implements EventSubscriberInterface {
    +      $exception = new CacheableBadRequestHttpException($cacheability, 'JSON API does not need that ugly \'_format\' query string! 🤘 Use the URL provided in \'links\' 🙏');
    

    Let's just cache this based on the _format param alone.


I went ahead and made all those changes. RTBC from my perspective.

Status: Needs review » Needs work

The last submitted patch, 35: 2929428-35.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

gabesullice’s picture

Status: Needs work » Needs review
StatusFileSize
new576 bytes
new22.35 KB

Forgot to update one thing.

wim leers’s picture

Status: Needs review » Reviewed & tested by the community
  1. +++ b/src/Context/FieldResolver.php
    @@ -238,7 +238,7 @@ class FieldResolver {
    -    $cacheability = (new CacheableMetadata())->addCacheContexts(['url.query_args']);
    +    $cacheability = (new CacheableMetadata())->addCacheContexts(['url.query_args:filter', 'url.query_args:sort']);
    
    +++ b/tests/src/Functional/UserTest.php
    @@ -462,7 +462,7 @@ class UserTest extends ResourceTestBase {
    +    $this->assertResourceErrorResponse(400, "Filtering on config entities is not supported by Drupal's entity API. You tried to filter on a Role config entity.", $collection_url, $response, FALSE, ['4xx-response', 'http_response'], ['url.path', 'url.query_args:filter'], FALSE, 'MISS');
    

    Nice!

  2. +++ b/src/Context/FieldResolver.php
    @@ -525,7 +525,8 @@ class FieldResolver {
    -      throw new CacheableBadRequestHttpException($message);
    +      $cacheability = (new CacheableMetadata())->addCacheContexts(['url.query_args:filter', 'url.query_args:sort']);
    +      throw new CacheableBadRequestHttpException($cacheability, $message);
    
    +++ b/src/Controller/EntityResource.php
    @@ -354,7 +354,8 @@ class EntityResource {
    -        throw new CacheableBadRequestHttpException(new CacheableMetadata(), sprintf("Filtering on config entities is not supported by Drupal's entity API. You tried to filter on a %s config entity.", $config_entity_type_id));
    +        $cacheability = (new CacheableMetadata())->addCacheContexts(['url.path', 'url.query_args:filter']);
    +        throw new CacheableBadRequestHttpException($cacheability, sprintf("Filtering on config entities is not supported by Drupal's entity API. You tried to filter on a %s config entity.", $config_entity_type_id));
    

    Good catch — I missed these!

I went ahead and made all those changes. RTBC from my perspective.

RTBC from mine too!

axle_foley00’s picture

Sorry for my lack of progress over the past few weeks. Thanks for working on this @gabesullice and @wim-leers, I unfortunately got bogged down with some work commitments.

wim leers’s picture

@axle_foley00 Don't apologize! :) You got this started, that's what matters most!

  • Wim Leers committed 2a9988b on 8.x-2.x
    Issue #2929428 by Wim Leers, gabesullice, axle_foley00: [>=8.5] Convert...
wim leers’s picture

Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.