Problem/Motivation

The ResourceIdentifier class that is used to add all entity references to the relationship is currently using an entity load to add all relationship entries to the jsonapi output.

This can lead to a huge set of queries being executed for collection requests, as all references of the nodes are loaded (paragraphs, users, terms, node references, ...)

In my example this leaded to 8372 for cold cache hits, because the client had a total of 1k paragraphs that were used on articles in the result set.

Using entity load for the relationships is heavy, as my json api requests are not requesting info about those relations. The paragraphs are only requested on the detail page of a node. The users are also loaded, while no user info is displayed in the responses.

SELECT "t".* FROM "user__field_password_expiration" "t" WHERE ("entity_id" IN (:db_condition_placeholder_0)) AND ("deleted" = :db_condition_placeholder_1) AND ("langcode" IN (:db_condition_placeholder_2, :db_condition_placeholder_3, :db_condition_placeholder_4)) ORDER BY "delta" ASC
SELECT "t".* FROM "paragraph__field_text_body" "t" WHERE ("entity_id" IN (:db_condition_placeholder_0)) AND ("deleted" = :db_condition_placeholder_1) AND ("langcode" IN (:db_condition_placeholder_2, :db_condition_placeholder_3, :db_condition_placeholder_4)) ORDER BY "delta" ASC

Comments

nils.destoop created an issue. See original summary.

nils.destoop’s picture

I'm using this patch to convert all the entity load to 1 query to the base table. The resource identifier only needs a uuid, bundle and entity type.

Next to that, I also implemented the hook_entity_field_access in my custom module, to set view access to false for entity references (like paragraphs) on json api collection requests that are not needed on lists in my project.
As extra tweak I used a custom version of the patch in https://www.drupal.org/project/jsonapi_extras/issues/3039331#comment-143...

Before changes: 8372 queries
After applying this patch + implementing access rules on the fields: 1486 queries
After also applying the patch for link fields: 1116 queries

nils.destoop’s picture

Issue summary: View changes
cilefen’s picture

Status: Active » Needs review

Can you measure the execution time of this proposal vs the main branch and report that here?

ranjith_kumar_k_u’s picture

StatusFileSize
new2.49 KB

Status: Needs review » Needs work

The last submitted patch, 5: 3255840-5.patch, failed testing. View results

nils.destoop’s picture

StatusFileSize
new418.54 KB
new229.81 KB
bbrala’s picture

Thank you for the profiling. Seems like there is some real performance to be had. I wonder though, would disabling the field (see https://www.drupal.org/docs/core-modules-and-themes/core-modules/jsonapi...) also work in reducing those calls?

nils.destoop’s picture

No it didn't. I use json api extras to disable some of those fields. But they still were loaded and removed afterwards. Implementing the field_access hook makes sure the related normalizer is not activated:

https://git.drupalcode.org/project/drupal/-/blob/9.4.x/core/modules/json...

yang_yi_cn’s picture

@nils.destoop can you explain how can I implement my own hook_field_access to speed it up?
I think I sometimes get hit by this issue too and it takes forever to load certain resources, which makes my app not usable.

nils.destoop’s picture

@yang_yi_cn my version:

It disables the social image relation and the paragraphs relation on collection requests (except for 1 content type, special case for the app).
For all other pages it also disables some basic fields.

function uitbe_jsonapi_entity_field_access($operation, \Drupal\Core\Field\FieldDefinitionInterface $field_definition, \Drupal\Core\Session\AccountInterface $account, \Drupal\Core\Field\FieldItemListInterface $items = NULL) {
  $routeParts = explode('.', Drupal::routeMatch()->getRouteName());
  if ($routeParts[0] === 'jsonapi' && $routeParts[2] === 'collection') {
    $collectionExcludedFields = [
      'field_social_image',
    ];

    if ($routeParts[1] !== 'node--landing_page') {
      $collectionExcludedFields[] = 'field_content';
    }

    if (in_array($field_definition->getName(), $collectionExcludedFields)) {
      return AccessResult::forbidden()
        ->addCacheContexts(['route.name']);
    }
  }

  if ($operation === 'view' && $account->isAnonymous()) {
    $excludedFields = [
      'uid',
      'vid',
      'langcode',
      'revision_uid',
      'revision_timestamp',
      'revision_log',
      'publish_on',
      'node_type',
      'paragraph_type',
    ];
    return AccessResult::forbiddenIf(in_array($field_definition->getName(), $excludedFields))
      ->cachePerPermissions();
  }

  return AccessResult::neutral();
}
leon kessler’s picture

@nils.destoop thanks for posting all your work, including profiling info.

We're also experiencing slow response times on cold cache requests.
e.g. hitting /jsonapi/node/page takes around 6 seconds. When adding something like ?include=field_reference it goes up to around 9 seconds.
Not quite as bad as your situation with paragraphs. Our site doesn't use paragraphs, but is quite heavy on taxonomy. There are around 7 or 8 taxonomy reference fields on each content type.

Our issue we have lots of cache tag invalidations occurring, due to taxonomy terms often being updated (so large chunks of content get cache wiped out in one go). So having 6-9 second response times on these requests is not ideal.

When trying your patch and example of hook_entity_field_access() the amount of db queries was reduced, from 1731 to 1072. However response times are not noticeably improved, so I'll continue investigating. Possibly there's not a huge amount we can do, and may turn to using cache warmers and some slight tweaking to how cache invalidation works.

Anyway, in regards to your patch. I've re-rolled it to apply cleanly to 9.3.x
I also encountered the following error:

The website encountered an unexpected error. Please try again later.
AssertionError: assert($value === NULL) in assert() (line 440 of core/modules/jsonapi/src/JsonApiResource/ResourceIdentifier.php).
assert(, 'assert($value === NULL)') (Line: 440)

This occurred on /jsonapi/node/page. It occurs when it tries to load type field (i.e. the node type). I added type to the list of $excludedFields in hook_entity_field_access and this fixes the issue. Although only for anonymous users.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.