Problem/Motivation

#2471473: REST responses should have proper cache tags ensured entity cacheability metadata is present. Dealing with entity & field access was deferred to an undefined follow-up. We should fix this ASAP because it could expose sensitive information for sites with complex entity access.

(Note that the user.permissions cache context is present by default, due to a permission being required to access any of the REST routes. This is why it's okay for this to be handled in the public issue queue.)

Proposed resolution

Associate entity & field access cacheability metadata with the response.

Remaining tasks

None.

User interface changes

None.

API changes

None.

Data model changes

None.

Original report

I jumped from beta5 top beta10 with a custom simple REST endpoint module, basically following this http://enzolutions.com/articles/2014/12/16/how-to-create-a-rest-resource...

My custom endpoint /rest/articles/get returns nids of all articles.

It worked and works fine, but since some update, stale content is being served.
When I clear the Drupal cache, my endpoints returns the new nids, so it must be a cache thing.

I suspect it has something to do with the finished support for cache-tags altough #2471473: REST responses should have proper cache tags claims it has been fixed in REST output.

Now for the weird part:

When I retrieve data from the same endpoint through Chrome's Dev HTTP Client, the results are up to date! (btw: I see "X-Drupal-Cache-Tags: config:rest.settings" in the headers for the response).

So what's going on here? How do I get fresh results when using guzzle as a http client? I've ruled out browser caching.

Comments

askibinski’s picture

Did some more debugging, when I perform the request through JS instead of Guzzle and add the cache: false option, I get the fresh results like in Dev HTTP Client:

$.ajax({
    method: "GET",
    url: "http://localhost/rest/list/articles",
    cache: false,
    contentType: "application/hal+json"
})
.complete(function( msg ) {
        console.log(msg);
})

Above code for javascript is similar to adding a timestamp parameter to your query in Guzzle.

But I would prefer my custom rest endpoint cache to be updated when the results change. Any documentation on how to do that?

clemens.tolboom’s picture

Issue summary: View changes
clemens.tolboom’s picture

Some questions:

How does wget / curl respond?
Are you logged in?
Can you share your guzzle code on ie github?

askibinski’s picture

curl also receives fresh content:

curl -i -X GET \
   -H "Accept:application/hal+json" \
   -H "Authorization:Basic XXXXXXXXXXXXX" \
 'http://.localhost/rest/list/articles'

Here is the guzzle code.

...

use GuzzleHttp\Client;

  $client = new Client([
    'base_url' => 'http://localhost/rest/list/articles',
    'defaults' => [
      'headers' => ['Accept' => 'application/hal+json'],
    ]
  ]);

$response = $client->get('/rest/list/articles'); // <--- stale results

$response = $client->get('/rest/list/articles?_=' . time()); // <--- fresh results

$data = $response->getBody();
}

update: the curl code uses authentication but it does not matter, if I enter "0" as username and leave password empty I still get fresh results through curl/dev client. But not so in php/js from the client. (unless I force to disable cache bij adding the cache parameter which adds a timestamp).

I'm pretty sure this has something to do with the new cachetags but I'm not sure how to use the cache api in order to make sure Drupal invalidates this particular cache entry when a new node is posted.

clemens.tolboom’s picture

wim leers’s picture

Title: Stale content from custom REST endpoint (cache tags issue?) » Cacheability metadata for REST responses incomplete
Version: 8.0.0-beta10 » 8.0.x-dev
Assigned: Unassigned » wim leers
Category: Support request » Bug report
Priority: Normal » Major
Issue tags: +D8 cacheability, +Security

The problems described in the issue so far sound like browser caching, plain & simple. If the response have a Cache-Control: max-age=60 header for example, the browser is going to cache the response for a minute, regardless of cache tags being invalidated. Cache tags can only invalidate responses on the server, not on the client.


But, I'm going to repurpose this issue for something closely related, because in EntityResource there's at least this problem wrt cacheability:

    $response = new ResourceResponse($entity, 200);
    // Make the response use the entity's cacheability metadata.
    // @todo include access cacheability metadata, for the access checks above.
    $response->addCacheableDependency($entity);
    return $response;
  }

So, I'll take a look at improving this.

wim leers’s picture

wim leers’s picture

wim leers’s picture

Status: Active » Needs review
StatusFileSize
new4.55 KB

The fix was trivial, it took me far more time to write test coverage. (Rolled against 8.1 but should apply cleanly to 8.0 too.)

wim leers’s picture

Title: Cacheability metadata for REST responses incomplete » REST entity resource missing entity & field access cacheability metadata
Issue summary: View changes
Issue tags: -Needs issue summary update
wim leers’s picture

Assigned: wim leers » Unassigned

Status: Needs review » Needs work

The last submitted patch, 9: rest_entity_access_cacheability-2485683-9.patch, failed testing.

wim leers’s picture

Assigned: Unassigned » wim leers

Unrelated test failures due to pre-existing tests asserting the expected cacheability metadata for the field_test_text field, which I changed here to have something to detect:

+++ b/core/modules/system/tests/modules/entity_test/src/EntityTestAccessControlHandler.php
@@ -59,4 +61,16 @@ protected function checkCreateAccess(AccountInterface $account, array $context,
+  protected function checkFieldAccess($operation, FieldDefinitionInterface $field_definition, AccountInterface $account, FieldItemListInterface $items = NULL) {
+    // Custom cacheability metadata for viewing the 'field_test_text' field.
+    if ($operation === 'view' && $field_definition->getName() === 'field_test_text') {
+      return AccessResult::allowed()->addCacheTags(['entity_test_accesss:field_test_text']);
+    }
+
+    return parent::checkFieldAccess($operation, $field_definition, $account, $items);
+  }

I'll work on addressing those.

neclimdul’s picture

Oh wow. Going to have to work to wrap my head around that.

Skim:

  1. +++ b/core/modules/rest/src/Plugin/rest/resource/EntityResource.php
    @@ -45,19 +45,25 @@ class EntityResource extends ResourceBase {
    +    $entity_access = $entity->access('view', NULl, TRUE);
    

    NULL

  2. +++ b/core/modules/rest/src/Plugin/rest/resource/EntityResource.php
    @@ -45,19 +45,25 @@ class EntityResource extends ResourceBase {
    +      var_dump($field_name);
    

    :-D

joshi.rohit100’s picture

+++ b/core/modules/rest/src/Plugin/rest/resource/EntityResource.php
@@ -45,19 +45,25 @@ class EntityResource extends ResourceBase {
-    if (!$entity->access('view')) {
+    $entity_access = $entity->access('view', NULl, TRUE);
+    if (!$entity_access->isAllowed()) {

NUL1 ?

wim leers’s picture

lol lol lol

I'm a moron.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new4.53 KB
new1.23 KB

First, fix the plain stupid things.

wim leers’s picture

I can't reproduce the failure in Drupal\rest\Tests\ReadTest. :(

Let's hope #17 will only have failures in Drupal\system\Tests\Entity\FieldAccessTest — the failures there I can reproduce.

Status: Needs review » Needs work

The last submitted patch, 17: rest_entity_access_cacheability-2485683-17.patch, failed testing.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new4.7 KB
new3.52 KB

This fixes the remaining failure.

Interdiff explanation: Rather than modifying EntityTestAccessControlHandler, I'm now just adding one more special value to entity_test_entity_field_access() to test for. In doing so, I have to use addCacheableDependency(). The surrounding code is still using the deprecated cacheUntilEntityChanges(). I think it's in scope here to update them.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community
+++ b/core/modules/rest/src/Tests/PageCacheTest.php
@@ -33,6 +33,7 @@ public function testConfigChangePageCache() {
+    $entity->set('field_test_text', 'custom cache tag value');

+1 for a more explicit value.

catch’s picture

Status: Reviewed & tested by the community » Needs review
+++ b/core/modules/rest/src/Tests/PageCacheTest.php
@@ -40,6 +41,7 @@ public function testConfigChangePageCache() {
+    $this->assertCacheTag('entity_test_accesss:field_test_text');

Are there three s on purpose? Looks like a typo that got copied everywhere.

wim leers’s picture

Status: Needs review » Needs work

Yep, that's a typo. Fixing.

wim leers’s picture

Status: Needs work » Reviewed & tested by the community
StatusFileSize
new4.7 KB
new2.77 KB
dawehner’s picture

He, good spot.

catch’s picture

Status: Reviewed & tested by the community » Fixed

For a minute I though it was a llama -> snake switch.

Committed/pushed to 8.1.x and cherry-picked to 8.0.x. Thanks!

  • catch committed ac415b1 on 8.1.x
    Issue #2485683 by Wim Leers: REST entity resource missing entity...

  • catch committed d5c659a on 8.0.x
    Issue #2485683 by Wim Leers: REST entity resource missing entity...
wim leers’s picture

For a minute I though it was a llama -> snake switch.

:P

LLAMAS FOREVER <3 <3

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.