REST views: row-level caching doesn't exist, unlike for other types of views

This helps quite a lot, see the attached screenshot.

Let's see whether this breaks stuff

Fair, let's see first, whether this breaks anything.

We need maybe add some tests around escaping etc. While implementing that in custom code, I had quite some issues around that.

Well, my actual usecase was a 120 seconds view down to 1.5 seconds, but yeah, this was too crazy for a core issue

Right, its all about cache invalidation. The simple usecase of a rest view, which just renders some of the fields of an entity, works fine, as each row has their cache tags. The advanced usecases are the once which cause troubles. Just imagine you embed a view inside the main view to render some fields of the current row entity. Then you would need to take into account cache tags for that referenced entity as well. All that complex bubbling logic is solved by render API, but sadly it's something we don't have in REST views.

One potential solution @jaesin and others talked about is to introduce intermediate objects, which carry around the metadata for the field rendering and finally extract the data out later, before we pass the data to the serializer to render everything.

Explain this dark magic?!

getField() calls out \Drupal\views\Plugin\views\style\StylePluginBase::renderFields which does some render caching one day.

Exactly, now the only problem is that we don't have any kind of bubbleable cacheable metadata, just the one on the row itself. I was thinking about adding an advanced option,
so you can get this single line, in case you want to and you know what you do.

To be honest the only thing I see possible, is to make it configurable. There is no clear API layer where we could opt out of caching.

This patch so far is obviously not enough!

#2 shows a significant perf win. What was the test setup?

I don't remember, but I guess its like 50 nodes in a rest feed or so. The actual site here decreased there numbers from 3 minutes to 1,5 seconds, but this was rendering the entirety of the entire site out with thousand of nodes.

Here is the basic approach I haven chosen for the client code:

Inside a custom field plugin which does complex stuff:

      $data = new RestFragmentData($result);
      $data->addCacheTags(['foo', 'bar']);
      $build = [
        '#markup' => $data->jsonSerialize(),
      ];

      CacheableMetadata::createFromObject($data)->applyTo($build);

      return $build;

Inside DataFieldRow:

        $value = $this->view->style_plugin->getField($row->index, $id);

        // Render caching converts RestDataFragment objects to its encoded json.
        // In order to get the data back, try to decode the json array.

        // First reset the last json error.
        json_decode(1);

        $data = @json_decode($value, TRUE);
        if (json_last_error() === 0) {
          if (is_array($data)) {
            array_walk_recursive($data, function (&$val) {
              $val = Html::decodeEntities((string) $val);
            });
          }
          $value = $data;
        }
      }

      $output[$this->getFieldKeyAlias($id)] = $value;

With RestFragmentData

class RestFragmentData implements MarkupInterface, CacheableDependencyInterface {

  use RefinableCacheableDependencyTrait;

  /**
   * The stored data.
   *
   * @var array
   */
  protected $data;

  /**
   * Creates a new RestFragmentData instance.
   *
   * @param array|string $data
   */
  public function __construct($data) {
    $this->data = $data;
  }

  /**
   * {@inheritdoc}
   */
  public function __toString() {
    return $this->jsonSerialize();
  }

  /**
   * {@inheritdoc}
   */
  public function jsonSerialize() {
    return json_encode($this->data);
  }

}

As we cannot enforce custom field plugins to do that crazy stuff, I don't see how this could be solved automatically without opting in as site builder.

I don't get that patch, honestly. You at least have to combine it somehow with my previous patch, because well, there is no row level caching triggered at the moment. Yes, the metadata of each row bubble to the entire view, but this is not the point of this issue to be honest, see first sentence of the issue summary.

To be clear, in the custom project I was working one we had a field plugin that returned raw data, and well it worked, so requiring cache ability metadata needs to be somehow supported and opt IN for this raw data.

Let me clarify why I think this issue is quite problematic when we want to keep BC.
While the patch in #2 proves that the simple case actually doesn't break anything, here is the what happened on the client project while dealing with it.

The patch in #2 got implemented and boom nothing invalided anymore. We had though a custom field plugin which just rendered stuff and returned a string. This one was specifically targeted towards REST, so we haven't thought about cacheability yet at that point in time. When we introduce simply #2 we gonna hit that problem on other sites as well.

On top of that advancedRender also potentially renders via the theme() function, which includes the template used by field templates. This can be obviously be problematic
when someone has any kind of HTML in their field template.

Given to make this BC compatible we need the following steps IMHO:

• Be able to skip field templates completly
• Be able to opt out / opt in into row level caching. I would vote for opt IN for rest views.

Well in the case above opt OUT would have totally broken the site.

The bigger problem is that you can currently only get ALL results for a REST view: there's no paging.

I honestly don't know who came up with that myth. You can just configure a view like any other view, and use ?page=1 and paginate. What doesn't work is having the appropriate link relations in the http header, which everyone would always use, right ;)

Mh. Well at least #2099281: [PP-1] REST views: pagination link relations is certainly not major in that case.