[Meta] Page rendering meta discussion

Drupal 8's page rendering pipeline has evolved several times now in the course of the last 3 years. Originally part of the WSCCI initiative, it split off to the SCOTCH initiative in 2012. After that effort was largely abandoned the WSCCI team has been attempting to complete at least the baseline of work. Unfortunately that has not been as well-organized as it could have been due to lack of resources.

This writeup documents the "8.0-target according to Crell" of where we are still trying to push the page building mechanism. Not all of this may be completed by release, but hopefully by laying it out big-picture we'll get a bit closer than we would have otherwise. I hope...

Overview

There are a few key underlying principles:

• Above the controller/block level, render arrays have no meaning. Data at this level should be represented by coarser grained objects.
• Controllers MAY return responses, but in the more typical case they will return a meaningful domain object.
• Turning a meaningful domain object into a Response is the responsibility of the View event.
• View listeners in Symfony's HttpKernel are not required to completely handle a controller result. It's entirely allowed for them to modify the result but not set a Response, and then that modified result is available for later-firing listeners. That allows for a very flexible pipeline of enhancing or converting an object from one to another in flexible stages.

The key meaningful domain objects for HTML-based output are, in increasing coarseness:

Render arrays

This isn't properly an "object" but for Drupal purposes we sort of treat it as such. I will refer to it as such for the purposes of this description.

HtmlFragment

This represents a post "turn render array into a string" string of HTML, plus associated metadata. This metadata for example consists of assets as well as cache information.

HtmlPage

This represents an HtmlPage with the entire contents between the body tags as a string, plus associated metadata (HTML links, assets etc.).

Response

A Symfony response object, whose body is now a string.

Note that, in concept, this list is not exhaustive. Any controller may return any meaningful domain object it wants, provided that there is a view listener registered that will turn it into a Response. That is implicit in the design of the Symfony HttpKernel pipeline. We're simply leveraging it. (Symfony does too in some cases; with FrameworkExtraBundle you can annotate a controller to specify a twig template to use, then return an array. A view listener will take the array and run it through the twig template specified.)

For each of those meaningful domain objects there is a free-standing service that converts it to the next level up:

render array->HtmlFragment

RenderHtmlRenderer (coming in #2256365: Factor render->fragment code out to a service)

HtmlFragment->HtmlPage

\Drupal\Core\Page\HtmlFragmentRendererInterface (although in practice I don't see any alternate implementations to DefaultHtmlFragmentRenderer)

HtmlPage->Response

\Drupal\Core\Page\HtmlFragmentRendererInterface\HtmlPageRendererInterface (and its core default, DefaultHtmlPageRenderer

The latter two, realistically, will only be used once per request but are still isolated operations unto themselves and thus their own services. The first could conceivably be used multiple times. (See next section.) They are collectively called "Renderers" because... that's the name we came up with when they went in. :-) (And they "render" one object into the next.) I am not opposed to a dedicated issue to rename them collectively, as long as it's consistent and not confusing with other subsystems.

Those services are entirely divorced from HttpKernel. However, they are wired into it as view listeners that fire in that order. The listener is simply a bridge to the functionality, as it should be. (See Drupal\Core\EventSubscriber\HtmlViewSubscriber.) There are also ample "gaps" between the priority of those listeners where other subsystems (core or contrib) may interject their own logic to either conditionally pre-empt the defaults, inject additional data into those objects (eg, additional metadata), and so on.

Note that this means, in concept, that one could return an entity from a controller, or a Form object itself, or some other classed object from a controller and as long as there's a service wired in via a View listener with higher priority than HtmlFragmentRenderer, and it converts whatever that object is to an HtmlFragment, everything will still work. It would not even, technically, be an API change. That's kinda nice. :-) That would allow for the pipeline described in https://www.drupal.org/node/2092647#comment-8874069 and following (which is a longer term goal but we cannot remove the alternate _-properties before 8.0.0, which means they must remain supported until at least 9.0.0).

It also means that those steps may vary depending on any conditions desired. For instance, an HtmlFragment (containing “the main content”) could be turned into an HtmlPage (with surrounding blocks) if the request format is HTML, but it could also be turned into an AjaxResponse (with just the main content, and no surrounding blocks, since that’s probably not useful when wanting an AJAX response) if the mime type is something else. We actually have an enormous amount of flexibility in how we manage the "thing that gets returned" by putting all of the logic between a very thin controller and the point that we have an actual response object.

Metadata

The above description referred to "associated metadata" several times. The term changes in just about every issue; I am using the term "associated metadata" for this writeup because I don't believe we have a firmly established noun Defined(tm).

Associated metadata refers, generically, to "things that this string of HTML needs in the head tag of the page or in a Response header". That includes, but is not limited to, a title (for the page), CSS, Javascript, Links, Meta tags, the HTTP response code, etc. Render arrays store that data somewhat haphazardly at the moment. HtmlFragment and HtmlPage incorporate it as structured objects.

For Drupal-specific purposes, cache tag information is also part of "associated metadata".

Note that this process allows for very robust "metadata alter" process with no additional work. Such alteration is just another View listener that happens to sit at the right position in the pipeline. The most obvious use case for that is CSS/JS aggregation. However, it is also the appropriate place for modules like Google Analytics, Metatags, and others to inject page-level information. (Eg, inject Google Analytics-required header tags, but only for HTTP 200 pages.) All they need do is register a listener at the right level of coarseness to modify the object at the appropriate level.

Why HtmlFragments?

Strictly speaking, there's nothing that HtmlFragments can do that a render array couldn't. They are both data structures that hold information. However, HtmlFragments have a key advantage: formality. Render arrays are an entirely implicit unstructured data structure. That makes them, by design, difficult to document, impossible to "discover", and difficult to reason about.

A few related links are, I think, helpful. The first is a short piece by chx on discoverability. The second is a longer piece on strong typing, which classes give but render arrays naturally are not:

http://www.drupal4hu.com/node/403
http://techblog.realestate.com.au/the-abject-failure-of-weak-typing/

Being easier to reason about and work with is at one level "just" a DX problem. However, that also makes it conceptually feasible to have, for example, the multi-stage conversion process described above. While hypothetically possible to represent HtmlFragment and HtmlPage as just anonymous arrays, that would make the conversion code more complex because you'd have to determine which type of array something is at any given time. Having structured, purpose-built data objects has a knock-on effect similar to having cleanly separated services. It allows for thought processes and conceptualizations that would have been too difficult to be feasible previously by abstracting concepts out to the right level.

For the moment, the primary use of that is to make the page layout process more consistent and simpler. (See next section.) However, it also makes it conceptually easier to reason about rendering blocks in isolation (necessary for ESI, for hInclude, for Ajax callbacks to refresh just one block on a page, etc.). That conceptually simpler page rendering process, in turn, will enable more experimentation and innovation in contrib and later core versions.

There are, I expect, many other benefits that we simply cannot see now, but just as splitting up our logic to clean services has enabled easy-peasy backend switching (https://www.drupal.org/node/2306083) or the normalization of the entity/field system was a prerequisite to generic REST support, I am confident that a commitment to a clean, object-oriented renderable/output pipeline will pay off dividends we haven't even thought of yet.

Page rendering

Obviously the most complex step in that process is the HtmlFragment->HtmlPage conversion process, also known as "rendering the page".

A Page, conceptually, consists of one or more HtmlFragments arranged in a layout. One of those HtmlFragments is the one returned by the controller or created by a previous View listener from whatever was returned by the controller. The others are... blocks. Blocks are just HtmlFragments, or rather HtmlFragmentInterface objects, whose content and metadata are calculated on the fly rather than assigned externally.

There is nothing in a page that is not an HtmlFragment or part of the layout. Nothing.

A layout consists of a template, regions, and configuration for what HtmlFragments go into what regions.

Because blocks are themselves HtmlFragments, no special handling is needed to merge them. At most, there may be interesting logic happening for loading blocks but that's it. And probably not even that as all blocks are plugins, so they should be fairly consistent.

Caching

Caching is, of course, a big deal for Drupal. We have a lot of it, and it's not always consistent. For the moment we're speaking specifically of partial output caching. Render arrays already have cache tags and callbacks in them but they are often difficult to use.

HtmlFragments already have the ability to carry cache tags and similar information, which represent the relevant cache tags that were used to generate the fragment's body. When a render array is converted to an HtmlFragment, those cache tags can be transferred over to it. Thus, future cache clears will affect a cached fragment. As of yet we do not

While HtmlFragment is coarser grained than render arrays, that's fine. Very often we don't need to go more coarsely grained, as a given page body or a given block do not vary on much at all. As the underlying render array's caching is still present even if we do bust the cache on a particularly context-sensitive HtmlFragment the render array caching below it is still in place.

However, having the fragment cached on its own allows us to then load and reuse it very cheaply. Think Panels pane caching, but for every block equally. Or for the regular go-to example, ESI/hInclude. That cache information can even be then listed in the URI of a solo-block. That likely won't be included in core in 8.0.0 but it opens up the potential to do so in 8.1.

Differences from status quo, circa August 2014

We are not quite at the big picture described above. There are three main areas where we are still falling short.

1) Wrapping controllers

This architecture predates the introduction of HtmlFragment. In this model (currently implemented in HEAD), the negotiation for what to do with "the thing the controller returns" happens before the controller is called. In the model described above, it happens after. That made sense when we had nothing more robust than render arrays to describe the return value. Now that we do, however, I believe post-controller is, in the long run, the better approach.

The work happening in #2256365: Factor render->fragment code out to a service is step one of transitioning to post-controller negotiation; all it attempts to do is create RenderHtmlRenderer as described above: no more, no less. Follow-ups (again, non-API breaking) would be to deprecate and remove HtmlControllerBase entirely. That of course brings up the question of what to do with forms (wrap them in an HtmlFragmentInterface-implementing object themselves, perhaps?) and the other wrapping controllers. What to do with those is a more interesting question than just moving code around, hence why it is not included in that issue.

For API-stability reasons we cannot and will not remove the alternate _-properties from routes, so those will remain supported. How we go about handling those, however, is not an API change and flexible for us to refactor as long as the promised behavior (eg, "a form will appear at this URL as the body of the page") remain intact.

It may also be wise to have another sub-type of HtmlFragmentInterface specifically for the controller fragment (HtmlMainFragmentInterface? Eh, naming is hard) that is uniquely able to carry things like the status code and page title, so that blocks cannot. It could also potentially be leveraged in the future for ordering of assets so that, e.g., main body CSS and JS can be loaded first, or prioritized in an HTTP 2.0 resource stream.

The other caveat is that currently controllers may return strings, not just render arrays. Strings are an even more primitive version of a render array, essentially. If we go all the way to removing the wrapping controllers then we will need some step that converts a string controller response to a render array or straight to an HtmlFragment. While that could be done as its own step, that seems unnecessary to me. Instead, I believe folding that if statement into RenderHtmlRenderer is the cleanest path. That does tacitly mean that strings are sort-of a render array, kind of, but I think that's acceptable in order to maintain current functionality (ie, no API break) and avoid a pointless wrapper class or extra event listener that would do nothing other than return ['#markup' => $parameter];

2) Assets

Currently, HtmlFragment still does not handle CSS and Javascript. Those are the "interesting" associated metadata as they require non-trivial preprocessing. Sam Boyer has a considerable amount of work done to build a framework for declaring and mangling those assets built on Assetic (#1762204: Introduce Assetic compatibility layer for core's internal handling of assets), but that work was abandoned months ago and my attempts to get someone else to pick it up have been unsuccessful.

At this time, I think it is best to just ignore aggregation and build the rest of it, relegating aggregation to a View listener with a big message saying "insert aggregation here". That would be fairly straightforward to do, in fact, as there's little hard logic there. It's just shuffling data around. I do not know if catch would support such an approach, however, but I do not have any better approach at this time that is actionable given the resources available. I do believe that HtmlFragment not supporting CSS/JS is a release-blocking issue.

3) Page building

I have no expectation whatsoever that the full vision of SCOTCH (multiple contextually-defined layouts) is going to happen in 8.0.0. However, in order to make that easy to implement via contrib we must, I believe, still finish lining up the dominos. That means two things, primarily:

1. Making blocks implement HtmlFragment directly (or via a child interface, but $block instanceof HtmlFragmentInterface should return true) so that page renderers (core's default or otherwise) can treat blocks and the main content area the same. To avoid massively rewriting all blocks, again, Sam Boyer suggested a trait that we shove into all existing blocks that lets them continue to work as they do now. That would mostly replace BlockBase (which is already extremely problematic as the API for its child classes is nothing at all like BlockInterface), but the patch would consist of 1-2 lines for each existing block, nothing significant. We could then rewrite core blocks at our leisure, even post 8.0.0, to use the updated interface directly.
2. Adapt DefaultHtmlPageRenderer to have no external sideways dependencies. That is, we do not need to turn DefaultHtmlPageRenderer into Panels. We just need to insulate it so that the rest of the system can't depend on its implementation details. That means, primarily, eliminate hook_page_alter() and hook_page_build(). The former should be easier, but the latter would need to be done in concert with the changes to blocks from point A (since blocks currently alter themselves onto the page via hook_page_build().) We would likely also need to internalize drupal_prepare_page(), as that logic is really part of the "default" page building process.

The resulting code doesn't have to be pretty! It need only limits its surface area to HtmlMainFragment and HtmlFragment-based blocks. It should still be hard coded to the current block configuration process. As long as that is done we can easily gut and rewrite it post-8.0.0 without an API break, and contrib can drop in alternate implementations, including a Chained implementation, without disrupting anything.

The primary blocker for all of these steps is resources. I do not personally have the time to do most of it; maybe 10% of it. If others want to help, and have the domain-area skills to do so, please let me know because I desperately need volunteer hours to get this done.

The steps

I've tried to create at least a few stub issues that move us along this path. They do, in many cases, have dependencies. That is sadly unavoidable. If we can agree on the general vision then some of them may be able to break down to smaller issues that on their own do nothing useful, but help move an issue forward without conflicting with another patch. That's up to the committers to decide if they're OK with.

Crell thinks are beta target:

#2256365: Factor render->fragment code out to a service - Create RenderHtmlRenderer service
#2334207: Revisit naming of "renderers" in the page building pipeline - Revisit naming of "renderers" in the page building pipeline
#2334029: Add js/css/libraries/head links to HTML fragments and pass it along to the page. Assigned to: dawehner - Add CSS/JS support to HtmlFragment
#2296951: Expand the support of head elements to allow every common head element. - Add arbitrary HEAD element support to HtmlFragment

Can probably be post-beta:

#2295609: Break the dependency of MaintenanceModeSubscriber on DefaultHtmlPageRenderer Assigned to: damiankloip - Eliminate usage of DefaultHtmlPageRenderer::renderPage() (which is a fugly BC hack) from maintenance page handling, so we can get closer to eliminating DefaultHtmlPageRenderer::renderPage().
#2334219: Make blocks expose HtmlFragmentInterface directly - Make blocks expose HtmlFragmentInterface directly (some variations of this could be post-beta, see discussion in the thread)
#2334225: Move CSS/JS preprocessing to a View listener - Move CSS/JS preprocessing to a View listener
#2334213: Split HtmlMainFragment off of HtmlFragment for the page body - Split HtmlMainFragment off of HtmlFragment for the page body
#2334217: Internalize drupal_prepare_page() - Eliminate drupal_prepare_page() (no module dev should be touching this anyway)
#2334229: Move controller render array -> HtmlFragment conversion to a View listener; eliminate HtmlPageController - Move controller render array -> HtmlFragment conversion to a View listener; eliminate HtmlPageController
#2334233: Move Modal/Dialog detection to a View listener - Move Modal/Dialog detection to a View listener
#2334235: Replace AjaxController with a View listener - Replace AjaxController with a View listener
#2334247: Replace FormController and children with wrapping closures and View listeners - Replace FormController and children with wrapping closures and View listeners
#2334191: Move main content fallback into the block configuration level - Eliminate special handing for missing main content block
#2334221: Move block building from hook_page_build() to DefaultHtmlPageRenderer - Move block building from hook_page_build() to DefaultHtmlPageRenderer