Problem/Motivation

The Twig initiative provoked discussion about how the Drupal render system worked, and what could be accomplished or refactored with the move to Twig. The promise of Twig demands a drillable, accessible variable structure for themers. This does not yet exist, and we should look to refactoring Render API to reach this point.

Consider some examples from a node template. Here's directly rendering the first item in a multi-value image field (the variable names here aren't exactly what we have yet, this is just an example, and syntax of translatables/plurals may differ):

  <!-- First image, with manual tag creation -->
  {{ hide(content.field_image.0) }}
  <img class="banner" src="{{ content.field_image.0.attrs.src }}" alt="{{ content.field_image.0.attrs.alt }}" />

  <!-- Remaining content -->
  {{ content }}

  <!-- Links -->
  {{ links }}

  <!-- Comments -->
  {{ comments }}

We cannot do this currently. This is because things like the image src are prepared in the field theme callback preprocessor, which isn't available yet to the node template. Furthermore, the structure of render arrays tends to be chaotic as any #-prefixed key is generally used as a tool to communicate state downstream the array chain. This is bad design. See http://lb.cm/arrays-of-doom

Further examples of what we want to do in Twig:

Field UI provides for user-configurable formatters. Here's the first image for the field rendered using its formatter:

  <!-- First image, field UI formatter -->
  {{ content.field_image.0 }}

  <!-- Remaining content -->
  {{ content }}

  <!-- Links -->
  {{ links }}

  <!-- Comments -->
  {{ comments }}

Suppose we want all items in the field (and any field wrappers):

  <!-- Images first -->
  {{ content.field_image }}

  <!-- Remaining content -->
  {{ content }}

  <!-- Links -->
  {{ links }}

  <!-- Comments -->
  {{ comments }}

...or if we just want to see the node content itself with the field rendered according to its formatter and associated field UI weight:

  <!-- All content -->
  {{ content }}

  <!-- Links -->
  {{ links }}

  <!-- Comments -->
  {{ comments }}

Here's a more elaborate example:

  <!-- First image individually with some template class. -->
  <div class="banner">
    {{ content.field_image.0 }}
  </div>

  <h2>{{ title }}</h2>

  <a class="permalink" href="{{ url }}" title="{{ created|format_date('n/j/Y') }}">{% trans %}Permalink{% endtrans %}</a>

  <!-- Remaining images. Perhaps we have altered their presenters. -->
  {{ content.field_image }}

  <!-- Remaining content. -->
  {{ content }}

  <!-- Links separately as a list of tasks in sentence form. -->
  {% if links %}
    <aside>
      <h4>{% trans %}Tasks:{% endtrans %}</h4>
      {% for i, link in links %}
        {{ link }}{% if i != (links.count - 1) %},{% else %}{% trans %}or{% endtrans %}{% endif %}
      {% endfor %}
    </aside>
  {% endif %}

  <!-- Comments with comment count. -->
  <h4>{% trans -%}
    {{ comment|count }} recent comment
    {% plural comment|count %}
    {{ comment|count }} recent comments
  {%- endtrans %}</h4>
  {{ comments }}

These shortcomings threaten the success Twig in D8, and they are not the fault of Twig. Though Twig does give us some benifits natively, if we are not able to achieve the above examples, we feel that D8 is simply shipping with something different, not necessarily something better.

Proposed resolution

We believe in order to achieve what we want to accomplish in the long-term, render arrays must move toward an Object Oriented design. We're referring to these objects as Renderables or Presenters.

At DrupalCon Portland, c4rl and msonnanaum discussed some potential design patterns with effulgentsia and moshe weitzman. It was clear at the time that given the proximitiy to the code freeze date (7/1/2013), this is a potentially frighteningly large undertaking, nevertheless, that is no reason to delay further as Twig contributor momentum increased significantly.

Guiding principles (in-progress):

  1. Renderables know how to display themselves. Each sub-component of a renderable knows how to display itself.
  2. Renderables know if they are empty.
  3. Renderables access their properties in an intuitive way.
  4. Renderables talk to their parents to understand context.
  5. Renderables have configured defaults.
  6. Renderables' implementaton can be altered and extended.

Potential implementation

We believe renderables will take the form of a deferred factory. That is, a renderable is defined with some default class that can be altered (in the same way that alters could change `#theme` parameters).

    // D6 REALITY. We invoke the theme layer and get markup too early, that
    // can't be altered. Aside from preprocessors, we don't have a good way to
    // alter variables. However, there are still some calls to theme() in the D7
    // codebase that need to go away; these are artifacts from the fact that
    // D7's render API wasn't implemented fully. :(

    $markup = theme('item_list', $items);
    // D7 REALITY. Though we can delegate the generation of markup to later in
    // the execution stack, these arrays can be come a free-for-all of #keys (to
    // see a good example just add an image field to a node and look at the 
    // devel render krumo output http://lb.cm/arrays-of-doom).

    // Furthermore, if anything inside $items is a further render array,
    // any variables its preprocessor invokes are not easily accessible in
    // a parent template.

    $render_array = array(
      '#type' => 'item_list',
      '#items' => $items,
    );
    $markup = render($render_array);
    // POTENTIAL D9 SOLUTION. A deferred factory. `Theme::create` specifies a
    // class `ItemList` that eventually will be invoked (this is so the class
    //  can be altered) and some arguments for the resulting class.

    // PHP's magic methods offer potential gains. We envision __construct()
    // working similar to a preprocessor, and __call() as an accessor method
    // that could use helper methods to get properties. Much TBD.

    $renderable = Theme::create('ItemList', array('items' => $items));
    $markup = $renderable->render();

    // Ordinarly we'd simply cast the renderable to a string, but PHP's
    // __toString() method cannot catch thrown exceptions until (at least)
    // PHP 5.5.

Remaining tasks

Wishlist (courtesy of Moshe Weitzman)

  1. Make it faster Don't make it slower
  2. Don't break caching

User interface changes

None, likely.

API changes

Sweeping, likely. Render arrays will be replaced with OO architecture. @todo More details

Comments

c4rl’s picture

I have been working on a proposal for revising Render API, and part of the proposal covers some of the ideas here. I hope to have a draft for review in the next few days.

One tenant of the proposal (as of now) is that the DX should remain relatively the same. That is, the API will still consist of render arrays, but the private internals would be where the OO architecture is used (here's a vague, conceptual diagram of what drives my current thinking: https://dl.dropbox.com/u/21427810/render.pdf).

So, for most developers' purposes, they wouldn't have to touch or instantiate objects directly, but rather render() would be a wrapper for these.

Revising Render API is a larger scope than what is covered by this present issue, but they could become convolved. I'll post any relevant updates here.

Crell’s picture

I would offer a counter point:

- Render API's purpose was to enable hook_page_alter() and friends. Those are already gone, killed by WSCCI and to be replaced by SCOTCH.
- We're moving to block-level page layout anyway, which is a much more manageable level to deal with.
- Render API has a performance cost. Moving it to OO, while cleaner, would not eliminate that cost and would likely increase it since method calls have a higher marginal cost than array accesses (and ArrayAccess interface is slower still). The real cost is that we're building an Abstract Syntax Tree at runtime, and then parsing it. Really, that's insane.
- Twig offers us the ability to use *any* object as a renderable-ish object, because of auto-escaping.
- Render API's DX is atrocious, inscrutable, and down right bad. Panels and Views bypass it entirely. I try to whenever I can, and there's parts of Drupal 7 I try to avoid having to deal with just to avoid Render API. And that's for PHP devs; we have already established that themers don't grok it either.
- Over-abstracting things in the name of flexibility has bitten us on the butt, hard. We should learn from that lesson.

Is render API even needed? With the newer, more powerful tools available to us and the shift from page-level to block-level thinking (even if that doesn't quite make it into core for D8, it will be in contrib and I fully expect it to be the default way that professionals actually build things) why bother putting OO lipstick on that pig? Just kill it and fry up some bacon.

c4rl’s picture

@Crell Thanks for the feedback. I, for one, am guilty of unfamiliarity with SCOTCH, at least in terms of the specifics of what a page callback returns and how it is intended to be handled. I'll get up to speed with SCOTCH to refine some of my ideas as they seem to cover similar ground.

Crell’s picture

SCOTCH is the "Blocks and Layout" initiative, led by Kris Vanderwater, spun off from WSCCI earlier this year. Readers digest version: The entire page is just a series of blocks and is rendered "top down", not bottom up like we do now. It's more akin to Panels. Moreover, all blocks are, or can be, rendered in separate requests using ESI, hInclude, or whatever. That means every block can be cached by HTTP, the way HTTP is designed to do. That means completely separate PHP processes, which in turn means that in the typical case the entire "page" never actually exists as a thing in memory at one time. Hence, no hook_page_alter(), and likely no good reason to have a hook_block_alter, either.

I think somewhere around half of the work that has gone into Drupal 8 is in some way related to that architectural shift. :-) We're close to being able to do it, but I fear feature freeze is closing in on us too fast for my taste...

jenlampton’s picture

@Crell, I 100% agree with your comment. To be clear, we are not recommending the use of Render API. We are actually trying to remove as much of it as possible from D8 (hopefully the only thing that will be left at the end of the day is forms). But, that does not remove our need for "renderable" objects.

The real goal is just to get structured data everywhere. Data that Twig (or any other theme engine, for that matter) can use to render anything. That's what we mean here by "renderable objects".

Crell’s picture

In general I agree with having more clearly structured data rather than anonymous arrays. That's good, and good for Twig. However, we have taken that to an absurd degree in the past. Let's not replace it with an object-oriented absurd degree instead, as that doesn't actually buy us anything but a slower runtime.

The reason that I shout from the roof tops to not use db_select() and to use db_query() whenever possible is that db_select() is, for all its awesome flexibility, demonstrably much slower: #1067802: Compare $connection->query() and $connection->select() performance. The same thing applies on the front-end, too. A "renderable object" is all well and good, but that extra, probably rarely-used flexibility comes at the cost of substantial performance impact. It's a tool to be used surgically, not as a typical case. Even if we are able to get big-wins from HTTP partial-page-caching via WSCCI/SCOTCH, we should try to keep our front end-generating code as performant as possible.

Surgical, targeted use of "render objects" I'm completely on board with. But not just s/render api/render objects/, which is what it sounded like the OP was suggesting.

c4rl’s picture

I have been working some revised ideas for improving render api (or changing it into something else). A lot of this work wasn't quite ready for integration into core as I had to work through the concepts, so I have much of my thinking now visible here: https://github.com/c4rl/renderapi

Please give this a read if you are interested in render api and what potential renderable objects may entail. I am eventually interested in getting the ideas working with Drupal via a sandbox project.

c4rl’s picture

Issue summary: View changes

Updated issue summary.

msonnabaum’s picture

I disagree with every performance-related argument in this thread.

jwilson3’s picture

The two performance points that I see being made by Crell seem sound:

- Render API has a performance cost, converting array structures to Objects will increase that cost.
- Moving towards an OO system to represent pieces of html to be rendered to the page isn't best use of resources if having that flexibility in place to alter the object's structure/attributes/appearance will be rarely used.

Not to sound crass, but can you back up your disagreement with qualifications or counter arguments?

msonnabaum’s picture

The original points are completely unqualified to begin with. It's a very rare case that we should ever consider the performance difference between arrays and objects, and any situation where we would requires careful measurement. Making these kinds of assumptions is dangerous.

Fabianx’s picture

The original points are completely unqualified to begin with. It's a very rare case that we should ever consider the performance difference between arrays and objects, and any situation where we would requires careful measurement. Making these kinds of assumptions is dangerous.

That is interesting.

In general I would agree, however unfortunately already in that little Twig we have committed doing all in Objects without the optimizations I have made would have made things much slower.

That might have been the nature of ReferenceObjects, but I also saw several times the performance implications of Attribute objects (I love them still).

Therefore while I would be happy to say that Arrays and Objects are equal in terms of performance, unfortunately other people in core don't share your pov and benchmarks also show that unfortunately there is a difference. Maybe with PHP 5.4 this gets much better ...

Thanks for your encouragement though. :-)

msonnabaum’s picture

That's not my point. My point is that the very idea that there's a performance issue to consider here and that it might be enough to not even attempt this is the definition of premature optimization.

If we make a change like this, the code path is likely to change some. None of us can easily predict what that will look like on the other end. Developers (including me) are *terrible* at predicting where performance bottlenecks will be. We need to avoid doing that in cases like this and just measure.

Fabianx’s picture

#12: That is a very good point. I apologize for kinda blocking this based on this assumption.

And yes, you're right: This is premature optimization.

Thanks!

Fabianx’s picture

Issue summary: View changes

added table

c4rl’s picture

Title: [meta] Renderable Objects » [meta] Refactor Render API
Issue tags: +API change, +API clean-up

Given recent discussion at DrupalCon Portland I've fleshed-out the description greatly and retitled this to suit the original intention, so please review.

Much of the description is revised from a duplicate I had created. #1899454: [meta] Refactor Render API

effulgentsia’s picture

From the issue summary:

Ordinarly we'd simply cast the renderable to a string, but PHP's __toString() method cannot catch thrown exceptions until (at least) PHP 5.5.

This question came up several times at DrupalCon, so just for reference, here's the PHP bug report for it: https://bugs.php.net/bug.php?id=53648. So yes, we need a render() method, not __toString(). Fortunately, that's an implementation choice that has no impact on the .twig files: they still print with {{ }} without needing to worry about what PHP function is used under the hood.

effulgentsia’s picture

Issue summary: View changes

Replace description from 1899454

thedavidmeister’s picture

Issue summary: View changes

update link issue

thedavidmeister’s picture

thedavidmeister’s picture

Issue summary: View changes

Updated issue summary.

c4rl’s picture

Issue summary: View changes

Added 2005970

c4rl’s picture

Issue summary: View changes

Fix typos

c4rl’s picture

Issue summary: View changes

Fix Alex's handle typo

c4rl’s picture

Just posed a philosophical question on #2004872-12: [meta] Theme system architecture changes that affects this issue's potential.

thedavidmeister’s picture

Things that could be issues here:

- "remove post_process from drupal_render() as it breaks drillability"
- "replace pre_process with proper alter hooks in drupal_render()"
- "drupal_render() should preserve the original variables in #original"
- "drupal_render() should have 'default' inline rendering behaviour and a way for #types to declare that they don't need the theme system"
- "drupal_render() should be the function responsible for ensuring that renderable arrays are "drillable" by Twig"

thedavidmeister’s picture

Issue summary: View changes

Emphasis

Fabianx’s picture

@c4rl: I totally love this issue and +1000 to that.

The OO approach is very clean and will be a dream for developers to use. It will also naturally give things a structure so that there are no more questions open to where to add things. A collection can be a collection and things are typed. I am really looking forward to that.

I don't think it will happen for D8 though. We spent months on the rather simple task of converting things to Twig.

I don't think it is necessary for D8 to happen as long as the examples in the issue summary work, i. e. that things are drillable. This is our requirement and there are many ways to achieve that. Some might involve OOP, some might not, others might involve some OOP.

What I also very much like about this issue is: Even if it might not make it till the end and some of it is postponed to D9, a lot of this issue is very very important like killing the theme() invocations in core.

So please continue the fine work! We're close to having a great theme system in D8 - one way or another!

thedavidmeister’s picture

Somebody should really open a "drupal_render() should be responsible for ensuring renderable thingies passed in as arguments are drillable" issue, since we can anticipate that themers will want and expect rendered elements that don't go through the theme system to be drillable too - ie. form elements.

c4rl’s picture

#20: @Fabianx

Good to hear. :) The more time I spend on this, I have looked at this the bigger and more complicated it is, and overlap with things like hook_element_info(), entityNG, etc. Sub-issues like #2006152-1: [meta] Don't call theme() directly anywhere outside drupal_render() are already big, and definitely a good first step whether OO or not. There's still a lot of Drupal 6 still around.

I came to the same conclusion as you: We need to get closer to drillability for D8, but likely won't have time for a refactoring of this scale. Twig (mostly syntax changes rather than API changes) took a long time (#1499460: [meta] New theme system was filed > 1 year ago).

I also realized this scramble we suffer due to the core release cycle -- that we are "stuck" with an API for 3 years (or longer). I re-watched @heyrocker's core convo from Portland http://portland2013.drupal.org/node/3863 Making Core Development Sustainable. Many core devs want to change it, so hopefully D9 will be closer than we think.

I definitely want to keep the momentum here putting together some design/DX ideas for D9. In the short term, let's forge ahead on the registry issue #1886448: Rewrite the theme registry into a proper service and the hook refactoring issue #2004872: [meta] Theme system architecture changes; those help with cruft and are a closer D8 reality.

c4rl’s picture

Given the size of this task, it is likely that we will not be able to accomplish the entirety of the long-term OO goals prior to D8 code freeze.

Thus, we've added a sub-issue to this as a stopgap for drillable structure pertaining to Twig. #2008450: Provide for a drillable variable structure in Twig templates

I'm going to leave this as active for now so that the sub issues don't get ignored, but effectively we should look to solving the must-have stopgaps on the original roadmap first. https://drupal.org/sandbox/pixelmord/1750250#roadmap

c4rl’s picture

Issue summary: View changes

Acknowledge reality.

marfillaster’s picture

symfony twig bridge bundle provides similar functionality to render out of order form elements.

basically the following are possible:

{#render all form elements, labels#}
{{form_widget(form)}}
{#render a single element, labels and errors has to be manually rendered#}
{{form_widget(form.field_1)}}
{#behaves like form_widget but skips previously rendered element#}
{{form_rest(form.field_1)}}

in addition, it also applies twig themes on each child elements
https://github.com/symfony/symfony/blob/master/src/Symfony/Bridge/Twig/R...

this is the twig extension that provides the functions
https://github.com/symfony/symfony/blob/master/src/Symfony/Bridge/Twig/E...

relevant docs

http://symfony.com/doc/master/book/forms.html#form-theming

thedavidmeister’s picture

Title: [meta] Refactor Render API » [meta] Refactor Render API to be OO

just updating title to more accurately reflect the discussion here.

webchick’s picture

Version: 8.x-dev » 9.x-dev

I can't really see doing this anymore in D8; we're well past API freeze now.

webchick’s picture

Issue summary: View changes

Add drillable issue

c4rl’s picture

Issue summary: View changes

Some tweaks to issue desc.

thedavidmeister’s picture

Issue summary: View changes

Are people still following this issue? Parts of the rendering process have been slowly turning OO but there's been no activity here in a while.

jessebeach’s picture

Given that this has been marked as 9.x-dev I gather that there's not enough free attention to give to it at the moment. I too hope that we pick back up with this issue once we 8.x finds stability.

thedavidmeister’s picture

fair enough. I thought maybe someone would be linking issues that are already being worked on, but I guess they will do that later.

sun’s picture

larowlan’s picture

star-szr’s picture

Status: Active » Postponed

Postponing because 9.x.

catch’s picture

Version: 9.x-dev » 8.1.x-dev

We should see what's doable in minor releases here.

We already have https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Render!Renderable...

andypost’s picture

Status: Postponed » Active

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

pwolanin’s picture

Please update the issue summary and add related issues to the issue metadata

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

geek-merlin’s picture

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

geek-merlin’s picture

Sigh. Even a simple self-rendering wrapper would save us from the critical twig sec issue: #2860607: Code execution via Twig templates (including inline)

markhalliwell’s picture

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.