This issue is postponed on #2759849: Create a new user-facing core theme. See #67 for details.

Problem/Motivation

Drupal's render & theme system are too complex to use. Let's improve this.

Goals

  1. Improve the TX by un-WTF-ifying the theme and render system, which is currently a maze of:
    • hook_theme() (with variables vs render element), preprocess hooks, theme suggestions and many more related hooks — all tied together using the theme registry
    • #type (@RenderElement plugins) vs. #theme, which are kind of the same thing but not really: when to use which is unclear
    • callback buffet: #pre_render, #post_render, #lazy_builder
    • mysterious keys in enormous render arrays AKA render arrays of DOOM
    • #render_children, #theme_wrappers and friends determining where the render system morphs into the theme system and back again
    • different systems calling each other: understanding the entire flow is nearly impossible, and probably rivals the complexity of some simpler biological organisms
    • better documentation and and examples are band-aid solutions — they address the symptoms, not the cause — @c4rl

    These have been known problems for years! First there was Form API, then Render API sprouted from that, and all the while there was the theme system, but starting in Drupal 7, the Render API and the theme system got deeply, deeply intertwined. Drupal 8 actually made it slightly better, but not enough.

  2. Improve the TX for non-JS front end people (markup & CSS people):
    • automatically generated pattern library (== all #type/@RenderElements — but without having to know those details)
    • automatically generated style guides for every theme (== pattern library with the theme's overrides/extensions applied)

    in other words: bring style guide-driven development to Drupal as a default rather than a labor-intensive, hacked-on after-thought (without the need to duplicate markup and thus keep them in sync).

  3. Long-term: make it possible to reuse templates on the client-side (JS people).
  4. Retain compatibility with the existing Render & Theme systems. Allow for a gradual transition.
  5. Support interface previews: #2632750: Interface previews

Requirements

To address all of those goals, I believe a component library can be the gateway to sanity a solution. It can be, if the following requirements are met:

  1. components are not deeply tied to Drupal, and in fact, can be developed independently of Drupal — this is how we can guarantee simplicity and ease of getting started: we actively prevent components from being tightly coupled to Drupal code
  2. components have:
    1. markup: *.html.twig (Twig template — which may include some logic to process received variables, just like in Drupal 8)
    2. assets: CSS, JS files
    3. metadata: YAML file

    Nothing else.

  3. modules and themes can specify components — modules can define patterns that any other module can use, themes can specify theme-specific components
  4. components are defined in a simple directory structure:
    <extension> (module or theme)
     |- components
        |- <component name>
           |- <component name>.yml
           |- <component name>.twig
    

    Concrete example:

    core/modules/system      
     |- components           
        |- label             
           |- label.yml      
           |- label.html.twig
           |- label.css
    cores/themes/classy      
     |- components           
        |- label             
           |- label.yml      
           |- label.html.twig
           |- ajaxified-label.js
    
  5. The YAML file specifies:
    1. the variables (inputs) of the component. For each variable:
      • type: only A) primitives such as string/integer/bool, B) arrays of primitives such as string[], C) other components: component or components to slot in other components (perhaps even component:<name> to only allow certain components ) — this enables 3 big wins:
        1. improved TX: type validation, to avoid weird bugs
        2. improved TX: no messy Doxygen/PHPDoc comments repeated in both templates and preprocess functions, and all overrides of either of those
        3. client-side re-rendering

        (Also: having type specifications in Twig templates instead is A) undesirable, B) quite likely impossible, C) quite likely impossible to parse without refactoring Twig, D) would pick up calculated variables.)

      • description
      • default value, if any
      • example value (to be used in pattern library & style guide)
      • preview value (to be used when the component's data is not available yet, because it's being used for an interface preview)
    2. documentation: purpose, when to use, how to use, accessibility, related links — in other words: information to show in the pattern library & style guides
    3. less important metadata: human-readable name, which other components this component includes, whether this component supports interface previews …
  6. The Twig template (*.html.twig) performs all the necessary processing of the variables received, this ensures we don't depend on preprocess functions. This removes the need for front-end developers to dive into PHP.
  7. components can be extended: add attributes, modify markup, and so on
  8. components can be composed: combine multiple components to create a new component
  9. To allow for a gradual transition, we cannot fully back away from render arrays nor the existing theme system. At best, we'll be able to remove render arrays, the current render system and the current theme system in Drupal 9.

Proposed resolution — or: how to transition

Aspect 0: components
Let components be defined as explained above.
Aspect 1: '#component' => …
Allow render arrays to use components by using '#component' => 'something', instead of '#type' => 'something' or '#theme' => 'something'.
The associated CSS and JS assets have been registered as an asset library automatically, and this asset library is attached automatically. Just define it in the YAML file.
This is familiar and similar. The transition is simple and understandable. But why would you want to do this? See the next few aspects.
Aspect 2: strict validation
Let Renderer strictly validate '#component' => … render arrays.
Remember that a render array is effectively a tree. Whenever a node/element/thing in the tree is a component, we can be more strict for just that one node/element/thing, but not for its supertree (parent) nor for its subtree (children). Example:
[
  '#type' => 'something'
  'child' => [
    '#component' => 'image',
    '#uri' => …,
    '#width' => …,
    '#height' => …,
    '#alt' => …,
  ]
]

In this case, the properties (#-prefixed values of the image component will be strictly validated: only the ones specified by the image component will be allowed, and their types will be validated). This allows us to bring sanity to render arrays, one component at a time, one conversion at a time.

Render arrays have certain special/reserved properties that are necessary for them to function correctly. Think #attached, #printed, et cetera. Those will be condoned, because they don't affect the way the component is rendered.
Aspect 3: opt-in eligible #type and #theme to become #component
Help with a graceful transition, and peaceful coexistence. We gradually remove #type => something and #theme => something and replace it with #component => something But it is hard/impossible to keep all contrib code working: we can't expect everybody to simultaneously change existing render arrays to use '#component' => 'something' as soon as there is a something component.
So, whenever the Renderer encounters #type or #theme, we map it to #component if and only if they are eligible, which is:
  • if it was #type, we only do the mapping if no #pre_render/#post_render/… are defined on the render array
  • if it was #theme, we only do the mapping if no preprocess hooks are defined for it in the theme registry

Finally, if it wasn't #component originally, we don't apply our strict validation.

Aspect 4: new things must be components
New patterns in Drupal core are implemented as components.
Eventually
Eventually, when all #type (@RenderElements) and #theme occurrences are gone from Drupal core, we drop the old system, and we tag Drupal 9.
At this point, we will have a render array representing the entire page. The render array representing the entire page is then once again a tree. But rather than an incomprehensible tree, it is a well-defined, well-structured tree. It can even be split up into two parts that fit into each other perfectly:
  1. a component tree: html → page → region → block → … — and for each of those components, there are holes (where the variables go)
  2. a variable tree: html variables → page variables → region variables → block variables → … — the variables on each of these levels fit into the holes in the component tree

(Conceptually speaking, because subtrees can of course still be lazily built. Details in that area TBD.)

At this point, render arrays are just an implementation detail, and we can easily simplify that implementation.

Long-form rationale

This part is less precise, but tells a hopefully helpful story of the two most important considerations: how we ended up with the current painful system (recommended reading: http://hackingdistributed.com/2016/04/05/how-software-gets-bloated/) and how we can at the same time start to make it possible to integrate with JS more easily.
And, how, funny enough, those actually need the same fundamental thing: simplicity.

JS: web apps vs web sites

"Apps" are the hot thing in software this decade. But building native apps for every platform is very expensive. So "web apps" are a thing: build once, run anywhere, deploy instantly.

Web apps must be written in JS (or at least compiled to JS), so, consequently, JavaScript is the hot language of this decade. As a result, we've seen enormous investments in JS: Node.js, browser engines' JS interpreters have become incredibly fast, and … JS-based frameworks. From jQuery UI to Angular to Ember to React to …

Everybody wants "an app". And so quite often what would have been a web site a few years ago now is a "web app". And since web apps are written in JS, this means Drupal is less likely to be chosen for those scenarios.

(I personally think the distinction between the two can be made by determining whether the business logic happens in JS on the client or on the server. Only if it's on the client, it is a "web app".)

But that doesn't mean there's no more need for "regular" web sites anymore. It doesn't make sense to build an app just to present hyperlinked information… because for that, we already have an app: the browser. We just need to feed it web sites: documents of structured content, with excellent accessibility & usability, beautiful layout & typography and most importantly: great information architecture.

(And indeed, there is a very, very blurry border between "web sites" and "web apps". I'm just trying to paint a picture of the two extremes, where the world of the web is still finding an equilibrium somewhere in between. And of course, in some cases, it makes sense to have parts on the client side and other parts on the server side.)

Drupal 8 is even better at building web sites than prior versions.

But Drupal is getting pressure to also support "web apps".

So we have a tension between wanting to improve Drupal for what it has traditionally done (improve front-end experience by improving its templates, its markup, removing preprocess functions, etc) and making it possible to build more app-like experiences with Drupal.
But Drupal is not written in JS. Perhaps at some point in the future, there will be tight integration. But we have no idea what that would look like, if it will happen.

Getting ready for the future, and facing the past

However, even with the current system, we have long-standing problems. Extremely frustrated reports go back to at least 2011. The theme system and render system are deeply intertwined. It's very confusing. The experience even to build just web sites (not web apps) is far from ideal. Drupal 7 introduced the render system, but let's not forget its origin: Form API. The render system was originally just for forms, but it's since been generalized to be used for all rendering.

  • Drupal 6: Form API + theme system
  • Drupal 7: Form API + theme system, and both depend on the render system

Drupal 8 has made big steps forward: Twig, removal of most preprocess functions, much cleaner templates, #theme is gone in favor of #type (but not at all completely…), no longer necessary to to sometimes call render() in templates …

Unfortunately, as soon as the render and theme system interact, it's still painful.

We started working on an experiment at Acquia, where we worked with the Angular & Ember.js teams to do a prototype of what a better commenting experience for Drupal would look like. A reference implementation in Drupal using the AJAX system ("the Drupal way") versus what they would do. They had to reuse our Twig templates. A big problem there was the fact that just about every template has "blobs of HTML": {{ content }}, which actually contained the majority of the interesting stuff. And those blobs of HTML are… yes, render arrays!

Of course, no JS is ever going to be able to render render arrays, because they're so deeply intertwined with PHP code. And it's impossible to automatically determine which Twig templates are associated with every subtree in a render array.

So this makes it effectively impossible to reuse our Twig templates in JS. Ideally, that would be possible, it'd make Drupal better prepared for the future. It's better to at least have that possibility than not to.

However, even today, and in fact, in years past, this very same problem has been a major frustration for themers: they could only go so far with achieving what they needed to achieve by creating/modifying templates and writing preprocess functions. Very often, they would need to dive deep into render arrays and implement lots of hooks.

Imagine if that weren't the case, and we'd have templates all the way down, rather than enormous blobs of the resulting HTML being defined in render arrays. Imagine if the resulting HTML was wholly based on templates. Templates all the way down. Power to the themers.

And imagine that rather than ill-defined variables, they'd actually have type hints. And validation. And examples. And actually usable documentation rather than incomplete (and duplicated) docblocks. Together, that would allow us to automatically generate a style guide/pattern library.

As a bonus, client-side (re-)rendering becomes possible.

References

A close-to-comprehensive list of the references I've used to write/build the above.

TX
Components
CSS/style guide
Other

Remaining tasks

Lots.

User interface changes

None.

API changes

None, only additions.

Data model changes

None, only additions.

Comments

Wim Leers created an issue. See original summary.

Wim Leers’s picture

I built a proof-of-concept that proves all of the above is indeed possible. The patch should apply cleanly to the commit tagged with 8.1.0-rc1.


Please look at this patch from two angles:

Front-end developer/themer
Look only at the files in /components/ subdirectories: the YAML and Twig files. And look at the attached screenshots. Compare how for example the pattern library screenshot never shows any assets, but the style guide screenshot does, because Classy's extension of the form-element-label component adds a CSS file.
Core developer
Everything else, which effectively amounts to looking at ComponentDiscovery and the places where it's called/used to integrate with existing systems with as little code as possible: the theme registry, asset library discovery, Renderer.

What's in this patch?

Preparation
I started by working on removing some templates' preprocess functions, because components must be self-contained, so this logic would have to live in Twig templates:
  1. #2694179: Remove template_preprocess_image()
  2. #2699635: Remove template_preprocess_form_element_label()
  3. #2701667: Remove template_preprocess_responsive_image()
Convert to components
Next, I converted image, container, form_element_label and responsive_image to components. Why those? Because they're a good mix of not having nesting or not (image vs container), being omnipresent and used by Form API (form_element_label), and showing the ability to compose/reuse components (responsive_image reuses image).
I left the old templates untouched, to keep this patch as small & digestible as possible. I only removed their hook_theme() entries, which effectively makes the system unaware of the existence of these templates.
Make components actually work, and prove that a graceful transition is possible
This patch hijacks integrates with the theme registry, Renderer, asset library discovery, provides a new YAML discovery mechanism to allow the simple directory structure outlined above to be used.
Automatic pattern library & per-theme style guide
See the attached components-pattern-library.png & components-style-guide.png
What's left TODO?
Plenty! There are plenty of details to figure out, plenty of things I haven't done yet: variants (e.g. imageimage--with-rollover), how to take care of lazy builder integration, how to handle #theme_wrappers, #prefix, #suffix. And much more. I also stumbled upon existing problems/bugs/limitations, such as #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance.
I wasn't able to make the responsive image component fully functional, because its preprocess function is… extremely complex. Once #2701667: Remove template_preprocess_responsive_image() is done, I can finish that. So that particular component is not fully functional.
So…
… please see this as a proof-of-concept that tries to provide a good starting point and with as many of the critical considerations as possible already incorporated.

And now… looking forward to all your feedback!

eatings’s picture


Imagine if that weren't the case, and we'd have templates all the way down, rather than enormous blobs of the resulting HTML being defined in render arrays. Imagine if the resulting HTML was wholly based on templates. Templates all the way down. Power to the themers.


This was the heady future longed for in the early days of the Twig project, and to some extent the commonly perceived (if inaccurately so!) main benefit of going Twig.

That we aren't there yet even with Twig feels much like a promise yet to be fulfilled, and at least for me why all the improvements in D8 were still something of a half-measure.

In the end, I love a lot of these ideas, perhaps this one most of all.

Wim Leers’s picture

#3: Twig and everything related to it is what makes this issue possible. It's why I started a new issue. Because we did go forward quite a bit, we untangled quite a lot, and so now we can/should/need to take it to the next level.

Wim Leers’s picture

Issue summary: View changes

Fixing a bug in the IS (something that I forgot to write in the IS but that is in the patch — see if ($expected_type === 'component' || $expected_type === 'components') {), thanks @mdrummond for pointing that out!

Fabianx’s picture

Great proposal!!!

---

So some things with my theme maintainer hat on:

- I would not do the same mistake and define just '#' keys,

I would do:

$build = [  '#component' => new Image('file.png'); ];
$build['#cache']['tags'][] = 'foo';

And then have a getProperties() method for a generic ThemeComponent.

In case we want something more loosely coupled I would do:

$build = [
   '#component' => [
     'type' => 'image',
     'alt' => '..',
   ],
   '#cache' => [
     'tags' => ['foo'].
   ],
];

So every component has its own strict properties within the component space or is directly an object.

That means other render array properties do not interfere and obviously component objects could implement CacheableInterface - if they wanted to.

-----

One of the biggest problems "How to traverse the tree" is not yet defined enough IMHO, because how do you get from a e.g. block component to its content and what _is_ its content?

Is the title of the block also a sub-tree? Or is it a property?

Based on the AmberJS example for using Twig in JS, that seems key to me to define that properly.

-------

And for BC reasons I would still convert:

#theme => 'image',
#alt => 'x',

to

#component => [
  'type' => 'image',
  'alt' => 'x',
],

internally and then just taking defined # properties and ignoring all others. In fact that is sometimes that the theme system originally had (variables => ...) and which then vanished through the render array 'render element'.

---

But great proposal and good idea to make this the thing within D8 cycle, then remove all deprecated things in D9!

+1

dawehner’s picture

I love the proposal. One relative simple concept which is known by many people and have a similar mind model in other areas. Just think for example of panels as a UI for defining bigger components and merging those together. Once we are there we could maybe use this component building metadata to build similar experiences in JS with whatever system you want.

There are a couple of fundamental questions which are IMHO more important than the concrete way of suing YML files and folders ...:

  • How far do we want to get with form elements, which currently how more logic than just render elements. Do we want them to be components as well?
  • IMHO the most tricky question on which the entire WSCCI/SCOTCH initiative stopped: How do we apply this mapping of the available data to the components. As far as I understand the proposal it would still be passed in manually by the code and anything UI related would come later?
  • How do we reuse existing components within another component/twig template?

Some other ideas what this proposal would allow:

  • Formatters could be also just components for a field item or a field list (depending on your needs)
  • The entity display configuration could be just a configurable component using field formatter components
  • A panel would be just a component UI, which uses some "layout" components and some more special components which contain the actual information.

Note:
In an ideal world the component discovery would return value objects (much like EntityType) which would live inside a Drupal\Component namespace, so we don't make the mistake to couple stuff too early.

stevector’s picture

Wim,

Thank you so much for pulling together the disparate conversations and requirements. You've summarized the situation beautifully. The backwards compatibility layer is especially important.

After reading through the patch I have a lot of detail questions but I'll save those for later and start with the bigger picture.

Do you see Components as defined here as an alternative to Layout Plugins? #2296423: Implement layout plugin type in core In the presentations and writing I've done on the topic I've said that Layout Plugins are the closest thing we have components in the Drupal ecosystem. Marc also summarized the idea in the blog post you link above. I know there is a long way to go on this patch but you've gotten incredibly far already. I suggest postponing adding Layout Plugins to Core until it can be determined whether components as they are defined here could be used in Display Suite and Panels. I'm guessing the answer to that question is "yes, with some tweaks (and maybe a compatibility layer between components and layout plugins)"

---------------

A big problem there was the fact that just about every template has "blobs of HTML": {{ content }}, which actually contained the majority of the interesting stuff.

I agree! This Drupal-WTF is one of the worst parts! I see how converting some theme functions to components has the chance to clean up this problem by breaking "content" into other variables. But for a template file like views-view-unformatted.html.twig I think it would still have {{ row.content }} where this variable is printing a component.

People would still do a similar type of digging to find out what happens inside of that content variable. Or do you envision something else that would better highlight the "templates all the way down" model that David also quoted in comment 3? When I've used Twig outside of Drupal (in Sculpin) I've done things like this in an image gallery twig file:

{% for image in images %}

  {% include 'image.html.twig' %}
{% endfor %}

That approach make it very comprehensible to the person opening the file which template file is used. Specific variables can be passed to the include. Twig's own namespacing system even allows for overriding image.html.twig without changing the include statement. Do you see a way that this component system could highlight, in a similar way, the usage of a component by a component?

catch’s picture

Overall I really like this. One issue not mentioned in the issue summary is #2060783: Remove the preprocess layer.. I was very sad we didn't get further with that before 8.0.x, and adding #component which does not support preprocess solves the bc problem very nicely. Additionally we make no guarantees per the backport policy as to what any particular render array or form array contains, so if they start to include #components that's completely doable. So it feels like a great way to continue unfinished business with the render/theme systems in minor releases. The issues to remove individual preprocess hooks by moving the logic to the templates are also encouraging in that regard (once I get past the Drupal 6/7 knee-jerk aversion to logic in templates).

@dawehner's three questions in #7 and Fabian's in #6 included two of mine I was typing out before I refreshed the page - the use of other components within a component, tree traversal (this is where 'everything is a block' and 'HtmlFragment' ran into serious trouble), as well as how much this gets applied to form elements.

The other major concern/question I have that's not mentioned above (unless I missed it, in which case apologies for restating):

There were two main ways we tried to handle conversions away from theme functions and php template to Twig in the 8.0.x cycle:

1. one-to-one conversions to Twig
2. many-to-one conversions via adding new #types, or using existing #types with theme suggestions, #1804614: [meta] Consolidate theme functions and properly use theme suggestions in core is the meta.

I had a vague hope that if we took #2 to its logical conclusion, we'd end up with something resembling a component library.

This was going OK, if slowly, for a while, but then mostly ran aground at #1876712: [meta] Convert all tables in core to new #type 'table', where we eventually had to fall back to converting individual table theme functions to individual Twig templates in issues like #1938912: Convert language content setting table theme to a twig template. This was due to a mixture of complexity of the conversions and the inability of #type table to handle every case, and lack of time compared to the rest of the release.

#1930840: [meta] Remove or consolidate many similar admin page theme functions in core and #2076301: Remove views-mini-pager.html.twig, use a pager theme suggestion instead are good examples of in-progress issues not related to tables.

Looking down https://api.drupal.org/api/drupal/core%21modules%21system%21system.modul..., it would be sad to have a system_modules_uninstall #component - or at least if that's not sad, it'd be good to document why it's not. There are lots of other examples.

If we agree it would be sad, we need a plan for narrowing down the large number of #types and #theme hooks to a smaller number of #components. Both in terms of how to do the implementations (a conversion of #type table to #component table now would be missing functionality necessary to convert #theme tables to #component tables), and how in the backwards compatibility layer for #theme would work if the specific #theme hook does not map 1-1 to a component.

Bringing this back to the issue summary:

- it mentions extending components, is that instead of, or in addition to, theme suggestions? (note the change to views_theme() in the pager issue linked above).
- Aspect 3 explicitly talks about a bc layer for 1-1 #type/#theme -> component conversions, this doesn't account for where there should never have been a #type or #theme in the first place and instead we re-use a different #component. I can't think of a way to resolve that except for a mapping somewhere which gets referred to if a #component can't be found.

catch’s picture

Forgot to actually include one of the main points of the last comment:

If we want to ensure that this eventually reduces the complexity of the theme and render systems rather than just adding yet another #thingy, then we should tackle tables earlier rather than later.

larowlan’s picture

Great ideas here - one other place these could be used is to auto-generate ckeditor widgets, so elements in the styleguide could be used by content-editors too.

jhodgdon’s picture

andypost pointed me here... I am working on #2697791: Add Plugin system for abstracted HTML-formatted text, which is in a way related. On #2697791-13: Add Plugin system for abstracted HTML-formatted text I just did a quick analysis of that issue vs. this one and would love it if someone could look that over and see if it is a fair assessment? I don't think they're replacements of each other, but if my issue should be a duplicate of this one (in other words, if these new components can be used for my purpose over there somehow), I'd love to understand how that could be, and/or help convert/create components so that I could use them for the help topics thing I'm actually working on.

Also looking for reviews of that patch. ;)

davidhernandez’s picture

One thing that concerns me is trying to move all of the logic into the template. I think for 90%+ cases we can do that, but it may not be a good idea to completely remove the pairing of functional code that we have now with preprocess. It isn't just cringing at the idea of logic in templates. We tried this already with some of the more complex use cases and failed. As catch mentioned tables are one, and views are another (especially if we continue to support altering markup and adding classes in views ui.) I wouldn't be surprised if forms find a way to throw a monkey wrench in too.

If the component's body parts all live together in one place, would it be possible to also provide an optional php file with the component that would contain its preprocess function? Or maybe it isn't even preprocessing that is needed since the component will be fairly strict about its data, but instead the component might need a helper function to do heavy lifting.

Alternatively, maybe the use cases are the problem (with the troubles we've had being a symptom of a bigger problem) and we need to rethink how we do them completely.

I'd like to add a property to specify where the component comes from. We rely entirely on inheritance now, which works well most of the time, but causes some problems. Being able to specify not just the type of component but its supplier might help with some of the admin theme/frontend theme issues we have. And also the whole Classy/Stable split.

Related to that, versioning would also be nice. The proposed workflow gets us to 9, but I don't think solves the core problem of templates/components being frozen. If we can version them, and create component libraries that have their own configuration, we can version them too. That solves the BC problem.

fago’s picture

In shot, big +1. This is is overally really needed and critical and imho the right path for allowing client-side rendering.

A few comments:
- Imo, this really needs to be developed decoupled. First off, the components need to spec-first, with a reference implementation in php. The PHP implementation needs to be pure PHP (i.e. a composer package) and the pattern library / styleguide generation needs to work without Drupal. That enables a front-end team to work/start working without Drupal easily. Something like https://sculpin.io/ can be a good start here.
- The component tree needs to be clearly specified as well, such that a re-implementation is easily possible. Glue/processing code is a bit problematic here, and probably one of the hard-parts that needs to be figured out.
- The spec should include have a deterministic, complete mapping to TypedData, such that components can be easily exposed as blocks / panels / ckeditor templates. (I'm working on TypedData based widgets for Rules what's something that should be useful her as well).

Imo, this would solve all problems you ran into for establishing a proper styleguide driven development workflow, like integrating with Drupals JS libraries etc. Decoupling that component code, means really decoupling Drupals front end from Drupal and is what's required for allowing a decoupled front-end development team to work *while* being able to directly use the result with Drupal. That would be a real game-changer.

Once a working spec and PHP component and Drupal integration is in place, I totally see companies pick that up quickly and let them help developing a complete coverage for all templates, such that a new base-theme completely built out of those components should be achievable in the rather near time future!

fago’s picture

Had a good discussion with dawehner about this. Just dropping some quick notes here before I forget details... :
- Implementation would have to be decoupled first, come with integration in Drupal and make re-using decoupled components possible. Then existing things can be converted step by step until everything is components...
- Extra layers of Drupalizm like entities and views can be added once a front-end impl is converted to Drupal. Those tools need to define / take over data mappings then similar as configuring a block in ctools or an action in Rules.
- Contextual links etc. could potentially be done by wrapping components / higher order components
- Problem of mapping data and passing data through is exactly the same problem as solved by Rules with contextual plugins and nested calls

Looking quickly at the patch, I think this mostly misses a decoupled implementation. Also, libraries need to be part of the component. Then we can allow extensions to provide components, but components need to work without Drupal also such that a frontend team can pick up / write things initially easily.

aleksip’s picture

Amazing work!

I have experimented with getting unmodified Drupal 8 Twig theme templates to work in the Twig edition of Pattern Lab. The end result is a decoupled implementation of a theme that can be worked on without Drupal. One of the challenges was how to handle Drupal's Twig extensions and the widely used Attribute objects.

I noticed that the example image component uses the file_url function and an Attribute object. With the proposed components being independent/decoupled, how are these handled?

andypost’s picture

For entities there's issue as well

mdrummond’s picture

Overall, I'm over the moon about this proposal.

This is very much along the lines of a big discussion that happened at MidCamp on moving towards a component-based theming system, which I wrote up in this post: http://www.marcdrummond.com/posts/2016/03/25/drupal-front-end-future

The key goal we discussed in that conversation was how to separate Drupal's data modeling from the appearance of a Drupal site.

Right now so much of Drupal theming is about decorating data models with markup. A component-based theming system would in theory allow us to move away from doing that.

However, the danger I see in the proposal above is simply translating all existing theme hooks and render element types into equivalent components. If we do that, we will have benefits, but we'll still have lots of components tightly related to data models.

I understand the desire to come up with a practical transition. I'm just leery of continuing to duplicate the same sub-optimal patterns. We already did this once when we moved theme functions one-to-one into equivalent templates. Do we really want to do that yet again with components?

------

Much of my thinking about components jives with Brad Frost's Atomic Design Constructs, which he breaks down as follows:

  • Atoms: The smallest level of components, often matches up with HTML elements
  • Molecules: A component that puts several Atoms together. Like a search box in a header.
  • Organisms: A component that puts several Molecules together. Like the entire header that includes the search box.
  • Templates: A component that puts together multiple Organisms into a full page. A site might have multiple layout templates.
  • Pages: An actual page that implements a given template. Like an About page for example.

For what it's worth, I think Organism is the fuzziest level in this breakdown. I've sometimes created another level I've called Super Organisms that can combine multiple organisms. But sometimes you need to combine multiple Super Organisms before you get to the Template level, so this can get pretty messy.

While all components are "a thing that can contain other things," I think it's useful to think about there being different types of components.

In terms of Drupal, I think it could be useful to think about components in a similar way:

  • Data component: An individual piece of data, essentially a field. So this could be one data item or a list of related data items.
  • Item component: A combination of multiple data components. An entity view mode is probably a good example of this.
  • Item series component: A combination of multiple item components into a list. Think a river of news. Often implemented with views (for automatic lists) or entity/nodequeues (for manual control of lists)
  • Section component: A combination of multiple arbitrary items or item series into a chunk of the page. Mini-panels is a pretty good equivalent to this.
  • Page component: A full page layout that includes specific sections (or an individual item or item series or maybe even a particular piece of data). Think a layout with a certain set of explicitly-declared blocks. Panel pages or panelizer would be good equivalents for this.

I could also see a label for something in between an Item and a Section like a Combo Component to help address the squishiness of the Organism level in Atomic Design.

Technically, each of these types of components could probably work in the same way. These labels are just a way to help think about the different ways components could be used. Labels like this might be helpful in user interfaces or a pattern library to help site administrators understand different types of components.

----

If you look at the templates folder for the Stable theme, you can find a breakdown of template types into the following groups:

  • admin
  • block
  • content
  • content-edit
  • dataset
  • field
  • form
  • layout
  • misc
  • navigation
  • user
  • views

That starts to help out sort how you might start mapping templates into components.

A lot of templates in field probably line up with Data components. block and content probably line up more with Item components. Views and dataset are kind of like Item List components. Layout templates probably would line up with Page components.

I do want to take some time to go through all our templates and try to do a breakdown of where they might fit into different types of components. I do want to point out though that some don't fit all that well.

A table template is not really a component, it's more of a markup pattern. Same thing with item-list. Maybe a Twig macro would be a better way to handle those sorts of situations?

----

I'm going to be writing up another post about one of our other discussions at MidCamp about different personas that the theme system needs to serve. For now, it's important to note that our theme system needs to serve two key uses: the Assembled Web and the front-end developer who cares a ton about Drupal's markup and uses that to implement a specific design.

A component-based theme system should likely have huge wins for front-end developers. Being able to define a template for an individual component that specifies, hey, this is the data I need for this template, and then keep all the markup for that component in one template would be amazing.

For that to really work, though, Data components would need to have some level of drillability. I don't want it so drillable that I have to reconstruct an img element every time I place an image in an Item component. However, I also want to have the option of skipping a generic field wrapper template, so if there is a data list, I can just put a ul and li for each data item within my Item template, then put the classes I want on that data list structure, or even on the individual piece of data. I'd rather not have to do that in several different templates in order to get the markup I want for a particular component.

However, this still needs to work for Site Assemblers who are using the UI to make appearance changes. So keep in mind that an Item component could include a data slot with one particular data item... or it might need to include one or more regions where several Data components could be dropped in and be ordered through the UI.

Ultimately, I think this system would be a huge win for Site Assemblers. This could be a much more outside-in model. Go to a particular type of page and then find some pre-configured components that can be dropped into various page slots. Add a section, and then add sub-components to that. Configure individual components as needed.

For Site Assemblers, it might be very useful to have some generic Layout options that can be chosen for components at different levels. One-column, Two-column, three-column, etc. Front-end developers will be much more likely to want to define very particular layouts for individual components. In essence a Layout is a generic way to say "hey, here are a certain set of slots you can put stuff into for a particular component."

----

One last thing to note. The definition of a component should include the context that it requires, essentially the data source and any data relationships between multiple data sources. An Item component may have several Data components specified, but it needs to get that data from somewhere. If that Item component is placed in an Item Series or a Section, then that larger component needs to pass that context/data source into the Item component. That context would then bubble up to the Section, so if that section is placed in a Page, the context/data source needs to be placed into the Section so that data can flow all the way down to the Data component.

I also think there should be some logic to translate that context/data into the individual pieces of data sent into a particular slot, whether that's called preprocess or whatever. I think it's great for templates to use logic to determine what class to use in markup, or even construct a class name from some source data. But processing a data source into data that should show up in that template? I don't think that sort of business logic belongs in a template.

Finally, that context/data source for each component is essentially how you can make this cacheable. It's the data sources that would define the cache contexts that this component should have.

----

Modules might end up being a common place to define components, themes might also define their own components. This probably goes without saying, but a theme should be able to override the markup/css/js in module-defined components.

----

Finding a solid way to transition this over the D8 cycle probably won't be easy, but I think we could get some huge benefits. A built-in pattern library? Totally possible.

This might end up having effects on other parts of Drupal as well. I'd love to see an Item Component Builder tool that would probably draw from elements of Views and View Modes. I'd also like to see an Item List Builder tool that would probably draw from Views and Entity/Node Queues.

Glad to see all the interest so far! This has me super excited about the future of Drupal's front-end.

Fabianx’s picture

Also re #18: Jacine wrote up a theme component library structure once here:

http://jacine.github.io/drupal/

Crell’s picture

Issue summary: View changes

Color me big +1 on this proposal. I've a few things to add, and a few things to echo and reiterate that others above have already said:

* Absolutely the right way to do this is in a separate stand-alone PHP library, independent of Drupal, and then integrate it back in. The fewer dependencies the better. (YAML and Twig are the only ones I see here.) Not just from a code-sharing perspective, but from a decoupling perspective. If we physically can't let higher-level Drupal tools leak in and pollute the implementation, they're less likely to do so inadvertently. That also helps keep the Twig templates "vanilla", and thus safe to ship client-side using Twig.js without extra work.

* I've added a link to the IS for a presentation I give on Design Systems in Drupal, which shows that Atomic Design and Drupal site building are already reasonably well-aligned, if you do your site building right (Content Types, View Modes, Views, Panels). Having the render system use the same mental model, as Daniel notes, can only be a good thing.

* Big +1 with Fabian that we really really ought to be using classes here. Note that does not mean that themers building new components need to define a PHP class; I suspect we can do that with a compile step or similar (since Twig already has a compile step). (Although maybe they should have the ability to do so in very edge cases, and that replaces the preprocess function?) In a sense, what we're talking about here is ViewModel objects. That is, strictly defined typed objects that represent a value object for the display layer, as opposed to the storage layer. Drupal currently doesn't really have that distinction and passes entities straight forward to the display layer, which is not cutting it. If we're going to formalize the concept of "component", we should double down on that formalization and define them as typed classes. That not only makes debugging much easier, it gives developers more flexibility to leverage the API directly if they need to. We already have the beginning of support for "renderable objects" in the render API; let's get the rest of the way down that path.

Go's templating system uses that same model. Go has a native template system that is very Twig-esque, but you start by passing it a single struct. That struct can have sub-values in it, but technically what you're rendering with the template is that struct specifically. That struct is a ViewModel, in essence. Sometimes it may be 99% the same as a data object, sometimes not, but it's conceptually distinct. That's a good thing. That specificity also gives us more information to push forward to Javascript, again making it easier for a component to work client side or server side with minimal to no alteration. We want that.

Another benefit of using classes is that objects are, frequently, cheaper than arrays in PHP. Pre-defined object properties take less memory than array keys do, and because objects pass by handle we don't need to pass big by-ref structures around, which is actually quite inefficient. There's a lot of places we could get efficiencies here, especially with newer PHP versions.

* Because cosmicdreams isn't here yet, I'll say it: Web Components. :-) Essentially we're defining a definition mechanism for a web component package, which we just happen to pre-render ourselves (client or server side). That makes leaving it as a Web Component later once that suite of specs is sufficiently reliable much easier. In fact, I would argue we should develop this stand-alone library specifically with Web Components in mind as the primary rendering environment, and "render up into full page on the server" as the alternate processing tool. That will put is in a very good place to go gang-busters on Web Components once the market is ready for people to go gang-busters on Web Components. (And if it's not, we've still got a good, separated conceptual model that we've built from that gives us flexibility.)

Thank you Wim for tying all these threads together!

Fabianx’s picture

Excellent comment by #20: I fully agree.

Can you elaborate a little more how templating works in Go?

I also think if every first implementation has a JS counterpart (with our new JS testing not impossible), then we are much more easily fit for the future, too.

Crell’s picture

Go's standard library includes a Twig-esque (although not quite the same as Twig) template language. It's official docs are here:

https://golang.org/pkg/text/template/

And the version for use with HTML, that does smart escaping, is here:

https://golang.org/pkg/html/template/

A somewhat more approachable introduction is here:

https://gohugo.io/templates/go-templates/

(Hugo is a Go static site generator, akin to Sculpin or Jekyll. I've not used it; their docs just came up first in a google search.)

To be clear, I'm not suggesting that we should model our system on Go necessarily. There's probably stuff to learn from it, just like any other system, but the main point is simply the idea of every template having a struct (value object) that it binds to; that struct is explicitly defined, and thus you get all of your explicit value type safety. It's another data point and prior art in favor of explicit classed objects (ViewModels) for each component, which can be passed to the template. (That way, themers can override a template but still know exactly the stuff they'll get from the object, PHP devs have a self-documenting data structure rather than an undocumentable anonymous array, etc.)

Fabianx’s picture

#22: The most interesting point is however, how do you drill down into the struct (value object).

e.g. How do you wrap one component e.g. an image, within another component.

If that is again a value object within a simple list (collection), then we should be good.

But that is something that I would like to explore first some more. Can you elaborate how go deals with structs within structs and lists?

aleksip’s picture

- A stand-alone implementation independent of Drupal, developed with Web Components in mind. Sounds really good to me! Also sounds like a project independent of Drupal, in a true 'off the island' way. A project that should be brought to the attention of the larger PHP community from the beginning?

- I'm still wondering about the fate of the current Twig extensions. I can't see how they can be used when we get to the 'everything is a decoupled component' stage?

legolasbo’s picture

FileSize
74.73 KB

I've been thinking about ways to improve the rendering system for quite some time these past few month, but I've never had time to actually write my thought down. Tonight I did manage to find some time to draw some UML in order to possibly answer #23. The UML diagram is still missing a lot of details, but I think it should be sufficient to communicate the basic idea.

I propose we create a primary interface (Component) from which all components derive. Besides that we should introduce another interface (ComponentCollection) from which fuzzy composed components such as tables derive. As shown in the UML diagram the component interface has two methods, render() and getProperties(). The render method does just that, render the component. getProperties() however returns an array of all relevant properties for the given component. for example, when called on a LabeledContainer it would return ['label', 'components'].

Drilling down and filtereing in the theme layer.

Assuming render() uses twig to render output a themer could do something like {{ SomeComponentName|properties }} to get a list of all available properties available to drill into. Assuming that SomeComponentName contains a property which is in fact another component, (s)he can then use {{ SomeComponentName.SomeSubComponent }} to just render the subcomponent or {{ SomeComponentName.SomeSubComponent|properties }} to discover the properties of the subcomponent.
We could do even more cool stuff like {{ SomeCompositedComponent|only_type('SomeComponentType') }} or {{ SomeCompositedComponent|without_type('SomeComponentType') }}

These filters will be trivial to implement because all components are based on the same interface.

Decoupling

Note how this diagram does not show any implementation details about Drupal, Twig or any other way of rendering the components. This could be the base of a totally platform agnostic package. Completely independent from Drupal or rendering frameworks. Especially if we also introduce a Renderer interface and require an implementation of that interface during the construction of a component. By doing so, we decouple what gets rendered, from how it gets rendered. That way we can create a Twig renderer, but also a JSON renderer, or any renderer for any output we want to generate. These renderers can even be implemented in completely separate packages.

Disclaimer ;)

Once again. This is a really incomplete representation of the way I've been thinking about implementing something like this. But I am curious to find out how you think about the general concept.

edit:
The UML has certain parts greyed out. that is done on purpose to emphasise the abstract classes and interfaces but still be able to demonstrate some concrete implementations.

moshe weitzman’s picture

This plan and PoC looks solid to me as well.

  1. I agree with others who highlight the relevance of Atomic Design. The pattern library and style guide images illustrate the problem of treating all components equally. No designer cares to see [Container] in their style guide. I look forward to tagging/grouping for components so we can hide things when needed.
  2. My vision for the style guide is that it is a learning and checklist tool for a designer. There the designer learns how to style each component and sees his progress along the way.
  3. Preprocess functions were a major innovation in in 2007, and really helped module developers and themers override behavior and style their sites. AFAIK, they are still in heavy use today. It would certainly help the "JS everywhere" case to get rid of them in favor of logic in templates. But for the average Drupal user, this could be a major regression. Needs experimentation. For background, see #137236-1: theme preprocess documentation does not exist
catch’s picture

Preprocess functions were a major innovation in in 2007, and really helped module developers and themers override behavior and style their sites

2007 was before we had view modes, display entities, formatters, hook_entity_view/field_attach_view_alter(), all page elements as blocks, and many other things which are both more powerful than preprocess, and sometimes undermined by it.

There are places where we've failed to apply those patterns yet, such as 'submitted by' and title display on nodes, we're going to need to replace those things properly when moving to components - so that we don't end up with a node display component that's only different to other content entities by one or two template variables. But yes generally I agree that there's a lot of work to be done to actually remove usage of preprocess.

template_preprocess_node() and template_preprocess_comment() still the best/worst examples:

https://api.drupal.org/api/drupal/core%21modules%21node%21node.module/fu...
https://api.drupal.org/api/drupal/core%21modules%21comment%21comment.mod...

Fabianx’s picture

One thing that should be deprecated ASAP is theme_wrappers as you can always use an upper level instead and its super confusing to have #theme and #theme_wrappers at the same level with the theme wrapper having not available most variables needed for context.

catch’s picture

JohnAlbin’s picture

Thanks for creating this issue, Wim! I would have been posted sooner, but I got sick with flu the same day you posted. :-p And I still haven't caught up with my work enough to read this whole thread. I will have to come back to it during Drupalcon.

In the meantime, here's my current thinking on adding components to Drupal:

This is too big a task to try to plan out and tackle in core. This part of the theme system touches too many parts of core, every single module, etc. The last time we tried to simplify the theme system without doing some contrib-side experimentation and themer by-in, we ended up with the Render API. I'd like to have some intense experimentation to get concepts in front of real-world front-end devs.

For the past 3+ years, I've been experimenting with different techniques for making components of HTML/CSS/JS and how to integrate them with Drupal.

While working with Drupal 7, it became clear that the "hard part" was going be figuring out:

  • how to route the variables used by a specific theme hook into a specific component.
  • And given a module's default choice of a component, how does a theme change that decision and route those variables into a different component?

Now that I've started studied Twig's documentation and have started porting Zen to Drupal 8, I've discovered that I can use Twig to do the routing of variables.

I currently have a components library built in Twig that includes no Drupal-specific extensions. (Side note, because there are no Drupal-specific extensions or variables, I'm able to build a style guide from these components by pushing the *.twig files through Twig.js in a Node.js build system.)

Zen uses Drupal's *.html.twig files solely as a means of getting at a theme hook's variables.

For example, classy's html.html.twig file has a skip-link styling on an <a> element. Zen has a skip-link component, so its html.html.twig file uses {% include "@STARTERKIT/navigation/skip-link/skip-link.twig" with {modifier_class: 'visually-hidden visually-hidden--focusable'} %}; modifier_class is a Twig variable that I defined in all my components to hold component variant classes.

Another example: the comment.html.twig template has a {{ new_indicator_timestamp }} variable. I am NEVER going to put that variable in a component because it will tie the data variable name to the HTML display and make it impossible to re-use with other data that does not have that variable. Instead I will be able to create Twig blocks in my component library Twig files and then do this in Zen's comment.html.twig:

{% embed "@STARTERKIT/component/box/box.twig" }
{% block header %}
  <mark class="hidden" data-comment-timestamp="{{ new_indicator_timestamp }}"></mark>
{% endblock %}
{% endembed %}

No yucky Drupal variables in my component library. Just perfectly re-usable Twig templates. Actually, I may not have a <mark> in my comment.tpl.php, I could replace that line with something like: {% include "mark.twig" with { attributes: ' data-comment-timestamp="{{ new_indicator_timestamp }}"' } %}.

In Zen, all of the *.html.twig files reside in /templates and Zen's component library (with CSS, JS and *.twig files) resides in /components.

The more I work on this, the more natural this feels for a front-end developer. While the "two separate groups of Twig files" concept feels weird and some part of me thinks this is "needless duplication" or like I'm "mis-using the Twig system", I still really like working this way. Drupal 8 front-end devs already know Twig and they don't have to write any PHP to use this dual-twig method.

Anyway, that's my input for now. I look forward to some intense discussions at Drupalcon!

effulgentsia’s picture

I currently have a components library built in Twig that includes no Drupal-specific extensions... In Zen, all of the *.html.twig files reside in /templates and Zen's component library (with CSS, JS and *.twig files) resides in /components.

Yay! Huge +1 to this! I think this is a very important distinction to make. That HEAD's current theme hooks / templates actually consist of 2 very different things: things that can be turned into components (e.g., image.html.twig) and things that don't make sense as components because their variables are coupled to Drupal data (e.g., node.html.twig). Having different terms for these different things (e.g., components and templates) is a helpful starting point.

While the "two separate groups of Twig files" concept feels weird and some part of me thinks this is "needless duplication" or like I'm "mis-using the Twig system", I still really like working this way.

Keeping non-component templates in Twig for now seems a good starting point. Ideally, however, we'd get to a point where these templates would contain 0 markup, and solely route to components for all of their markup. This would mean that a theme could control every single piece of markup solely through overriding components, and not be required to override any non-component templates. This is huge, because most contrib modules currently define their own theme hooks for the presentation of their own module-specific data concepts, which results in it being impossible for generic themes to provide theme-specific implementations of every contrib module's templates. But, if in a componentized architecture, the contrib module templates are just routing layers to core-defined components, then a generic theme could end up controlling all markup, even as various modules get installed on a site, and this was a key piece of Jacine's original vision for componentizing (see the "Your Output is not Special" section). Once we achieve such a separation, it might become natural to ask whether Twig is the best language for expressing markup-free variable routing to components, or whether something purely declarative, like YAML, would be a better fit. Hard to answer that now though, since we don't know what logic will be needed in these templates once they're all properly separated.

The Twig template (*.html.twig) performs all the necessary processing of the variables received, this ensures we don't depend on preprocess functions. This removes the need for front-end developers to dive into PHP.

For components (as opposed to the non-component templates), I really like this, because getting this code out of PHP allows for client-side re-rendering via Twig.js! One thing to consider though is that each module can also implement preprocessing. But I think we can come up with a pattern by which modules could still do this via Twig files, and then we have some mechanism to aggregate all of these Twig files (that preprocess variables for a given Twig component) into the compiled Twig component.

But processing a data source into data that should show up in that template? I don't think that sort of business logic belongs in a template.

I agree, but I think this concern applies solely to the non-component templates, since those are the only ones that deal with Drupal-bound variables. And unfortunately I don't currently see a way out of keeping this code in PHP, but perhaps it makes sense to move it from a hook_preprocess() to a hook_view_model_alter(). This unfortunately presents a barrier to client-side re-rendering of such a template, so it would be nice to reduce such alter implementations to their bare minimums, and do as much as is sensible in Twig-implemented preprocessing instead.

effulgentsia’s picture

Once we achieve such a separation, it might become natural to ask whether Twig is the best language for expressing markup-free variable routing to components, or whether something purely declarative, like YAML, would be a better fit.

I also want to tie this in with #8. Perhaps these variable routers could end up being Layout/Panels configurations. But that's pure speculation at this point: seems to me like there's a bunch of steps that need to happen here before broaching that, though experimenting with such speculation could yield some great insights.

catch’s picture

and things that don't make sense as components because their variables are coupled to Drupal data (e.g., node.html.twig).

The node-specific information in node.html.twig is removable though. The reason it's still there is because we haven't completely ported node rendering from Drupal 4-6 hard-coded variables to display modes etc., but if we finish that then it'll be a three-line template or so.

cosmicdreams’s picture

Thanks @crell for the shoutout in #20.

Yes, I've been listening and reading these comments and am growing more and more excited about the prospect that this mechanism could mean great things for Web Components. To understand why, first a definition.

Web components are exactly the kind of components we're talking about here, just (eventually) supported natively by the browser. Specifically, "Web Components" as defined by http://webcomponents.org/ are a set of standards that define the browsers ability to create, register, and package templates and an API for interacting with them. It seems pretty clear to me, given the momentum of this effort that one day the browser could be our themeing layer. If that sounds fantastical it's because it is.

If that future was imminent, this effort to component-ize Drupal would be vital. We would need to be able to define the structure, appearance, behaviors of components. We would need to build a set of components that have well define responsibilities. We would need to understand how to use these components together so that we can build more complex components / features / pages / etc.

My current thinking about Web Component's interaction with a system like a CMS is that they benefit from two different kinds of data deliveries.

1. Initial data that establishes state / content
2. Responses to interactive behaviors.

But the main thing I want to say is that I'm excited that we're talking about this effort.

mdrummond’s picture

One of the things I've been doing the last few weeks is doing some refresher looks at Ember, Angular and React. React in particular exemplifies a component-based model, but it seems like there's a general movement away from MVC and towards components.

With React, a component can take in props (static data) or state (data that might change). I think that's exactly the sort of thing you're talking about, cosmicdreams.

One of the things I'm having a hard time getting my head around is how to keep a next-gen theming system DRY.

In particular, if we want to have more robust JS interactions in core, there are a lot of potential areas of duplication.

Let's just say for the sake of argument we're able to make use of Twig.js to reuse templates on the server-side as well as the client-side. That would be one element of keeping things DRY, but that alone is not all that we need.

Somehow we're sending data into component templates. In our current render system that involves setting up render arrays and preprocess hooks to allow for alterations to the data that ultimately gets sent into templates. Even if we come up with a new PHP-based component render system that we open source and make available to other PHP projects (as well as ourselves), how do we duplicate that logic on the client side?

The render system is in the business of taking structured Drupal data and turning it into markup. There's a lot of logic that goes on to make that happen.

In theory you could put a whole ton of logic into a template, and depend on that for transforming raw Drupal data whether it comes from a REST endpoint or internally through Drupal. I'm not sure I'm keen on that: putting some display logic in templates to make markup decisions seems fine by me. But data transformation logic? No, not a fan of that in a template.

So if you don't want that, and you want to be able to share templates between the server side and client side, so that you can do good things like virtual DOM diffing for quick page updates, that is almost assuredly going to mean duplication of logic in both PHP and JS.

This is a little bit off topic for this thread, but it's just something I've been mulling over, and it seems somewhat connected. I don't really have good answers on how to solve this right now.

Wim Leers’s picture

Thanks for all the insightful comments! There’s lots of great ideas, thinking, critiques and so on in there. I deliberately avoided commenting here for weeks (although I read every single comment shortly after posting), to avoid steering the discussions in a certain direction.

I’m happy to see that everybody is at least “+1 for concept”. That makes this issue/patch a great starting point :)

To make sense of the ~9000 words, ~50000 characters or ~45 minutes of reading that you all have posted since #2, I:

  1. replied to all points of every single comment in a short manner
  2. recorded all the topic areas that have been brought up as “conclusions” (I can’t find a better word)
  3. whenever topic areas were brought up more than once, I counted their votes
  4. you’ll see that “Conclusions” section at the bottom of each per-comment reply, and you’ll see it grow towards the end :)

If you want to, you can ignore all that and just look at the 24 conclusions:

  1. Must-have: component definition [4]
  2. Must-have: component tree traversal algorithm [4]
  3. Must-have: BC [4]
  4. Must-have: strategy for form elements [2]
  5. Nice-to-have: UI to map data to component: DS, Panels, entity displays… [5]
  6. Must-have: avoid coupling by putting it in \Drupal\Component [5]
  7. Must-have: sane inheritance/extending/overriding of components [4]
  8. Must-have: remove preprocess [4]
  9. Must-have: no component-per-UI [2]
  10. Must-have: current component-per-UI instances (using #theme or #type) continue to work in D8, but cannot be made to use components, this is what we need the D9 break for
  11. Must-have: tackle complex cases first, for example table — this will inform/guide the architecture
  12. Nice-to-have: auto-generate CKEditor Widgets
  13. Won’t-have: use TypedData for types of inputs
  14. Must-have: infrastructure to support functionality like contextual links/quick edit/…
  15. Must-have: strategy for Twig functions, and particularly Attribute [2]
  16. Must-have: atomic design categorization [2]
  17. Must-have: support for both site builder & themers.
  18. Won’t-have: drillability [5]
  19. Must-have: modules and themes can define components.
  20. Must-have: well-defined mechanism for generating input tree, necessary for client-side re-rendering. [4]
  21. Nice-to-have: any object implementing RenderableInterface returns a component
  22. Nice-to-have: style guide with checklist as progress indicator for themed
  23. Must-have: develop through experimentation, expect dozens of iterations, definition of done is when it is proven to work for all of core, require validation by real-world front-end developers
  24. Must-have: strategy for static vs dynamic data, and whether it matters

Top priorities to discuss, based on both number of times surfaced/contentiousness and how much it blocks progress/impacts direction:

  • 18: Drillability: desired or not?
  • 8: Preprocess removal: everybody wants it, but do we really understand all consequences?
  • Related: #theme_wrappers.
  • 3: Backwards compatibility: does this mean the best we can do is to still have “render arrays” (but no more doom), or can/should we do that different?
  • 20: mechanism for generating input tree

Let's discuss at least those top priorities at DrupalCon New Orleans, hopefully more, but especially those.


Comment #6 (@FabianX):

  • Objects are a problem too: they can encapsulate logic that we cannot automatically map to JS. So they interfere with the goal of being able to re-render on the client side.

    SO: What is acceptable is a generic object/class, what is not is a component-specific one.

  • Traversing the component tree is indeed not yet well-defined. Because a component is not yet well defined. Added to conclusions.
  • BC: very very strongly disagree. “taking defined properties and ignoring others” is part of the problem we’re solving: the lack of strictness, and the resulting confusion/ambiguity.

Conclusions:

  1. Must-have: component definition
  2. Must-have: component tree traversal algorithm
  3. Must-have: BC

Comment #7 (@dawehner):

  • Form elements: I think they should become components, perhaps in a slightly different form (no pun intended). Symfony has a similar-yet-different concept to our form elements, for example. Definitely TBD. Added to conclusions.
  • How components receive their data: when I worked on this, I definitely did not think about/have the ambition to solve that problem too. I think having strictly defined inputs already helps us get much further to being able to realize that. (I especially fear there will be so many possible combinations that accept a particular type of data, which means the user will be presented with far too many options. Furthermore, letting users define a tree of components seems fraught with UX problems.) Added to conclusions.
  • Reuse: already demonstrated in patch: see responsive-image.html.twig.
  • Formatters + entity display configuration: Maybe! See above concerns.
  • Panel: Same as above.

I really like your thinking around formatters + entity display + panels. But I can’t yet see a sane path to get there. Hopefully in some time, we will :)

Conclusions:

  1. Must-have: strategy for form elements
  2. Nice-to-have: UI to map data to component
  3. Must-have: avoid coupling by putting it in \Drupal\Component

Comment #8 (@stevector):

  • Importance of compatibility layer: couldn’t agree more! Recorded vote.
  • Components as layout plugins? I re-read your blog post to ensure I fully grok your intent. Great post :) I remember it from Marc’s post, but it’s better to read the source: https://www.palantir.net/blog/explaining-panels-why-i-use-panels

    dawehner already brought Panels up before you, and like I told him: I did not think about this.

    I think postponing layout plugins (#2296423: Implement layout plugin type in core) is nice in principle, but rather pointless in practice: it will take a year at best for this to become fully usable. We shouldn’t put everything on hold for this: Display Suite/Panels/Layout plugins as they are currently designed still are a very valuable tool. Recorded as vote for 5.

  • RE: “templates all the way down?” Yes, I really think we need it to be components/templates all the way down. For a first example, see the Responsive Image component, which also uses that Twig include syntax :)
  • RE: Twig namespaces for overriding component implementation: yes, exactly, this is how I envision themes will extend/override components. This is why I mentioned #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance as a blocker in #2. Added to conclusions.

Conclusions:

  1. Must-have: BC [1+1]
  2. Nice-to-have: UI to map data to component: DS, Panels, entity displays… [1+1]
  3. Must-have: sane inheritance/extending/overriding of components

Comment #9 (@catch):

  • RE: removing preprocess (#2060783: Remove the preprocess layer.) and how this solves BC: indeed! Added to conclusions. Recorded as vote for 3.
  • RE: no guarantees on what a render array contains (“old stuff” or just components) helping BC: indeed! Recorded as vote for 3.
  • RE: things already mentioned: recorded votes for 2, 4.
  • RE: many-to-one conversions via new #types: I wasn’t involved in this effort, but if I were, I think I’d have hoped for the same :) Totally agreed that we shouldn’t end up with component-per-UI — that’d largely defeat the purpose. Added to conclusions. I’ve seen there is a good suggestion for this by one of the commenters further down :)
  • RE: how the BC layer for #theme can work if there’s no 1:1 relation to a component: it can’t work. This is what we need to break BC for. Those cases need to be rewritten to use this theme component library in D9. We keep BC by letting the current code work, not by automatically converting it to components. Added to conclusions.

Conclusions:

  1. Must-have: component tree traversal algorithm [1+1]
  2. Must-have: BC [2+1]
  3. Must-have: strategy for form elements [1+1]
  4. Must-have: remove preprocess
  5. Must-have: no component-per-UI
  6. Must-have: current component-per-UI instances (using #theme or #type) continue to work in D8, but cannot be made to use components, this is what we need the D9 break for
  7. Must-have: tackle complex cases first, for example table — this will inform/guide the architecture

Comment #10 (@catch):

  • RE: tackle tables sooner rather than later: +1! Added to conclusions.

Conclusions:

  1. Must-have: tackle complex cases first, for example table — this will inform/guide the architecture

Comment #11 (@larowlan):

  • RE: auto-generate CKEditor Widgets: wow, that’s super interesting! I did not at all think about that! I have my doubts about that though, because I know that e.g. the image2 CKEditor Widget (which Drupal 8 extends) is extremely complex, because the <img> tag can be both inline and block… and because the accompanying JS code must include logic matching the exact HTML. Nevertheless, added to conclusions.

Conclusions:

  1. Nice-to-have: auto-generate CKEditor Widgets

Comment #12 (@jhodgdon):


Comment #13 (@davidhernandez):

  • RE: concern about removing preprocess entirely. I hear you. It may surprise you, but I agree. But for a different reason than you think probably: some of the logic in preprocess functions simply doesn’t belong there; some of that logic should’ve occurred before we talk to the theme system/templates/components. Because components should only have logic to deal with the simple, primitive data they receive, not with complex Drupal objects. Recording as vote for 8.
  • RE: property to specify the provider of the component. Recording as vote for 7.
  • RE: versioning: I’m not sure that’d help at all, unless we kept all old versions of components around. And in that case, bug fixes to a component that a theme is extending will not be picked up… I think this is a problem that is not unique to the component system, I don’t think this needs to be part of the scope.

Conclusions:

  1. Must-have: sane inheritance/extending/overriding of components [1+1]
  2. Must-have: remove preprocess [1+1]

Comment #14 (@fago):

  • RE: develop decoupled: +1, recorded as vote for 6.
  • RE: component tree specified, recorded as vote for 2.
  • RE: mapping to TypedData: wouldn’t that violate the first thing you said? TypedData has the concepts of computed properties, settings, and more. How will we make that work in JS?

    Don’t get me wrong, I see the attractive/interesting parts of it, but I think there’s also a pretty big risk in coupling it to TypedData.

    On further thought, there’s an even bigger problem: TypedData does not require primitives, but in fact allows for arbitrarily complex nesting. See e.g. \Drupal\Core\Entity\Plugin\DataType\EntityAdapter. For now, added to conclusions as won’t-have.

Conclusions:

  1. Must-have: component tree traversal algorithm [2+1]
  2. Must-have: avoid coupling by putting it in \Drupal\Component [1+1]
  3. Won’t-have: use TypedData for types of inputs

Comment #15 (@fago):

  • RE: Extra layers of Drupalizm like entities, views, block, rules: recorded as vote for 5.
  • RE: contextual links as higher-order components? Added to conclusion.
  • RE: “Problem of mapping data and passing data through is exactly the same problem as solved by Rules with contextual plugins and nested calls” → Let’s discuss this in IRC or IRL, because I don’t understand why this is the case.

Conclusions:

  1. Nice-to-have: UI to map data to component: DS, Panels, entity displays… [2+1]
  2. Must-have: infrastructure to support functionality like contextual links/quick edit/…

Comment #16 (@aleksip):

  • RE: Twig functions like Url and Drupalisms in Twig like Attribute: the number of Twig functions is extremely limited. We’d require those to be implemented both in PHP and JS. Any module providing a Twig extension that adds more Twig functions would have to also provide a JS implementation. Regarding Attribute: no good answer to that for now. Added to conclusions.

Conclusions:

  1. Must-have: strategy for Twig functions, and particularly Attribute

Comment #18 (@mdrummond):

  • RE: MidCamp discussion: yes, that’s one of my references :)
  • RE: Atomic Design: +1 for the principle. I think it may be worthwhile to categorize each component as one of those. Whether we’ll have what Atomic Design calls “templates” and “pages” will remain to be seen though, since those are more fitting for non-interactively-built sites, i.e. some systems other that are not Drupal. Added to conclusions.
  • RE: concrete examples and suggestions: I think we’ll better be able to assess those things while we convert more things to components. I don’t think it makes sense to speculate in big ways right now.
  • RE: personas: +100 for “Assembled Web” (aka site builder) and front-end developer (aka themed). Added to conclusions.
  • RE: drillability: I’m not convinced that it is necessary, or even desirable. Drillability implies two things: 1) templates modified for the specific names of children in a particular given data structure, 2) complex data structures. Both of those are highly undesirable. I think part of the point here is that each template only receives simple data structures (i.e. respecting the limitations I proposed in the issue summary), because: A) otherwise too complex/frustrating themer experience, B) otherwise no ability to provide strict validation, and hence good error messages, C) impossible to re-render on client side.

    Added to conclusion.

  • RE: data component + ordering in UI, as well as layout: recording that as another vote for 5.
  • RE: definition of component should include required context: “ essentially the data source and any data relationships between multiple data sources” and “[…] it needs to get that data from somewhere.” I find these statements confusing, vague and contradicting to what you said earlier. My proposal is explicitly stating that each component defines its inputs, the inputs must be primitives/lists of primitives/other components, and that is it. We have a component tree and a matching variable/input tree. So, a component does not “need to get its data from somewhere”, it’s already getting it from that input tree. We don’t want components to have data-retrieval-logic: then we make them too complex again, much like today’s system.

    I suspect you meant something different though, so let’s discuss that in IRC/IRL.

    Perhaps you meant the code that is responsible for generating the component+input trees? Because that totally is something that is very important: without that being well-defined, it will be impossible for JS to generate or retrieve an updated input tree, which is necessary for client-side rerendering. Added to conclusion.

  • RE: modules and themes can define components: yes, and this is already the case.
  • RE: themes can override/extend markup/css/js in module-defined components: yes, and I’d say base theme’s components can be as well. Recorded as a vote for 7.

Conclusions:

  1. Nice-to-have: UI to map data to component: DS, Panels, entity displays… [3+1]
  2. Must-have: sane inheritance/extending/overriding of components [2+1]
  3. Must-have: atomic design categorization
  4. Must-have: support for both site builder & themers.
  5. Won’t-have: drillability
  6. Must-have: modules and themes can define components.
  7. Must-have: well-defined mechanism for generating input tree, necessary for client-side re-rendering.

Comment #20 (@Crell):

  • RE: stand-alone: +1, recorded as vote for 6.
  • RE: using classes: not necessarily, see my answer to @Fabianx.
  • RE: those classes should be auto-generated, compiled, like Twig: sure, that would be fine. But what would we gain?
  • RE: those classes are ViewModel objects with strictly defined properties: yep, that is completely in line with what I already did. Note that D8 must work on PHP 5.6, so we can’t typehint to string/int/…, so that limits the potential usefulness: we’d have to check those explicitly anyway. The current patch doesn’t use classes yet, but it could, and it’s equally strict already. However, I’m still not sure what we would gain. We’d have to generate both PHP code and JS code (we’d want 1:1 PHP and JS ViewModel objects), even though in both cases we could use a simple key-value data structure. However, I totally see the value for PHP developers/auto-complete wise. Perhaps it makes more sense for that to be a translation layer, to aid PHP developers, but it’d not be a strict/required part of the “theme component library” PHP component. This needs further discussion. I’ll record this as a vote for 1.
  • RE: “Drupal currently doesn’t really have that distinction and passes entities straight forward to the display layer” → this is exactly part of the problem, and is why I’m insisting on only allowing primitives/lists of primitives/other components as inputs. I’ll record this as a vote for both 18 and 20.
  • RE: “We already have the beginning of support for “renderable objects” in the render API; let’s get the rest of the way down that path.” I’m not sure about this yet, because what if two sites want to use different components to represent/render the same object? Added to conclusion.
  • RE: objects cheaper than arrays: this is a premature micro-optimization. Furthermore, I’ll call out that Drupal 8 spends a significant amount of time in the autoloader to load all those classes, so realistically speaking this may even cost us more, net. This is an implementation detail for which it is too early.
  • RE: web components: of course, but this too is still too early: we don’t even know what exactly they will look like. Of course I agree there’s strong overlap/similarities.

Conclusions:

  1. Must-have: component definition [1+1]
  2. Must-have: avoid coupling by putting it in \Drupal\Component [2+1]
  3. Won’t-have: drillability [1+1]
  4. Must-have: well-defined mechanism for generating input tree, necessary for client-side re-rendering. [1+1]
  5. Nice-to-have: any object implementing RenderableInterface returns a component

Comment #23 (@Fabianx):

  • RE: drillability: see my answer to @mdrummond in #18. Recorded as vote for 18.

  • Won’t-have: drillability [2+1]


Comment #24 (@aleksip):

  • RE: stand-alone: recorded as vote for 6.
  • RE: Twig extensions: see my earlier answer to you.

Conclusions:

  1. Must-have: avoid coupling by putting it in \Drupal\Component [3+1]

Comment #25 (@legolasbo):

  • RE: UML/PHP class architecture proposal: I think that’s simply prematurely detailed, we will have to work towards something like that, but only through doing the work of converting complex cases will we know what the full set of requirements is, which I suspect this architecture won’t be able to fulfill.

    However, thanks a lot for doing that — it’s great to be able to reference an already-made diagram for what is a possible implementation!

  • RE: drilling: see my answer to @mdrummond in #18. Recorded as vote for 18.
  • RE: JSON and HTML renderers: well, as long as we deal with primitives/lists of primitives/other components, you could indeed easily render it into JSON or XML or whatever. But… generally speaking that will be rather pointless, because theme components are about displaying information in a coherent, patterned way to humans. Yet JSON, XML and others are about exchanging data between machines. In some cases, there would be some value: you could omit certain fields, rename properties, post-process property values, et cetera. But that’s it. This would IMO quickly break down because you’d end up passing entire, complex (Drupal or not) data objects to components, which kinda defeats the purpose of components. Again, see my thoughts on drillability.

Conclusions:

  1. Won’t-have: drillability [3+1]

Comment #26 (@moshe weitzman):

  • RE: atomic design: recorded as vote for 16.
  • RE: style guide as learning and checklist tool: oh, wow, that makes so much sense! I think this is quite exciting. I don’t know why I didn’t think of that! Added to conclusions.

    RE: totally agreed we need to deeply, 110% understand the consequences of removing preprocess. Indeed needs experimentation. Recorded as vote for 8.

Conclusions:

  1. Must-have: remove preprocess [2+1]
  2. Must-have: atomic design categorization [1+1]
  3. Nice-to-have: style guide with checklist as progress indicator for themed

Comment #28 (@Fabianx):

  • RE: I agree that #theme_wrappers are extremely confusing. In a component world, they wouldn’t make sense anymore.

Comment #30 (@JohnAlbin):

  • RE: flu: given the awesomeness of what you wrote next, I think it’d be an interesting although cruel experiment to give you the flu more often :P
  • RE: experimentation: +1000000. I’m sorry if that was not clear: I do not at all intend to design something top-down, without actually applying it to all the real-world problems that core offers. This will be a big undertaking, but IMO this will only be done once we have a component-based alternative for every single #theme and #type in core. And preferably the trickiest contrib modules too. But I think D8 core is broad/diverse/complex enough for it to be representative “of the world”. I’d love counter-examples though. Added to conclusions.
  • RE: get it in front of real-world front-end devs: that aspect is more difficult. How many is enough? Must they be from outside Drupal? I agree with the principal though. Merged this with your previous point’s conclusion addition. Ideally we’d have a concrete set of people giving their sign-off. Can you think of a list of people?
  • RE: “components library built in Twig that includes no Drupal-specific extensions” + skip-link example + comment example + “the more natural this feels for a front-end developer”: this sounds so very awesome! We should validate it in a deeper discussion, but I’m very excited by A) the approach, B) the fact that you have it working already.

    This helps with:

  • the definition of what a component is (1)
  • avoiding coupling (6)
  • showing the importance of extending/overriding components (7)
  • to confirm we don’t need to have a component-per-UI and provides a neat solution, although it currently still kind of is a component-per-UI, since you still have component.html.twig and map that to other components, but you made it far more acceptable — especially because comment.html.twig and friends live in a separate directory: /templates — one that even helps with BC! (9)
  • shows one possible strategy for Twig functions (15)

Conclusions:

  1. Must-have: component definition [2+1]
  2. Must-have: avoid coupling by putting it in \Drupal\Component [4+1]
  3. Must-have: sane inheritance/extending/overriding of components [3+1]
  4. Must-have: no component-per-UI [1+1]
  5. Must-have: strategy for Twig functions, and particularly Attribute [1+1]
  6. Must-have: develop through experimentation, expect dozens of iterations, definition of done is when it is proven to work for all of core, require validation by real-world front-end developers

Comment #31 (@effulgentsia):

  • RE: helpful starting point: +1, recorded as vote for 3.
  • RE: “ideally these non-component templates would contain 0 markup and solely route to components for all their markup” → +1, recorded as vote for 1, 2 and 9.
  • RE: “But, if in a componentized architecture, the contrib module templates are just routing layers to core-defined components, then a generic theme could end up controlling all markup, even as various modules get installed on a site and this was a key piece of Jacine’s original vision for componentizing” → such a great insight! I totally missed that!
  • RE: “And unfortunately I don’t currently see a way out of keeping this code in PHP”: indeed this is perhaps even the hardest problem of all. Recorded as vote for 8 and 20.

32 (@effulgentsia):

RE: “Perhaps these variable routers could end up being Layout/Panels configurations.” → another interesting insight! Panels & friends would generate such templates then. Recorded as vote for 5.

Conclusions:

  1. Must-have: component definition [3+1]
  2. Must-have: component tree traversal algorithm [3+1]
  3. Must-have: BC [3+1]
  4. Nice-to-have: UI to map data to component: DS, Panels, entity displays… [4+1]
  5. Must-have: remove preprocess [3+1]
  6. Must-have: no component-per-UI [2+1]
  7. Must-have: well-defined mechanism for generating input tree, necessary for client-side re-rendering. [2+1]

Comment #34 (@cosmicdreams)

see my response to Crell regarding web components. It’s too early to talk about concrete things, but it’s definitely worth keeping in our heads. And I’m sure you’ll help with that :)


Comment #35 (@mdrummond):

  • RE: props (static data) vs state (dynamic data): yeah, we don’t have this distinction at all. I’m not 100% certain yet we need it, but I could definitely see how it may be necessary for client-side re-rendering. Although I don’t yet see how that could work in a way that makes sense for both JS and PHP. Needs investigation. Added to conclusion.
  • RE: “Somehow we’re sending data into component templates.” + “Even if we come up with a new PHP-based component render system […] how do we duplicate that logic on the client side?“ → exactly, and that’s what 19 is all about. Recording this as another vote for 19.
  • RE: “In theory you could put a whole ton of logic into a template, and depend on that for transforming raw Drupal data “ → nope, this is a no-go, this is why I want the only acceptable inputs to be primitives/lists of primitives/other components. This is why I think drillability itself is a problem. Recording as votes for 18 and 20.

Conclusions:

  1. Won’t-have: drillability [4+1]
  2. Must-have: well-defined mechanism for generating input tree, necessary for client-side re-rendering. [3+1]
  3. Must-have: strategy for static vs dynamic data, and whether it matters
tkoleary’s picture

What Wim said

polynya’s picture

Wow, thanks for the great summary Wim. Can I add some +1s, especially for 1: component definition and 19: modules and themes can define components.

As an extension to 19, a module should be able to define the path to a folder containing components. The project I'm part of is building web pages from components defined in an external pattern library. We don't want to alter the Drupal repo just to update a component or to add a new one. As part of this, I built UI Components based on Wim's patch. We'll probably be using this until the features are in core.

I'm really disappointed I won't be at DrupalCon next week, but I hope to see you all in Dublin!

kalpaitch’s picture

I am working with Polynya above. In our particular use a component could define a twig template, css styles, some js behaviours or any combination of these. We have some components which do not have a twig template, obviously they wouldn't be directly renderable.

On requirement #4 Directory structure. When using an external pattern library we should not assume a directory structure, it would be nice to be able to alter these paths. This matters for things like asset paths.

On requirement #5: The YAML file specifies. We are currently integrating our own components from a pattern library and have added additional values that we feel we need:
* declaration of shared assets, images/sprites, fonts
* dependency on other components for twig, css, js or the above assets
* basic validation on variables (e.g. we feel it would be the components duty to set a character limit on say a title field)
* configuration variables which provide some options (e.g. add a class to a component dictating whether it should be a landscape or portrait variant of a component)

As you can see we are beginning to feel that the pattern library should be informing the application/Drupal about a limited set of conditions it has for the use of a component.

Thanks all for the big push on this topic.

heddn’s picture

It might be apt to mention https://www.drupal.org/project/components as a contrib module doing some work in this space.

Wim Leers’s picture

#39:

We have some components which do not have a twig template, obviously they wouldn't be directly renderable.

The patch in #1 already includes an example of this: a theme may extend an existing component, and do nothing except add a CSS file.

This matters for things like asset paths.

You're already free to put assets anywhere in the patch in #1.

additional values that we feel we need:
* declaration of shared assets, images/sprites, fonts
* dependency on other components for twig, css, js or the above assets
* basic validation on variables (e.g. we feel it would be the components duty to set a character limit on say a title field)
* configuration variables which provide some options (e.g. add a class to a component dictating whether it should be a landscape or portrait variant of a component)

The first two bullets are already supported by the patch in #1. The third bullet is an interesting one that could indeed be valuable, we should investigate that. The last bullet would simply be another input. Or, if it's truly a variant (like imageimage--with-rollover, then there'd be a component X with X--portrait and X--landscape variants. Like the issue summary says: dealing with variants still is fairly undefined. That's definitely still going to happen.

kalpaitch’s picture

Nice work. I couldn't see it mentioned but namespacing of components could be useful. I can imagine an external library with no knowledge of the components in Drupal having clashes but not wanting to overwrite or extend said components.

Crell’s picture

I don't recall what I was thinking when I said we could auto-generate PHP classes, honestly. It's been a while. :-) Generally, though, my point is that we want as strong a type checking as possible, both for DX and for simplifying the code itself. We've tried anonymous structs, we know we hate it, so let's try typed/named structs this time. :-) (PHP lacks formal structs, so classes are the closest we've got.)

One thing to consider on the DRY/duplication question is that Rails takes the opposite approach: The latest Basecamp (co-developed with latest Rails) sends pretty much everything back to the server to be rerendered and then injected into the page. That avoids a ton of duplication. Of course, it also adds overhead, and that overhead is larger for PHP than Ruby because of the need to rebootstrap every time. That could be solved with websockets, but then we'd need a websocket PHP server. (See debates in various other issues.) In any event, the take-away I'd offer is that we don't necessarily have to have perfect 1:1 parallelism between PHP and JS. If there are some things that require going back to the server to rerender while JS can natively do much/most of it itself, I think that's fine. (For some definition of "much/most", of course.)

Wim Leers’s picture

Generally, though, my point is that we want as strong a type checking as possible, both for DX and for simplifying the code itself.

+100000000

Implementation details TBD, we'll figure those out.

Bojhan’s picture

Fascinating to see this thread. We've tried several times to build a solid component/style library for Seven but failed to do so as the technical limitations keep getting in the way. While this clearly focuses on a much larger problem, it's nice to see that it might lift some of those limitations. It might be a bit early to say that atomic categorization is the most sensible - but I think for Seven we can go with whatever default is provided and adjust accordingly.

Great work, and good to see the structured consensus building - a good example for others who wish to tackle difficult challenges.

mdrummond’s picture

I took some time and reviewed Wim's mega-review as well as the original patch.

What has been developed so far seems to be geared mostly towards how to implement the lowest level of components, which I had described as data components. That's where simple variable types and strict typing would be most useful, and where the mapping of theme functions to components is likely to be most relevant.

However that is just one part of what is necessary for a component-based theming system.

There are lots of "votes" for no drillability for templates which, to be clear, tends to be somebody saying "we need drillability!" and then that getting counted as "we will not have drillability."

I think drillability means a couple things. One, an item component (let's say an event teaser) could have a list of dates and times when that event takes place. Each of those date/time items would be a data item, while the list itself would be the data component. Think of a data component as a field, and a data item as one piece of data in a multivalue field. So could there be a template for a date/time data component. Sure? But what I care about most is that event teaser component. I'd like that template to have something like (ignore whether this is correct Twig syntax, the concept is what's important.)

<ul {{ datetime.items.Attributes.addClass('event__datetime-list') }}>
{% for item in datetime.items %}
  <li {{item.Attributes.addClass('event__datetime-item') }}>{{ item.value }}</li>
{% endfor %}
</ul>

This would in theory allow you to still take advantage of whatever gets added in via whatever replaces preprocess, but still allow you to define the exact markup that will be used for that data component within the item component. This sort of drillability is essential because without this you will still see numerous templates in order to create the markup for one item component with multiple data components.

The other thing that is necessary is being able to nest components multiple levels deep. A page component contains multiple section templates each of which could contain combo components, component series (river of news) or individual item components, which would contain multiple data components. Component nesting is absolutely necessary.

What is also missing from this right now is where the actual context/data gets into templates, and particularly how that travels from high-level components down all the way to low-level data components. For example, let's say that in our event teaser item component has a data component that contains the event organizer name, wrapped in a link to that organizer's page on the site. The data component needs the name and a URL, but those values need to come from somewhere.

Let's the event teaser is nested in a list of events. That is is nested in a main content section for an events landing page, which is nested in an event landing page component. So where do we get that organizer name and URL? Do we pass up the need for those simple values all the way to the top of the chain? Ultimately that info is likely to live in a user object. I think it's probably useful for that event organizer data component to know that its caching will vary based on that particularly user object info, so that if that user info is updated (typo in name is fixed), then the caching for that event organizer in that teaser gets updated. Ultimately that caching info needs to bubble up through all those levels that contain that data component, all the way up to the event landing page component. So do we pass that user object all the way down to that data component, which takes care of transforming the user object into the values we need in the template? I don't know the right answer, but we need to figure that out.

Right now, we use preprocess to transform Drupal data into template variables. This patch sort of does that transformation in a Twig template, mixing that data logic in the same template as the markup and the logic to determine classes for that markup. I think we could improve this by having one Twig template that serves as the preprocess replacement, and another for the markup/classes logic. That would allow us to keep themer-friendly markup-based templates, while also having the replacement for preprocess logic in Twig, where it could be shared between client and server down the road.

Part of the benefit of preprocess is allowing multiple modules and themes in the theme tree to affect the variables for a particular template. I think we could replicate this by allowing modules/themes to register a preprocess Twig template that could add something to an Attributes object if it exists (or create one if not), or things like that. In the final component compilation, those preprocess templates could be included one after the other in a proper order in order to prep variables, while the final component template in the chain takes in those values. The patch documentation notes that variables can only be added by components: I'm not sure if that excludes changing the values of variables, but that seems a pretty valid use case.

-----

I also wanted to note that this work on components is really very closed to the work on layouts and blocks.

What I care most about for components is that one component represents one chunk of the design. This initial work focuses on custom component templates, which is great for front-end developer focused implementations. This will ultimately need to work for site builders as well. For those use cases, components will need regions where an arbitrary number of smaller components can be placed: essentially, semi-generic layouts for components.

So as the layouts and blocks initiatives proceed, we should make sure we look ahead to the future to make sure layouts and components work well together.

cyb.tachyon’s picture

Just noting that Progressively Decoupled Blocks (PDB) is also doing some work in this space for ingesting JS Framework-based components to Drupal Blocks (and maybe soon Twig namespace components) - we commented on the connection here: https://www.drupal.org/node/2707849

Sandbox: https://www.drupal.org/sandbox/mrjmd/2664138

Wim Leers’s picture

FYI: Here is the link to the notes from the ​Theme Component Library​ discussion we had at DrupalCon: https://docs.google.com/document/d/1TJjvsF8NpRIW_YrP7hdb3ougnc7Kk7nZFP09... — they include the diagrams that were drawn.

aleksip’s picture

I think that one thing closely related to this issue is how stand-alone components are developed, packaged and used. I just published a blog post containing some thoughts about stand-alone component projects.

It would be great if we could reach a consensus on how to do this and gain component interoperability not just between different Drupal themes, or between Drupal and decoupled client-side front-ends, but also between Drupal and tools like Pattern Lab.

cosmicdreams’s picture

Do we have actionable steps to pursue now that we've hashed some of the details out at Drupalcon?

Wim Leers’s picture

We didn't hash out details at DrupalCon. The discussion was productive in that we've had even more requirements and goals stated by even more people than those who participated in this issue so far. But because of that, almost no actionable steps were identified (and in that sense, it was not productive at all). It is absolutely impossible to write up an initiative proposal like https://www.drupal.org/core/initiative-proposal-template.

The only actionable steps are basically:

  1. bring John Albin's #30 into this patch
  2. bring the aspects of this patch to John Albin's Zen for Drupal 8 theme

i.e. bring the two together, because they mostly cover different ground.

That is also what is listed at the very bottom of that Google doc:

  1. Try Zen try to break it
  2. Update Zen: let a component have a YAML File that lists its assets, register it as a library automatically, attach it automatically -> see Wim’s patch
  3. Move away from namespaces in Zen, but register theme hooks -> see Wim’s patch
  4. Figure out how to ensure that components can be Angular, Ember, whatever components: ensure that the inputs a presenter passes on to components are serializable (into JSON)
  5. Look for outside solutions
davidhernandez’s picture

A good first step would be documenting a proposed API, and how a themer would actually use it for various use cases. We've talked some in person about use cases (user stories) but haven't written them down. All the wiz bang zoom here is meaningless if the TX turns to shit and we end up with a worse experience, undoing all the progress made with 8. At the end of the day, a themer/site builder, not a render system engineer, needs to work with this. I need to know how I would actually make things, how inheritance from core->module->base theme->base theme2->sub theme will work. How to override some things versus other things, etc.

Wim Leers’s picture

All the wiz bang zoom here is meaningless if the TX turns to shit and we end up with a worse experience, undoing all the progress made with 8. At the end of the day, a themer/site builder, not a render system engineer, needs to work with this.

+1000.

But, nothing is fully formed yet. It is impossible to fully design the API now, because we need it to work in a BC way, so no doubt that we'll need compromises. IMO the only way to achieve what you describe (and again, I agree with that) is by actually doing the work of converting parts or the entirety of Drupal core, to verify that it works in all cases, and to demonstrate/prove that the TX is in fact better.

So, for at least weeks (and probably longer), this issue will remain fairly vague.

I will be working on applying the few next steps that we did identify over the coming days.

Wim Leers’s picture

I started working on this again a few days ago. Unfortunately, after applying the patch, I noticed several big problems… which somehow I didn't notice before due to last-minute changes, or because of changes in Drupal core itself.

  1. interdiff-1.txt: The pattern library (/admin/components/pattern_library) was not rendering the components correctly. This was an easy fix.
  2. interdiff-2.txt: The style guide (/admin/components/style_guide/bartik etc.) was having a similar problem. The prior point/fix didn't fix it. Turns out there was a significant problem with template inheritance. i.e. #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance. That prevented even my explicit {% extends "@system/image.html.twig" %} from working.

    To address that, I took on the task of actually fixing #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance. Thanks to an idea from @Fabianx combined with some research of my own, I've got a working solution for that, which uses the theme registry at Twig template compilation time to transform extends "FOO.html.twig" to extends ['path/to/parent/theme/FOO.html.twig', 'path/to/grandparent/theme/FOO.html.twig'] and include "FOO.html.twig" to include ['path/to/current/theme/FOO.html.twig', 'path/to/parent/theme/FOO.html.twig']. In other words: this transforms Twig templates at compilation time to respect our theme registry, but still uses native Twig functionality, therefore simplifies Twig debugging, but doesn't put the burden on the themer to specify every parent theme template.

    See #2387069-26: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance.


Apply individual commits to recreate all of the work (my entire local branch) by applying the "individual commits" patch using git am PATCH! (Created using git format-patch origin/8.2.x..HEAD --stdout > PATCH.)

Wim Leers’s picture

My next steps:

  • Get #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance to RTBC.
  • Horizontal extension AKA modules implementing hook_preprocess(): for example Contextual Links and Quick Edit needs to be able to affect/alter any/some templates, without there being a direct relationship to them. @mdrummond has proposed to let those modules provide a Twig template that would be automatically prefixed to (all or only the appropriate, TBD) other Twig templates, which would allow them to effectively alter the variables that those templates will end up printing, without that needing to live in Drupal PHP land. Because once it's compiled into Twig, it can also be rendered outside of Drupal, and even on the client side.

    Doing this first because #54 has already forced me to dive deep into Twig land, and this touches on similar areas.

  • Make components non-Drupal-specific AKA implement what John Albin describes in #30.

    Doing this second even though it's really the most important thing, because I think #54 + the first thing listed here will result in me working on this with more complete Twig knowledge in my head.

Wim Leers’s picture

Done:

  1. I got #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance to pass all tests, have expanded test coverage, and so to an RTBC-worthy state.
    In doing so, I worked quite a bit with found a bug/limitation in Twig, for which I filed an upstream PR: https://github.com/twigphp/Twig/pull/2069.
  2. I skipped horizontal extension for now — in working more on #2387069, I got plenty of exposure to Twig internals.
  3. So I started working on presenters, AKA implement what John Albin describes in #30. See attached patch. For now I didn't put presenters in a distinct location: I just modified some existing templates to not do their own rendering, but {% include … and {% embed … some components. Not sure yet if this is best or not, but it means less change for sure. It seems to work out fine for John Albin, Micah Goodbolt, Phase 2 and others.

Next:

  1. working with John Albin to bring some of the concepts of this patch to John Albin's Zen theme and his https://www.drupal.org/project/components module — most notably the YAML files that centralize all the metadata, and automatically attaching asset libraries, just like the patch in this issue does
  2. attempt to convert the most representative thing of all to presenters+components: nodes. Because the tricky thing is that in Drupal we do: controller -> render array -> render the render array, and the rendering of the render array looks like this: render array (and its callbacks) -> Twig -> render array (and its callbacks) -> Twig -> REPEAT. It's those (#pre_render) callbacks that look at the configuration the site builder set up in Field UI and convert those into render array declarations. And then it's the rendering of those render array declarations that ends up calling into the appropriate Twig templates. This happens both at the "field in entity" level (field order + formatters) and the "item in field" level (field formatter settings). In other words: it's not Twig templates all the way down, and we need it to be if we're going to make this work. So we'll need Field UI's configuration changes to cause a Twig template to be generated to match that configuration. A very important consequence of this will be that themers can copy/paste this generated Twig template and use it as a starting point. So, this has the potential to also address the long-standing frustration themers have that site builders can mess up their work — if a themer provides a customized entity template, then we would effectively disable the field UI for that entity type, to signal that this is not possible. (Yes, lots of details to be figured out, but hopefully that sounds exciting at a high level!)
  3. Then with that better understanding of the impact of presenters, clean up/refocus this patch. The introduction of presenters as a concept means all of Drupal's existing templates can simply be updated to use components. Which means much of what is in the issue summary is then … well, no longer accurate. The #component stuff in render arrays, for example. If we go forward with the plan of "presenters" as John explained on this issue and for example Micah Goodbolt has observed many people discover independently (which is always a good sign), then that means the V1 vision I explained in the issue summary no longer holds: roughly speaking, it would not be render arrays that know about components, it would only be Twig "presenter" templates.

I'm starting to think that the best way to make this first leap forward is to help refine Zen + the components module, and then introduce a componentized_bartik theme in 8.3 (or a completely different theme), which would then serve as an example. Because it already is a steady improvement for the TX. That would not be a solution for Unify & simplify render & theme system […], but it would reduce the surface area of the render & theme system, and that's why I think that could be a great first step:

  1. very doable: no API changes necessary
  2. Themer Experience gain AKA smaller pain-surface-area: more stuff in Twig, less stuff in render arrays/PHP
  3. which means the grander ambition that this issue has then becomes easier to realize (smaller surface), even more so because we'll gain valuable experience (sites already using this technique)

This would be the first time that Drupal's front-end experience is actually improving rapidly. And it'd be completely opt-in.

The most important caveat would be that we need to be careful we don't paint ourselves in a corner that wouldn't allow the bigger goals from being achieved.

Fabianx’s picture

+1 to #57. I do think this would be great to add components in this step-by-step fashion.

Wim Leers’s picture

Just discussed #56 in the weekly component-based rendering meeting that happened today.


There's resounding support for what I proposed in #56:

I'm starting to think that the best way to make this first leap forward is to help refine Zen + the components module, and then introduce a componentized_bartik theme in 8.3 (or a completely different theme), which would then serve as an example.

markconroy and lauriii expressed strong interest in actually helping make this happen. (Nobody opposed — +1s from: markconroy, lauriii, davidhernandez, cyb.tachyon, prestonso.)

And as it happens, lauriii was already working on an new theme initiative proposal anyway! lauriii was happy to slightly modify that initiative proposal to ensure that it doesn't just demonstrate best practices at large, but also specifically those for using components.
mdrummond then mentioned that mherchel was also very interested in helping out with a new theme.

Conclusion: it looks like we'll get a New theme initiative for 8.3, and as part of that, we'll provide guidance on how to do component-based themeing in D8.


polonya todl me he's interested in helping me out with working with John Albin to bring some of the concepts of this patch to John Albin's Zen theme and his https://www.drupal.org/project/components module. He's already created a "UI components" sandbox module based on the patch in this issue. So it only makes sense to collaborate with him on bringing some of the things in this patch to Zen, so people can already start to benefit from some of the work we've done here.

Jacine’s picture

FileSize
22.72 KB

Re: #2702061-56: Unify & simplify render & theme system: component-based rendering (enables pattern library, style guides, interface previews, client-side re-rendering), Next.2

So we'll need Field UI's configuration changes to cause a Twig template to be generated to match that configuration. A very important consequence of this will be that themers can copy/paste this generated Twig template and use it as a starting point. So, this has the potential to also address the long-standing frustration themers have that site builders can mess up their work — if a themer provides a customized entity template, then we would effectively disable the field UI for that entity type, to signal that this is not possible. (Yes, lots of details to be figured out, but hopefully that sounds exciting at a high level!)

Long-standing frustration is pretty much an understatement, but yes! ;)

In my failed quest to respond to this thread earlier, this was one of the issues I started to write up. I also started to take some rough notes, on things I think can improve the situation, and started on a comp containing a limited version of the manage display screen. I see this issue as a big a barrier to ending up with a good result here. It's also one of those issues that causes conflict/divide within the front-end Drupal community. We have:

  1. Mothership themer: Those that just do everything they possibly can in templates (everything except formatters), screw the consequences. For these sites, the Manage Display UI, is 100% frustration and nonsense as no matter what settings are applied, the template never reflects them.
  2. Zen themer: Those that do a mixture of things (like setting formatters, label settings). There's usually a heavier emphasis on field theming. For these sites, the Manage Display UI is 90% frustration. They may be able to re-order some or fields, change formatters, and possibly even labels... but maybe not. It's a crap shoot. Also, click, wait, click, click, click, wait, undo stupid defaults, rinse, repeat.
  3. Everything in between CSS themer and Site Builder: Manage UI almost works for them, but they probably end up with Panels, DS or Paragraphs anyway, because most designs require markup grouping/layouts. Their UX is still terrible because of stupid defaults, and a million clicks, and their HTML is straight up terrible.

Bottom line: The situation is not ideal, or even really acceptable in any of the above cases. By trying to be all things to all people, we have created this mess. It would be beneficial to all involved to stop ignoring the fact that there is a huge disconnect between UI Config (Manage Display screens) and templates, and that they invalidate each other causing a terrible user experience on all sides. It appears we are finally willing to admit that it is hindering our plans to improve Drupal in general. Yay, that's the first step toward recovery. LOL.

My rough comp is attached (PDF), and my notes are in this gist, which I've pasted below:

{#
  2-way communication between UI config screens and templates.
  -----------------------------------------------------------------------------
  The Manage Display screen should be aware of anything this template affects.
  This information is less useful if the base template is in use, i.e. node.twig
  as opposed to node--article.twig, but consistency in communication is more
  important, so doing it globally is best. The template should be gleaned using
  the same logic as theme hook suggestions.
  It's presence alone doesn't necessarily mean the theme wants full control over
  its display. However, in these cases where both UI and template settings are
  competing, displaying a message with a heads up that is happening with X
  template, is at least something, as this is a huge problem that is nearly
  impossible to document meaningfully for the many stakeholders typically have
  access to these screens, such as site builders, administrators, back and front
  end developers.
#}

{#
  Disable configuration
  ------------------------------------------------------------------------------
  Explicitly state this template is taking over the configuration. This might be
  something that can only be done with a specific override, as opposed to a base
  hook template. Not sure. What I'd expect it to do this, is the following:
    1. Provide a message stating that the template has overridden the ability to
       use this screen.
    2. Disable drag and drop UI.
    3. Provide simpler UI that allows 3 things to happen for each field (in
       order of importance).
        a. Set field formatter and settings.
        b. Toggle field visibility
        c. Set label settings (which should be hidden by default like in Views
           module and located in a single settings form with formatter settings.
#}

{% set ui.configuration.ordering = FALSE %}
{% set ui.configuration.formatters = TRUE %}

{#
  Disable fields
  ------------------------------------------------------------------------------
  Need to disable fields that we don't want, like "hidden" on "Manage Display"
  screens. I realize we have {{ content|without('field_whatever') }}, but this
  isn't the same thing. This should stop the field data from loading, as well
  as prevent any configuration, such as assigning a formatter and settings.
#}
{% set ui.configuration.fields = [
    'field_category',
    'field_tags',
    'field_node_links',
    'field_node_author',
    'field_user',
  ]
%}
Fabianx’s picture

Jacines post (which is great), remembered me that I worked on an MVP for a two-way communication with the theme for layouts in core at DrupalCon and discussed that with several people:

https://docs.google.com/document/d/1JGmMwColdzXrJUqjKX1jkwbwHTd9eHKd2GS_...

What the document does not reflect is that there are several things mixed together (theme suggestions as layout finder, layout plugins (which already exist), etc.).

I am leaving it here anyway. I do think even Jacines part could be combined with my approach:

Technically the easiest implementation is to create a {ui_configuration} twig tag, e.g.

Instead of:

{% set ui.configuration.ordering = FALSE %}
{% set ui.configuration.formatters = TRUE %}

it would be:

{% ui_configuration %}
  {% set configuration.ordering = FALSE %}
  {% set configuration.formatters = TRUE %}
{% end ui_configuration %}

The advantage of an explicit element is that two way communication gets easier as this would be compiled to:

  class SomeTemplate() {
    public function getUiConfiguration($configuration_defaults) {
      $configuration = $configuration_defaults;
      $configuration['ordering'] = FALSE;
      $configuration['formatters'] = TRUE;

      return $configuration;
    }
  }

So it is as simple as loading the template and calling getConfiguration() on it to communicate with the UI.

Note: I am not debating whether or not the approach to set the variables in this way or those variable is a good idea or not, I am showing how a technical implementation could look.

Edit, this would look for the UI code something like this in pseudo-code:

$template = $this->twig->loadTemplate($template_name);

if ($template instanceof Drupal\Core\Render\ConfigurableUiTemplate) {
  $configuration = $template->getConfiguration();
}

cosmicdreams’s picture

In this possible future where the template has told Drupal that it will be control over the display of fields for an entity type:

1. Does this control need to be as granular as having control on a display mode level?
2. We should probably provide a message in the UI informing the administrator of the specific file that has control over the display.
3. Provide logic to conditionally control ordering but not formatting and vice versa

cosmicdreams’s picture

Also, would this diminish the need for modules like fences or other modules that attempt to "clean" the markup of fields?

jonathanshaw’s picture

There's also the den of controversy that is #2289619: Add a new framework base theme to Drupal core. Should this be born in mind as a future possibility to allow for?

Andre-B’s picture

2. We should probably provide a message in the UI informing the administrator of the specific file that has control over the display.

+1 on that. similar to how views states overridden template files.

Provide logic to conditionally control ordering but not formatting and vice versa

+1 there need to be ways to not abandon the field ui completely / make it even harder for devs / site builders to figure out where and how output was rendered. Also those template overrides should have a way and best practice to still be extendable (adding new fields). They probably should also respond to deleting fields without breaking the twig layout. This will require thorough examples and best practices to keep as much flexibility as possible.

tkoleary’s picture

Something that I think is getting lost amidst the technical "fog of war" here are the "people problems."

Jacine hinted at this with her three personas and other comments have touched on the themer vs. site builder experience, but I still think that there are some mis-matched priorities underlying many of the comments.

On the one hand there are those who are advocating for flexibility for the themer, or the ability to override, or neutralize the ability of site builders to control display in certain instances. In other words this priority:

  1. Themer
  2. Site builder

On the other hand there are those who are working to introduce more standardization in the theme layer, I think with the idea that it will lead to more configurability and put more control over display back in the hands of site builders. In other words this priority:

  1. Site builder
  2. Themer

The problem here I think is that both the site builder and the themer are at the service of the site owner who is the ultimate customer, particularly if we assume, as I think we can in most cases, that the site builder's task is—like the themer's—time constrained to the period around launch, whereas the site owner must manage the site and make updates on an on-going basis. Additionally one could argue that since code precedes configuration, priority should be:

  1. Site Owner
  2. Site builder
  3. themer

But I think in most cases this is false, since site builder and themer are often working in tandem and can pass things back and forth, suggesting the most common case is:

  1. Site Owner
  2. Site builder/themer

If we agree that this is the case then it changes the way we think about things a bit. What it means is that, whatever route the themer and the site builder take, the "system" that they produce must serve the site owner who is their proximate user. We can assume (notwithstanding exceptions) that:

  • the site owner does not code
  • the site owner's knowledge of Drupal configuration is less than that of the site builder
  • the site owner will want to be able to configure as many things as possible
  • the site owner will need to make updates at the field or template layer months or years beyond launch
  • forcing the site owner to incur the expense of added or re-assigned staff is not optimal

There are many possible directions this might suggest, but before we go there, do people feel that the above assumptions are generally true?

mdrummond’s picture

One of the things I brought up at DrupalCon NOLA, and which I went over in detail in a recent presentation I gave about component-based theming (http://2016.tcdrupal.org/session/wont-you-take-me-chunk-y-town-component...), is that in the long term, new and improved site builder tools to work with components are going to be important.

One way to look at a component is that it's a series of slots that can accept data. Each slot can either have a 1:1 ratio, where it accepts an individual piece of data (with requirement on what sort of data is expected for that slot), or a 1 to many ratio, where it can accept multiple pieces of data.

A 1 to many slot would be sort of like a region. In a section component, a 1 to many slot might contain multiple item components. In a page component, a 1 to many slot might contain multiple section components. In an item component, a 1 to many slot might contain multiple data components (essentially fields).

With a 1 to many slot, a themer should not expect to control the order of the components within it. If that level of control is necessary, create a 1:1 to slot for an individual piece of data. That can set expectations for site builders that some slots on a component have controls on how things are ordered; others, not so much.

With a 1:1 slot, a themer would be free to move those around within the component's markup as necessary.

With a 1:many slot where a site builder is controlling the order, I like the idea of Drupal generating a Twig template file behind the scenes that contains a set of include statements specifying that order. If that auto-generated Twig file could be included within the main component template file, that would simplify the process of using that component's markup in external sources like client-side JS and pattern libraries.

One thing that Panels does to help site builders is allowing for the inclusion of an admin layout CSS file in order to give basic layout styling within the admin view to help show how various layout regions relate to each other. Allowing some sort of component layout admin CSS file could do the same thing, helping to set expectations for site builders.

Integration of components with site builder tools might be a longer term goal, but I think that's going to be a more reliable way to communicate expectations with site builders what they can do with the UI in terms of making changes to the appearance.

Wim Leers’s picture

Status: Needs review » Postponed

So, as I suggested in #56, and as was confirmed by others in #57+#58, we're going to let this issue rest for a while. We're shifting our focus to #2759849: Create a new user-facing core theme instead. Hence marking this issue Postponed.


This issue at a high level

The high-level direction I set for this issue has garnered wide support. But as soon as we start to look at more specific technical details or more specific goals, there are dozens of strong opinions. It's already extremely hard to unify & simplify something that has grown organically for about a decade (it has become clear after posting this issue and working on it more that even all the cache tag work I did in Drupal 8 core was laughably trivial and tiny in comparison). But add to that the requirement that we cannot break BC, yet we must provide more simplicity, plus dozens of opinions/goals to unify… that makes this issue very hard to move forward.

So, instead, let's focus on #2759849: Create a new user-facing core theme for Drupal >=8.3.x core and https://www.drupal.org/project/components for Drupal >=8.1.x contrib.


This issue at a low level

The low-level direction I set for this issue has gotten lots of "yays", but over time it's become clear that the precise technical plan I've written out in the issue summary will not work. I started out with "make everything a component", where "component" means "make all #type and #theme stuff #componen, and let those components be much more strict and defined without any PHP code". I still think this is one of the most doable conversion processes that yields tangible benefits.
However, since then it's become clear that this is not ambitious enough: it still doesn't solve the one #type/#theme/#component per use case problem that catch pointed out in #9 (which we tried to address during the conversion to Twig in Drupal 8, but we failed) — @mdrummond also voiced a similar concern (just translating all existing […] into equivalent components). @JohnAlbin posted a most excellent providing his thinking and experimentation around this. In his approach, we don't even need #component; components would live solely in Twig + accompanying metadata file . His solution for getting data into those Twig-based components: presenters, which "route" or "map" Drupal data into Twig components. This is most elegant. And better yet, at least two other teams independently came up with the same approach! See https://micah.codes/a-new-design-system-architecture/. Perhaps best of all: this doesn't require any big API additions or changes!

So, lots of good reasons to get much more experience with "presenters" in #2759849: Create a new user-facing core theme, before we continue this issue/patch. Because presenters are key to maximize reuse of components across different "Drupal thingies": different presenters can use the same components easily. And components can easily be written in a Drupal-independent manner.

In other words: onwards in #2759849: Create a new user-facing core theme!


Reading & watching

In the #components channel in the "Drupaltwig" slack, I've been given many interesting links by people providing feedback. Recommended reading/watching material for sure. For your sake, I'm including that whole list plus my brief thoughts on them:

The React/Flux world's parallels to "presenters & components": "container components & components":
… but turns out they are mostly very different, and serve rather different purposes (React uses GraphQL-based pull to get data, Drupal still mostly uses push, mostly due to site builder-based configurability). I don't yet see anything at this time to bring to the Drupal world. It's also a fairly different definition of "components".
On presenters & components as a sound architectural direction:

The first two provide convincing arguments supporting this direction. The latter I would consider a hack that's only useful/interesting in a very narrow context, I don't see how it's generically useful (it's flawed, as the project page itself already points out) — although it does show how things hidden behind #pre_render can be a problem.

On fetching data for components:

I think this may end up be essential if we want to have a simpler rendering system that's ready for the future. If we can "bubble up" our data requirements much like we already bubble up assets and cacheability metadata (but those happen *during* rendering, this would have to happen *before* rendering), then interesting possibilities open up.

On PatternLab integration:

I think we should totally look into bringing that to #2759849: Create a new user-facing core theme, I'll take that on.

On the status of Web Components and how that could potentially be useful:
  1. https://www.youtube.com/watch?v=J4i0xJnQUzU&feature=youtu.be&t=33m50s
Links that were recommended, but that turned out to be completely unrelated:
  1. https://www.youtube.com/watch?v=_ctXyGUU7gI&feature=youtu.be&t=1h8m13s was super trivial and not worth watching. It shows how Craft uses Twig, but it turns out there's nothing interesting there, it's basically using Twig to do page assembly, just like PHP was used to do page assembly in the early 2000s
  2. https://docs.expressionengine.com/latest/channel/channel_entries.html → basically a super simple, extremely limited Views module, but without a UI, just a particular syntax.

My status

I'm currently working on things that must happen before 8.2.x is frozen: features for editor.module, ckeditor.module and rest.module. I'm also working on closing the few issues that big_pipe.module has, so that it can be either marked non-experimental, or at least beta-level stability in Drupal 8.2.

Then starting at the beginning of August… I'll be AFK for about a month (getting married and honeymoon!). When I get back, hopefully #2759849: Create a new user-facing core theme will already have started doing some work, or has at least defined a design direction, so that I can play the role of technical unblocker there.

In other words, this issue is put on hold a bit. I will try to contribute https://www.drupal.org/project/components in the coming weeks, I definitely will when I get back, and I encourage everyone here to do the same!

Wim Leers’s picture

Issue summary: View changes
Wim Leers’s picture

Some people told me that the first Medium link is not being linkified in my comment. Turns out this is either a D7 core bug, or a bug on d.o. Filed an issue: #2765681: URLs not linkified (D7 core bug? Works fine in D8).

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

webchick’s picture

So, watching how #2759849: Create a new user-facing core theme has spun completely off the rails to arguing about technical implementation details, I'm really wondering if it's a good idea to postpone this issue (which we know we need) on that issue (which is "nice to have" by comparison). We know front-end developers find Drupal difficult to work with because our front-end code is not componentized enough. I have never heard of anyone complaining that they refuse to use Drupal because Bartik looks dated. (That target audience is going to replace the default theme anyway.)

The rationale for needing a firm target to try and work out the details here makes a ton of sense to me. But then why not take one of our existing themes (either Seven or Bartik) and compontentize it? Then a) we can start on it more or less immediately, b) we know when we're done, and c) we're not pushing another 6+ months process to find a designer (without a budget) in front.

Thoughts?

catch’s picture

Yeah it also seems odd to postpone this. It's one thing to stop working on it actively, but there's no hard dependency.

Additionally, I think there's still a lot of mileage in #1804614: [meta] Consolidate theme functions and properly use theme suggestions in core for minor releases. The less unique theme hooks core is using, the less things have to be considered for porting to components (this was my original hope for the Twig conversion too, but it ended up being quicker to do lots of ports in the end).

Bojhan’s picture

Status: Postponed » Active
davidhernandez’s picture

I think it was postponed on the belief that it is hard to decide what kind of system we want without experiment on an end result first. You don't know what tools you need until you try to build the ship. It is a bit of a chicken and egg.

Lukas von Blarer’s picture

@davidhernandez true. But @webchick's suggestion to use an existing core theme as a proof of concept seems to be the perfect solution to this, right?

cosmicdreams’s picture

Seems like a good idea, as long as we acknowledge that whatever theme is made in the future may not actually reuse these higher level components or may need lower level components to diversify. There might be a lot of rework. I still think the exercise will be educational as this is one of the things we want to investigate: how much reuse of components we will achieve.

If we analyze an administrative theme like Seven, we should also acknowledge that the while there is some commonality of components with user-facing themes there a large difference of use and therefore a large potential for describing components that user-facing themes may not use.

All I'm really trying to say is that I think this effort will be great, if seen as a warm-up or practice run of the work we will need to do to itemize, categorize, and describe the components that the new theme will need to provide. If we focus on the process rather than the results we may be able to get this process down for when we need to do it for the theme.

mdrummond’s picture

Wrote up a blog post with some thoughts I've been having on this subject: https://www.marcdrummond.com/posts/2016/09/26/why-base-drupal-theme-syst...

Gábor Hojtsy’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: theme system » Plan