Problem/Motivation
Drupal's render & theme system are too complex to use. Let's improve this.
Goals
- Improve the TX by un-WTF-ifying the theme and render system, which is currently a maze of:
hook_theme()
(withvariables
vsrender element
), preprocess hooks, theme suggestions and many more related hooks — all tied together using the theme registry#type
(@RenderElement
plugins) vs.#theme
, which are kind of the same thing but not really: when to use which is unclear- callback buffet:
#pre_render
,#post_render
,#lazy_builder
- mysterious keys in enormous render arrays AKA render arrays of DOOM
#render_children
,#theme_wrappers
and friends determining where the render system morphs into the theme system and back again- different systems calling each other: understanding the entire flow is nearly impossible, and probably rivals the complexity of some simpler biological organisms
better documentation and and examples are band-aid solutions — they address the symptoms, not the cause
— @c4rl
These have been known problems for years! First there was Form API, then Render API sprouted from that, and all the while there was the theme system, but starting in Drupal 7, the Render API and the theme system got deeply, deeply intertwined. Drupal 8 actually made it slightly better, but not enough.
- Improve the TX for non-JS front end people (
markup & CSS people
):- automatically generated pattern library (== all
#type
/@RenderElement
s — but without having to know those details) - automatically generated style guides for every theme (== pattern library with the theme's overrides/extensions applied)
in other words: bring style guide-driven development to Drupal as a default rather than a labor-intensive, hacked-on after-thought (without the need to duplicate markup and thus keep them in sync).
- automatically generated pattern library (== all
- Long-term: make it possible to reuse templates on the client-side (
JS people
). - Retain compatibility with the existing Render & Theme systems. Allow for a gradual transition.
- Support interface previews: #2632750: Interface previews/skeleton screens through optional "preview" or "placeholder" templates
Requirements
To address all of those goals, I believe a component library can be the gateway to sanity a solution. It can be, if the following requirements are met:
- components are not deeply tied to Drupal, and in fact, can be developed independently of Drupal — this is how we can guarantee simplicity and ease of getting started: we actively prevent components from being tightly coupled to Drupal code
- components have:
- markup:
*.html.twig
(Twig template — which may include some logic to process received variables, just like in Drupal 8) - assets: CSS, JS files
- metadata: YAML file
Nothing else.
- markup:
- modules and themes can specify components — modules can define patterns that any other module can use, themes can specify theme-specific components
- components are defined in a simple directory structure:
<extension> (module or theme) |- components |- <component name> |- <component name>.yml |- <component name>.twig
Concrete example:
core/modules/system |- components |- label |- label.yml |- label.html.twig |- label.css cores/themes/classy |- components |- label |- label.yml |- label.html.twig |- ajaxified-label.js
- The YAML file specifies:
- the variables (inputs) of the component. For each variable:
- type: only A) primitives such as
string
/integer
/bool
, B) arrays of primitives such asstring[]
, C) other components:component
orcomponents
to slot in other components (perhaps evencomponent:<name>
to only allow certain components ) — this enables 3 big wins:- improved TX: type validation, to avoid weird bugs
- improved TX: no messy Doxygen/PHPDoc comments repeated in both templates and preprocess functions, and all overrides of either of those
- client-side re-rendering
(Also: having type specifications in Twig templates instead is A) undesirable, B) quite likely impossible, C) quite likely impossible to parse without refactoring Twig, D) would pick up calculated variables.)
- description
- default value, if any
- example value (to be used in pattern library & style guide)
- preview value (to be used when the component's data is not available yet, because it's being used for an interface preview)
- type: only A) primitives such as
- documentation: purpose, when to use, how to use, accessibility, related links — in other words: information to show in the pattern library & style guides
- less important metadata: human-readable name, which other components this component includes, whether this component supports interface previews …
- the variables (inputs) of the component. For each variable:
- The Twig template (
*.html.twig
) performs all the necessary processing of the variables received, this ensures we don't depend on preprocess functions. This removes the need for front-end developers to dive into PHP. - components can be extended: add attributes, modify markup, and so on
- components can be composed: combine multiple components to create a new component
- To allow for a gradual transition, we cannot fully back away from render arrays nor the existing theme system. At best, we'll be able to remove render arrays, the current render system and the current theme system in Drupal 9.
Proposed resolution — or: how to transition
- Aspect 0: components
- Let components be defined as explained above.
- Aspect 1:
'#component' => …
- Allow render arrays to use components by using
'#component' => 'something'
, instead of'#type' => 'something'
or'#theme' => 'something'
. - The associated CSS and JS assets have been registered as an asset library automatically, and this asset library is attached automatically. Just define it in the YAML file.
- This is familiar and similar. The transition is simple and understandable. But why would you want to do this? See the next few aspects.
- Aspect 2: strict validation
- Let
Renderer
strictly validate'#component' => …
render arrays. - Remember that a render array is effectively a tree. Whenever a node/element/thing in the tree is a component, we can be more strict for just that one node/element/thing, but not for its supertree (parent) nor for its subtree (children). Example:
[ '#type' => 'something' 'child' => [ '#component' => 'image', '#uri' => …, '#width' => …, '#height' => …, '#alt' => …, ] ]
In this case, the properties (
#
-prefixed values of the image component will be strictly validated: only the ones specified by the image component will be allowed, and their types will be validated). This allows us to bring sanity to render arrays, one component at a time, one conversion at a time.
- Render arrays have certain special/reserved properties that are necessary for them to function correctly. Think
#attached
,#printed
, et cetera. Those will be condoned, because they don't affect the way the component is rendered. - Aspect 3: opt-in eligible
#type
and#theme
to become#component
- Help with a graceful transition, and peaceful coexistence. We gradually remove
#type => something
and#theme => something
and replace it with#component => something
But it is hard/impossible to keep all contrib code working: we can't expect everybody to simultaneously change existing render arrays to use'#component' => 'something'
as soon as there is asomething
component. - So, whenever the
Renderer
encounters#type
or#theme
, we map it to#component
if and only if they are eligible, which is:- if it was
#type
, we only do the mapping if no#pre_render
/#post_render
/… are defined on the render array - if it was
#theme
, we only do the mapping if no preprocess hooks are defined for it in the theme registry
Finally, if it wasn't
#component
originally, we don't apply our strict validation.
- if it was
- Aspect 4: new things must be components
- New patterns in Drupal core are implemented as components.
- Eventually
- Eventually, when all
#type
(@RenderElement
s) and#theme
occurrences are gone from Drupal core, we drop the old system, and we tag Drupal 9. - At this point, we will have a render array representing the entire page. The render array representing the entire page is then once again a tree. But rather than an incomprehensible tree, it is a well-defined, well-structured tree. It can even be split up into two parts that fit into each other perfectly:
- a component tree: html → page → region → block → … — and for each of those components, there are holes (where the variables go)
- a variable tree: html variables → page variables → region variables → block variables → … — the variables on each of these levels fit into the holes in the component tree
(Conceptually speaking, because subtrees can of course still be lazily built. Details in that area TBD.)
- At this point, render arrays are just an implementation detail, and we can easily simplify that implementation.
Long-form rationale
This part is less precise, but tells a hopefully helpful story of the two most important considerations: how we ended up with the current painful system (recommended reading: http://hackingdistributed.com/2016/04/05/how-software-gets-bloated/) and how we can at the same time start to make it possible to integrate with JS more easily.
And, how, funny enough, those actually need the same fundamental thing: simplicity.
JS: web apps vs web sites
"Apps" are the hot thing in software this decade. But building native apps for every platform is very expensive. So "web apps" are a thing: build once, run anywhere, deploy instantly.
Web apps must be written in JS (or at least compiled to JS), so, consequently, JavaScript is the hot language of this decade. As a result, we've seen enormous investments in JS: Node.js, browser engines' JS interpreters have become incredibly fast, and … JS-based frameworks. From jQuery UI to Angular to Ember to React to …
Everybody wants "an app". And so quite often what would have been a web site a few years ago now is a "web app". And since web apps are written in JS, this means Drupal is less likely to be chosen for those scenarios.
(I personally think the distinction between the two can be made by determining whether the business logic happens in JS on the client or on the server. Only if it's on the client, it is a "web app".)
But that doesn't mean there's no more need for "regular" web sites anymore. It doesn't make sense to build an app just to present hyperlinked information… because for that, we already have an app: the browser. We just need to feed it web sites: documents of structured content, with excellent accessibility & usability, beautiful layout & typography and most importantly: great information architecture.
(And indeed, there is a very, very blurry border between "web sites" and "web apps". I'm just trying to paint a picture of the two extremes, where the world of the web is still finding an equilibrium somewhere in between. And of course, in some cases, it makes sense to have parts on the client side and other parts on the server side.)
Drupal 8 is even better at building web sites than prior versions.
But Drupal is getting pressure to also support "web apps".
So we have a tension between wanting to improve Drupal for what it has traditionally done (improve front-end experience by improving its templates, its markup, removing preprocess functions, etc) and making it possible to build more app-like experiences with Drupal.
But Drupal is not written in JS. Perhaps at some point in the future, there will be tight integration. But we have no idea what that would look like, if it will happen.Getting ready for the future, and facing the past
However, even with the current system, we have long-standing problems. Extremely frustrated reports go back to at least 2011. The theme system and render system are deeply intertwined. It's very confusing. The experience even to build just web sites (not web apps) is far from ideal. Drupal 7 introduced the render system, but let's not forget its origin: Form API. The render system was originally just for forms, but it's since been generalized to be used for all rendering.
- Drupal 6: Form API + theme system
- Drupal 7: Form API + theme system, and both depend on the render system
Drupal 8 has made big steps forward: Twig, removal of most preprocess functions, much cleaner templates,
#theme
is gone in favor of#type
(but not at all completely…), no longer necessary to to sometimes callrender()
in templates …Unfortunately, as soon as the render and theme system interact, it's still painful.
We started working on an experiment at Acquia, where we worked with the Angular & Ember.js teams to do a prototype of what a better commenting experience for Drupal would look like. A reference implementation in Drupal using the AJAX system ("the Drupal way") versus what they would do. They had to reuse our Twig templates. A big problem there was the fact that just about every template has "blobs of HTML":
{{ content }}
, which actually contained the majority of the interesting stuff. And those blobs of HTML are… yes, render arrays!Of course, no JS is ever going to be able to render render arrays, because they're so deeply intertwined with PHP code. And it's impossible to automatically determine which Twig templates are associated with every subtree in a render array.
So this makes it effectively impossible to reuse our Twig templates in JS. Ideally, that would be possible, it'd make Drupal better prepared for the future. It's better to at least have that possibility than not to.
However, even today, and in fact, in years past, this very same problem has been a major frustration for themers: they could only go so far with achieving what they needed to achieve by creating/modifying templates and writing preprocess functions. Very often, they would need to dive deep into render arrays and implement lots of hooks.
Imagine if that weren't the case, and we'd have templates all the way down, rather than enormous blobs of the resulting HTML being defined in render arrays. Imagine if the resulting HTML was wholly based on templates. Templates all the way down. Power to the themers.
And imagine that rather than ill-defined variables, they'd actually have type hints. And validation. And examples. And actually usable documentation rather than incomplete (and duplicated) docblocks. Together, that would allow us to automatically generate a style guide/pattern library.
As a bonus, client-side (re-)rendering becomes possible.
References
A close-to-comprehensive list of the references I've used to write/build the above.
- TX
-
- #1382350: [discussion] Theme/render system problems, particularly #18 by @sun (four years ago!), @c4rl in #45, @Snugug in #56, @Crell in #59, @c4rl in #66
- #2008450: Provide for a drillable variable structure in Twig templates, particularly #1 by @effulgentsia, questioning why
#pre_render
and preprocess hooks need to be separate concepts - #1441320: Evaluate Twig template engine for Drupal core (as an alternative for PHPTemplate and possibly even theme functions), particularly #17 by @msonnabaum, and #61 by @Crell + @msonnabaum: the reason render arrays/API is unable to allow for rendering on either client or server side is that it is deeply tied with PHP callbacks. If it were just templates + variables, then we could easily send that to the client side and render there
- #1843798: [meta] Refactor Render API to be OO, particularly this from the IS:
This is because things like the image src are prepared in the field theme callback preprocessor, which isn't available yet to the node template
— but this issue still doesn't go far enough in addressing theme/render API pains, as well as #2 by Crell:Is render API even needed? With the newer, more powerful tools available to us and the shift from page-level to block-level thinking (even if that doesn't quite make it into core for D8, it will be in contrib and I fully expect it to be the default way that professionals actually build things) why bother putting OO lipstick on that pig? Just kill it and fry up some bacon.
- #2004872: [meta] Theme system architecture changes
- #2291449: Add Twig template inheritance based on the theme registry, enable adding Twig loaders
- Components
-
- #1804488: [meta] Introduce a Theme Component Library
- #2235485: [meta] Automated Theme Component Library - Overall concept/discussion
- http://jacine.net/post/19652705220/theme-system
- http://jacine.github.io/drupal/
- https://events.drupal.org/losangeles2015/sessions/drupal-9-components-li...
- http://www.marcdrummond.com/posts/2016/03/25/drupal-front-end-future
- http://patternlab.io —
Atomic design
- https://guides.emberjs.com/v1.10.0/components/
- https://github.com/eexit/twig-context-parser (note how it only works for "set" calls, it turns out to be impossible to automatically parse variables, and even if we could, we couldn't determine the expected types)
- http://webcomponents.org/
- Design systems and Drupal (Slides - Video), a talk that Crell has given a number of times on how Drupal's site building model actually fits really really well with design component thinking/Atomic design. Having our rendering layer have the same mental model can only help to reduce complexity and confusion.
- CSS/style guide
-
- #2102191: Discuss the availiable solutions to document the Seven style guide
- #2293627: [meta] Document Human Interface Guidelines and make Seven style guide, particularly #19 by @jhodgdon, #21 by @LewisNyman
- https://www.advomatic.com/blog/decoupling-drupal-without-losing-your-hea...
- https://github.com/scaninc/kss-php/
- https://github.com/kss-node/kss-node
- https://github.com/scaninc/ScanKssBundle
- http://johnalbin.github.io/flower-power/
- Other
Remaining tasks
Lots.
User interface changes
None.
API changes
None, only additions.
Data model changes
None, only additions.
Comment | File | Size | Author |
---|---|---|---|
#59 | manage-display.pdf | 22.72 KB | Jacine |
#56 | component_PoC-2702061-56-do-not-test.patch | 73.56 KB | Wim Leers |
#56 | component_PoC-individual_commits-2702061-56-do-not-test.patch | 163.58 KB | Wim Leers |
#54 | component_PoC-2702061-54-do-not-test.patch | 65.73 KB | Wim Leers |
#25 | Components UML.png | 74.73 KB | legolasbo |
Comments
Comment #2
Wim LeersI built a proof-of-concept that proves all of the above is indeed possible. The patch should apply cleanly to the commit tagged with
8.1.0-rc1
.Please look at this patch from two angles:
/components/
subdirectories: the YAML and Twig files. And look at the attached screenshots. Compare how for example the pattern library screenshot never shows any assets, but the style guide screenshot does, because Classy's extension of theform-element-label
component adds a CSS file.ComponentDiscovery
and the places where it's called/used to integrate with existing systems with as little code as possible: the theme registry, asset library discovery, Renderer.What's in this patch?
image
,container
,form_element_label
andresponsive_image
to components. Why those? Because they're a good mix of not having nesting or not (image
vscontainer
), being omnipresent and used by Form API (form_element_label
), and showing the ability to compose/reuse components (responsive_image
reusesimage
).hook_theme()
entries, which effectively makes the system unaware of the existence of these templates.hijacksintegrates with the theme registry,Renderer
, asset library discovery, provides a new YAML discovery mechanism to allow the simple directory structure outlined above to be used.image
→image--with-rollover
), how to take care of lazy builder integration, how to handle#theme_wrappers
,#prefix
,#suffix
. And much more. I also stumbled upon existing problems/bugs/limitations, such as #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance.And now… looking forward to all your feedback!
Comment #3
eatingsThis was the heady future longed for in the early days of the Twig project, and to some extent the commonly perceived (if inaccurately so!) main benefit of going Twig.
That we aren't there yet even with Twig feels much like a promise yet to be fulfilled, and at least for me why all the improvements in D8 were still something of a half-measure.
In the end, I love a lot of these ideas, perhaps this one most of all.
Comment #4
Wim Leers#3: Twig and everything related to it is what makes this issue possible. It's why I started a new issue. Because we did go forward quite a bit, we untangled quite a lot, and so now we can/should/need to take it to the next level.
Comment #5
Wim LeersFixing a bug in the IS (something that I forgot to write in the IS but that is in the patch — see
if ($expected_type === 'component' || $expected_type === 'components') {
), thanks @mdrummond for pointing that out!Comment #6
Fabianx CreditAttribution: Fabianx as a volunteer commentedGreat proposal!!!
---
So some things with my theme maintainer hat on:
- I would not do the same mistake and define just '#' keys,
I would do:
And then have a getProperties() method for a generic ThemeComponent.
In case we want something more loosely coupled I would do:
So every component has its own strict properties within the component space or is directly an object.
That means other render array properties do not interfere and obviously component objects could implement CacheableInterface - if they wanted to.
-----
One of the biggest problems "How to traverse the tree" is not yet defined enough IMHO, because how do you get from a e.g. block component to its content and what _is_ its content?
Is the title of the block also a sub-tree? Or is it a property?
Based on the AmberJS example for using Twig in JS, that seems key to me to define that properly.
-------
And for BC reasons I would still convert:
to
internally and then just taking defined # properties and ignoring all others. In fact that is sometimes that the theme system originally had (variables => ...) and which then vanished through the render array 'render element'.
---
But great proposal and good idea to make this the thing within D8 cycle, then remove all deprecated things in D9!
+1
Comment #7
dawehnerI love the proposal. One relative simple concept which is known by many people and have a similar mind model in other areas. Just think for example of panels as a UI for defining bigger components and merging those together. Once we are there we could maybe use this component building metadata to build similar experiences in JS with whatever system you want.
There are a couple of fundamental questions which are IMHO more important than the concrete way of suing YML files and folders ...:
Some other ideas what this proposal would allow:
Note:
In an ideal world the component discovery would return value objects (much like
EntityType
) which would live inside aDrupal\Component
namespace, so we don't make the mistake to couple stuff too early.Comment #8
stevectorWim,
Thank you so much for pulling together the disparate conversations and requirements. You've summarized the situation beautifully. The backwards compatibility layer is especially important.
After reading through the patch I have a lot of detail questions but I'll save those for later and start with the bigger picture.
Do you see Components as defined here as an alternative to Layout Plugins? #2296423: Implement layout plugin type in core In the presentations and writing I've done on the topic I've said that Layout Plugins are the closest thing we have components in the Drupal ecosystem. Marc also summarized the idea in the blog post you link above. I know there is a long way to go on this patch but you've gotten incredibly far already. I suggest postponing adding Layout Plugins to Core until it can be determined whether components as they are defined here could be used in Display Suite and Panels. I'm guessing the answer to that question is "yes, with some tweaks (and maybe a compatibility layer between components and layout plugins)"
---------------
I agree! This Drupal-WTF is one of the worst parts! I see how converting some theme functions to components has the chance to clean up this problem by breaking "content" into other variables. But for a template file like
views-view-unformatted.html.twig
I think it would still have{{ row.content }}
where this variable is printing a component.People would still do a similar type of digging to find out what happens inside of that content variable. Or do you envision something else that would better highlight the "templates all the way down" model that David also quoted in comment 3? When I've used Twig outside of Drupal (in Sculpin) I've done things like this in an image gallery twig file:
That approach make it very comprehensible to the person opening the file which template file is used. Specific variables can be passed to the include. Twig's own namespacing system even allows for overriding image.html.twig without changing the include statement. Do you see a way that this component system could highlight, in a similar way, the usage of a component by a component?
Comment #9
catchOverall I really like this. One issue not mentioned in the issue summary is #2060783: Remove the preprocess layer.. I was very sad we didn't get further with that before 8.0.x, and adding #component which does not support preprocess solves the bc problem very nicely. Additionally we make no guarantees per the backport policy as to what any particular render array or form array contains, so if they start to include #components that's completely doable. So it feels like a great way to continue unfinished business with the render/theme systems in minor releases. The issues to remove individual preprocess hooks by moving the logic to the templates are also encouraging in that regard (once I get past the Drupal 6/7 knee-jerk aversion to logic in templates).
@dawehner's three questions in #7 and Fabian's in #6 included two of mine I was typing out before I refreshed the page - the use of other components within a component, tree traversal (this is where 'everything is a block' and 'HtmlFragment' ran into serious trouble), as well as how much this gets applied to form elements.
The other major concern/question I have that's not mentioned above (unless I missed it, in which case apologies for restating):
There were two main ways we tried to handle conversions away from theme functions and php template to Twig in the 8.0.x cycle:
1. one-to-one conversions to Twig
2. many-to-one conversions via adding new #types, or using existing #types with theme suggestions, #1804614: [meta] Consolidate theme functions and properly use theme suggestions in core is the meta.
I had a vague hope that if we took #2 to its logical conclusion, we'd end up with something resembling a component library.
This was going OK, if slowly, for a while, but then mostly ran aground at #1876712: [meta] Convert all tables in core to new #type 'table', where we eventually had to fall back to converting individual table theme functions to individual Twig templates in issues like #1938912: Convert language content setting table theme to a twig template. This was due to a mixture of complexity of the conversions and the inability of #type table to handle every case, and lack of time compared to the rest of the release.
#1930840: [meta] Remove or consolidate many similar admin page theme functions in core and #2076301: Remove views-mini-pager.html.twig, use a pager theme suggestion instead are good examples of in-progress issues not related to tables.
Looking down https://api.drupal.org/api/drupal/core%21modules%21system%21system.modul..., it would be sad to have a system_modules_uninstall #component - or at least if that's not sad, it'd be good to document why it's not. There are lots of other examples.
If we agree it would be sad, we need a plan for narrowing down the large number of #types and #theme hooks to a smaller number of #components. Both in terms of how to do the implementations (a conversion of #type table to #component table now would be missing functionality necessary to convert #theme tables to #component tables), and how in the backwards compatibility layer for #theme would work if the specific #theme hook does not map 1-1 to a component.
Bringing this back to the issue summary:
- it mentions extending components, is that instead of, or in addition to, theme suggestions? (note the change to views_theme() in the pager issue linked above).
- Aspect 3 explicitly talks about a bc layer for 1-1 #type/#theme -> component conversions, this doesn't account for where there should never have been a #type or #theme in the first place and instead we re-use a different #component. I can't think of a way to resolve that except for a mapping somewhere which gets referred to if a #component can't be found.
Comment #10
catchForgot to actually include one of the main points of the last comment:
If we want to ensure that this eventually reduces the complexity of the theme and render systems rather than just adding yet another #thingy, then we should tackle tables earlier rather than later.
Comment #11
larowlanGreat ideas here - one other place these could be used is to auto-generate ckeditor widgets, so elements in the styleguide could be used by content-editors too.
Comment #12
jhodgdonandypost pointed me here... I am working on #2697791: Add Plugin system for abstracted HTML-formatted text, which is in a way related. On #2697791-13: Add Plugin system for abstracted HTML-formatted text I just did a quick analysis of that issue vs. this one and would love it if someone could look that over and see if it is a fair assessment? I don't think they're replacements of each other, but if my issue should be a duplicate of this one (in other words, if these new components can be used for my purpose over there somehow), I'd love to understand how that could be, and/or help convert/create components so that I could use them for the help topics thing I'm actually working on.
Also looking for reviews of that patch. ;)
Comment #13
davidhernandezOne thing that concerns me is trying to move all of the logic into the template. I think for 90%+ cases we can do that, but it may not be a good idea to completely remove the pairing of functional code that we have now with preprocess. It isn't just cringing at the idea of logic in templates. We tried this already with some of the more complex use cases and failed. As catch mentioned tables are one, and views are another (especially if we continue to support altering markup and adding classes in views ui.) I wouldn't be surprised if forms find a way to throw a monkey wrench in too.
If the component's body parts all live together in one place, would it be possible to also provide an optional php file with the component that would contain its preprocess function? Or maybe it isn't even preprocessing that is needed since the component will be fairly strict about its data, but instead the component might need a helper function to do heavy lifting.
Alternatively, maybe the use cases are the problem (with the troubles we've had being a symptom of a bigger problem) and we need to rethink how we do them completely.
I'd like to add a property to specify where the component comes from. We rely entirely on inheritance now, which works well most of the time, but causes some problems. Being able to specify not just the type of component but its supplier might help with some of the admin theme/frontend theme issues we have. And also the whole Classy/Stable split.
Related to that, versioning would also be nice. The proposed workflow gets us to 9, but I don't think solves the core problem of templates/components being frozen. If we can version them, and create component libraries that have their own configuration, we can version them too. That solves the BC problem.
Comment #14
fagoIn shot, big +1. This is is overally really needed and critical and imho the right path for allowing client-side rendering.
A few comments:
- Imo, this really needs to be developed decoupled. First off, the components need to spec-first, with a reference implementation in php. The PHP implementation needs to be pure PHP (i.e. a composer package) and the pattern library / styleguide generation needs to work without Drupal. That enables a front-end team to work/start working without Drupal easily. Something like https://sculpin.io/ can be a good start here.
- The component tree needs to be clearly specified as well, such that a re-implementation is easily possible. Glue/processing code is a bit problematic here, and probably one of the hard-parts that needs to be figured out.
- The spec should include have a deterministic, complete mapping to TypedData, such that components can be easily exposed as blocks / panels / ckeditor templates. (I'm working on TypedData based widgets for Rules what's something that should be useful her as well).
Imo, this would solve all problems you ran into for establishing a proper styleguide driven development workflow, like integrating with Drupals JS libraries etc. Decoupling that component code, means really decoupling Drupals front end from Drupal and is what's required for allowing a decoupled front-end development team to work *while* being able to directly use the result with Drupal. That would be a real game-changer.
Once a working spec and PHP component and Drupal integration is in place, I totally see companies pick that up quickly and let them help developing a complete coverage for all templates, such that a new base-theme completely built out of those components should be achievable in the rather near time future!
Comment #15
fagoHad a good discussion with dawehner about this. Just dropping some quick notes here before I forget details... :
- Implementation would have to be decoupled first, come with integration in Drupal and make re-using decoupled components possible. Then existing things can be converted step by step until everything is components...
- Extra layers of Drupalizm like entities and views can be added once a front-end impl is converted to Drupal. Those tools need to define / take over data mappings then similar as configuring a block in ctools or an action in Rules.
- Contextual links etc. could potentially be done by wrapping components / higher order components
- Problem of mapping data and passing data through is exactly the same problem as solved by Rules with contextual plugins and nested calls
Looking quickly at the patch, I think this mostly misses a decoupled implementation. Also, libraries need to be part of the component. Then we can allow extensions to provide components, but components need to work without Drupal also such that a frontend team can pick up / write things initially easily.
Comment #16
aleksipAmazing work!
I have experimented with getting unmodified Drupal 8 Twig theme templates to work in the Twig edition of Pattern Lab. The end result is a decoupled implementation of a theme that can be worked on without Drupal. One of the challenges was how to handle Drupal's Twig extensions and the widely used
Attribute
objects.I noticed that the example image component uses the
file_url
function and anAttribute
object. With the proposed components being independent/decoupled, how are these handled?Comment #17
andypostFor entities there's issue as well
Comment #18
RainbowArrayOverall, I'm over the moon about this proposal.
This is very much along the lines of a big discussion that happened at MidCamp on moving towards a component-based theming system, which I wrote up in this post: http://www.marcdrummond.com/posts/2016/03/25/drupal-front-end-future
The key goal we discussed in that conversation was how to separate Drupal's data modeling from the appearance of a Drupal site.
Right now so much of Drupal theming is about decorating data models with markup. A component-based theming system would in theory allow us to move away from doing that.
However, the danger I see in the proposal above is simply translating all existing theme hooks and render element types into equivalent components. If we do that, we will have benefits, but we'll still have lots of components tightly related to data models.
I understand the desire to come up with a practical transition. I'm just leery of continuing to duplicate the same sub-optimal patterns. We already did this once when we moved theme functions one-to-one into equivalent templates. Do we really want to do that yet again with components?
------
Much of my thinking about components jives with Brad Frost's Atomic Design Constructs, which he breaks down as follows:
For what it's worth, I think Organism is the fuzziest level in this breakdown. I've sometimes created another level I've called Super Organisms that can combine multiple organisms. But sometimes you need to combine multiple Super Organisms before you get to the Template level, so this can get pretty messy.
While all components are "a thing that can contain other things," I think it's useful to think about there being different types of components.
In terms of Drupal, I think it could be useful to think about components in a similar way:
I could also see a label for something in between an Item and a Section like a Combo Component to help address the squishiness of the Organism level in Atomic Design.
Technically, each of these types of components could probably work in the same way. These labels are just a way to help think about the different ways components could be used. Labels like this might be helpful in user interfaces or a pattern library to help site administrators understand different types of components.
----
If you look at the templates folder for the Stable theme, you can find a breakdown of template types into the following groups:
That starts to help out sort how you might start mapping templates into components.
A lot of templates in field probably line up with Data components. block and content probably line up more with Item components. Views and dataset are kind of like Item List components. Layout templates probably would line up with Page components.
I do want to take some time to go through all our templates and try to do a breakdown of where they might fit into different types of components. I do want to point out though that some don't fit all that well.
A table template is not really a component, it's more of a markup pattern. Same thing with item-list. Maybe a Twig macro would be a better way to handle those sorts of situations?
----
I'm going to be writing up another post about one of our other discussions at MidCamp about different personas that the theme system needs to serve. For now, it's important to note that our theme system needs to serve two key uses: the Assembled Web and the front-end developer who cares a ton about Drupal's markup and uses that to implement a specific design.
A component-based theme system should likely have huge wins for front-end developers. Being able to define a template for an individual component that specifies, hey, this is the data I need for this template, and then keep all the markup for that component in one template would be amazing.
For that to really work, though, Data components would need to have some level of drillability. I don't want it so drillable that I have to reconstruct an img element every time I place an image in an Item component. However, I also want to have the option of skipping a generic field wrapper template, so if there is a data list, I can just put a ul and li for each data item within my Item template, then put the classes I want on that data list structure, or even on the individual piece of data. I'd rather not have to do that in several different templates in order to get the markup I want for a particular component.
However, this still needs to work for Site Assemblers who are using the UI to make appearance changes. So keep in mind that an Item component could include a data slot with one particular data item... or it might need to include one or more regions where several Data components could be dropped in and be ordered through the UI.
Ultimately, I think this system would be a huge win for Site Assemblers. This could be a much more outside-in model. Go to a particular type of page and then find some pre-configured components that can be dropped into various page slots. Add a section, and then add sub-components to that. Configure individual components as needed.
For Site Assemblers, it might be very useful to have some generic Layout options that can be chosen for components at different levels. One-column, Two-column, three-column, etc. Front-end developers will be much more likely to want to define very particular layouts for individual components. In essence a Layout is a generic way to say "hey, here are a certain set of slots you can put stuff into for a particular component."
----
One last thing to note. The definition of a component should include the context that it requires, essentially the data source and any data relationships between multiple data sources. An Item component may have several Data components specified, but it needs to get that data from somewhere. If that Item component is placed in an Item Series or a Section, then that larger component needs to pass that context/data source into the Item component. That context would then bubble up to the Section, so if that section is placed in a Page, the context/data source needs to be placed into the Section so that data can flow all the way down to the Data component.
I also think there should be some logic to translate that context/data into the individual pieces of data sent into a particular slot, whether that's called preprocess or whatever. I think it's great for templates to use logic to determine what class to use in markup, or even construct a class name from some source data. But processing a data source into data that should show up in that template? I don't think that sort of business logic belongs in a template.
Finally, that context/data source for each component is essentially how you can make this cacheable. It's the data sources that would define the cache contexts that this component should have.
----
Modules might end up being a common place to define components, themes might also define their own components. This probably goes without saying, but a theme should be able to override the markup/css/js in module-defined components.
----
Finding a solid way to transition this over the D8 cycle probably won't be easy, but I think we could get some huge benefits. A built-in pattern library? Totally possible.
This might end up having effects on other parts of Drupal as well. I'd love to see an Item Component Builder tool that would probably draw from elements of Views and View Modes. I'd also like to see an Item List Builder tool that would probably draw from Views and Entity/Node Queues.
Glad to see all the interest so far! This has me super excited about the future of Drupal's front-end.
Comment #19
Fabianx CreditAttribution: Fabianx as a volunteer commentedAlso re #18: Jacine wrote up a theme component library structure once here:
http://jacine.github.io/drupal/
Comment #20
Crell CreditAttribution: Crell at Palantir.net for Acquia commentedColor me big +1 on this proposal. I've a few things to add, and a few things to echo and reiterate that others above have already said:
* Absolutely the right way to do this is in a separate stand-alone PHP library, independent of Drupal, and then integrate it back in. The fewer dependencies the better. (YAML and Twig are the only ones I see here.) Not just from a code-sharing perspective, but from a decoupling perspective. If we physically can't let higher-level Drupal tools leak in and pollute the implementation, they're less likely to do so inadvertently. That also helps keep the Twig templates "vanilla", and thus safe to ship client-side using Twig.js without extra work.
* I've added a link to the IS for a presentation I give on Design Systems in Drupal, which shows that Atomic Design and Drupal site building are already reasonably well-aligned, if you do your site building right (Content Types, View Modes, Views, Panels). Having the render system use the same mental model, as Daniel notes, can only be a good thing.
* Big +1 with Fabian that we really really ought to be using classes here. Note that does not mean that themers building new components need to define a PHP class; I suspect we can do that with a compile step or similar (since Twig already has a compile step). (Although maybe they should have the ability to do so in very edge cases, and that replaces the preprocess function?) In a sense, what we're talking about here is ViewModel objects. That is, strictly defined typed objects that represent a value object for the display layer, as opposed to the storage layer. Drupal currently doesn't really have that distinction and passes entities straight forward to the display layer, which is not cutting it. If we're going to formalize the concept of "component", we should double down on that formalization and define them as typed classes. That not only makes debugging much easier, it gives developers more flexibility to leverage the API directly if they need to. We already have the beginning of support for "renderable objects" in the render API; let's get the rest of the way down that path.
Go's templating system uses that same model. Go has a native template system that is very Twig-esque, but you start by passing it a single struct. That struct can have sub-values in it, but technically what you're rendering with the template is that struct specifically. That struct is a ViewModel, in essence. Sometimes it may be 99% the same as a data object, sometimes not, but it's conceptually distinct. That's a good thing. That specificity also gives us more information to push forward to Javascript, again making it easier for a component to work client side or server side with minimal to no alteration. We want that.
Another benefit of using classes is that objects are, frequently, cheaper than arrays in PHP. Pre-defined object properties take less memory than array keys do, and because objects pass by handle we don't need to pass big by-ref structures around, which is actually quite inefficient. There's a lot of places we could get efficiencies here, especially with newer PHP versions.
* Because cosmicdreams isn't here yet, I'll say it: Web Components. :-) Essentially we're defining a definition mechanism for a web component package, which we just happen to pre-render ourselves (client or server side). That makes leaving it as a Web Component later once that suite of specs is sufficiently reliable much easier. In fact, I would argue we should develop this stand-alone library specifically with Web Components in mind as the primary rendering environment, and "render up into full page on the server" as the alternate processing tool. That will put is in a very good place to go gang-busters on Web Components once the market is ready for people to go gang-busters on Web Components. (And if it's not, we've still got a good, separated conceptual model that we've built from that gives us flexibility.)
Thank you Wim for tying all these threads together!
Comment #21
Fabianx CreditAttribution: Fabianx as a volunteer commentedExcellent comment by #20: I fully agree.
Can you elaborate a little more how templating works in Go?
I also think if every first implementation has a JS counterpart (with our new JS testing not impossible), then we are much more easily fit for the future, too.
Comment #22
Crell CreditAttribution: Crell at Palantir.net for Acquia commentedGo's standard library includes a Twig-esque (although not quite the same as Twig) template language. It's official docs are here:
https://golang.org/pkg/text/template/
And the version for use with HTML, that does smart escaping, is here:
https://golang.org/pkg/html/template/
A somewhat more approachable introduction is here:
https://gohugo.io/templates/go-templates/
(Hugo is a Go static site generator, akin to Sculpin or Jekyll. I've not used it; their docs just came up first in a google search.)
To be clear, I'm not suggesting that we should model our system on Go necessarily. There's probably stuff to learn from it, just like any other system, but the main point is simply the idea of every template having a struct (value object) that it binds to; that struct is explicitly defined, and thus you get all of your explicit value type safety. It's another data point and prior art in favor of explicit classed objects (ViewModels) for each component, which can be passed to the template. (That way, themers can override a template but still know exactly the stuff they'll get from the object, PHP devs have a self-documenting data structure rather than an undocumentable anonymous array, etc.)
Comment #23
Fabianx CreditAttribution: Fabianx as a volunteer commented#22: The most interesting point is however, how do you drill down into the struct (value object).
e.g. How do you wrap one component e.g. an image, within another component.
If that is again a value object within a simple list (collection), then we should be good.
But that is something that I would like to explore first some more. Can you elaborate how go deals with structs within structs and lists?
Comment #24
aleksip- A stand-alone implementation independent of Drupal, developed with Web Components in mind. Sounds really good to me! Also sounds like a project independent of Drupal, in a true 'off the island' way. A project that should be brought to the attention of the larger PHP community from the beginning?
- I'm still wondering about the fate of the current Twig extensions. I can't see how they can be used when we get to the 'everything is a decoupled component' stage?
Comment #25
legolasboI've been thinking about ways to improve the rendering system for quite some time these past few month, but I've never had time to actually write my thought down. Tonight I did manage to find some time to draw some UML in order to possibly answer #23. The UML diagram is still missing a lot of details, but I think it should be sufficient to communicate the basic idea.
I propose we create a primary interface (
Component
) from which all components derive. Besides that we should introduce another interface (ComponentCollection
) from which fuzzy composed components such as tables derive. As shown in the UML diagram the component interface has two methods,render()
andgetProperties()
. The render method does just that, render the component.getProperties()
however returns an array of all relevant properties for the given component. for example, when called on aLabeledContainer
it would return ['label', 'components'].Drilling down and filtereing in the theme layer.
Assuming render() uses twig to render output a themer could do something like
{{ SomeComponentName|properties }}
to get a list of all available properties available to drill into. Assuming that SomeComponentName contains a property which is in fact another component, (s)he can then use{{ SomeComponentName.SomeSubComponent }}
to just render the subcomponent or{{ SomeComponentName.SomeSubComponent|properties }}
to discover the properties of the subcomponent.We could do even more cool stuff like
{{ SomeCompositedComponent|only_type('SomeComponentType') }}
or{{ SomeCompositedComponent|without_type('SomeComponentType') }}
These filters will be trivial to implement because all components are based on the same interface.
Decoupling
Note how this diagram does not show any implementation details about Drupal, Twig or any other way of rendering the components. This could be the base of a totally platform agnostic package. Completely independent from Drupal or rendering frameworks. Especially if we also introduce a Renderer interface and require an implementation of that interface during the construction of a component. By doing so, we decouple what gets rendered, from how it gets rendered. That way we can create a Twig renderer, but also a JSON renderer, or any renderer for any output we want to generate. These renderers can even be implemented in completely separate packages.
Disclaimer ;)
Once again. This is a really incomplete representation of the way I've been thinking about implementing something like this. But I am curious to find out how you think about the general concept.
edit:
The UML has certain parts greyed out. that is done on purpose to emphasise the abstract classes and interfaces but still be able to demonstrate some concrete implementations.
Comment #26
moshe weitzman CreditAttribution: moshe weitzman at Acquia commentedThis plan and PoC looks solid to me as well.
Comment #27
catch2007 was before we had view modes, display entities, formatters, hook_entity_view/field_attach_view_alter(), all page elements as blocks, and many other things which are both more powerful than preprocess, and sometimes undermined by it.
There are places where we've failed to apply those patterns yet, such as 'submitted by' and title display on nodes, we're going to need to replace those things properly when moving to components - so that we don't end up with a node display component that's only different to other content entities by one or two template variables. But yes generally I agree that there's a lot of work to be done to actually remove usage of preprocess.
template_preprocess_node() and template_preprocess_comment() still the best/worst examples:
https://api.drupal.org/api/drupal/core%21modules%21node%21node.module/fu...
https://api.drupal.org/api/drupal/core%21modules%21comment%21comment.mod...
Comment #28
Fabianx CreditAttribution: Fabianx as a volunteer commentedOne thing that should be deprecated ASAP is theme_wrappers as you can always use an upper level instead and its super confusing to have #theme and #theme_wrappers at the same level with the theme wrapper having not available most variables needed for context.
Comment #29
catchOpened #2714509: Remove usages of #theme_wrappers.
Comment #30
JohnAlbinThanks for creating this issue, Wim! I would have been posted sooner, but I got sick with flu the same day you posted. :-p And I still haven't caught up with my work enough to read this whole thread. I will have to come back to it during Drupalcon.
In the meantime, here's my current thinking on adding components to Drupal:
This is too big a task to try to plan out and tackle in core. This part of the theme system touches too many parts of core, every single module, etc. The last time we tried to simplify the theme system without doing some contrib-side experimentation and themer by-in, we ended up with the Render API. I'd like to have some intense experimentation to get concepts in front of real-world front-end devs.
For the past 3+ years, I've been experimenting with different techniques for making components of HTML/CSS/JS and how to integrate them with Drupal.
While working with Drupal 7, it became clear that the "hard part" was going be figuring out:
Now that I've started studied Twig's documentation and have started porting Zen to Drupal 8, I've discovered that I can use Twig to do the routing of variables.
I currently have a components library built in Twig that includes no Drupal-specific extensions. (Side note, because there are no Drupal-specific extensions or variables, I'm able to build a style guide from these components by pushing the *.twig files through Twig.js in a Node.js build system.)
Zen uses Drupal's *.html.twig files solely as a means of getting at a theme hook's variables.
For example, classy's html.html.twig file has a skip-link styling on an
<a>
element. Zen has a skip-link component, so its html.html.twig file uses{% include "@STARTERKIT/navigation/skip-link/skip-link.twig" with {modifier_class: 'visually-hidden visually-hidden--focusable'} %}
; modifier_class is a Twig variable that I defined in all my components to hold component variant classes.Another example: the comment.html.twig template has a {{ new_indicator_timestamp }} variable. I am NEVER going to put that variable in a component because it will tie the data variable name to the HTML display and make it impossible to re-use with other data that does not have that variable. Instead I will be able to create Twig blocks in my component library Twig files and then do this in Zen's comment.html.twig:
No yucky Drupal variables in my component library. Just perfectly re-usable Twig templates. Actually, I may not have a
<mark>
in my comment.tpl.php, I could replace that line with something like:{% include "mark.twig" with { attributes: ' data-comment-timestamp="{{ new_indicator_timestamp }}"' } %}
.In Zen, all of the *.html.twig files reside in /templates and Zen's component library (with CSS, JS and *.twig files) resides in /components.
The more I work on this, the more natural this feels for a front-end developer. While the "two separate groups of Twig files" concept feels weird and some part of me thinks this is "needless duplication" or like I'm "mis-using the Twig system", I still really like working this way. Drupal 8 front-end devs already know Twig and they don't have to write any PHP to use this dual-twig method.
Anyway, that's my input for now. I look forward to some intense discussions at Drupalcon!
Comment #31
effulgentsia CreditAttribution: effulgentsia at Acquia commentedYay! Huge +1 to this! I think this is a very important distinction to make. That HEAD's current theme hooks / templates actually consist of 2 very different things: things that can be turned into components (e.g., image.html.twig) and things that don't make sense as components because their variables are coupled to Drupal data (e.g., node.html.twig). Having different terms for these different things (e.g., components and templates) is a helpful starting point.
Keeping non-component templates in Twig for now seems a good starting point. Ideally, however, we'd get to a point where these templates would contain 0 markup, and solely route to components for all of their markup. This would mean that a theme could control every single piece of markup solely through overriding components, and not be required to override any non-component templates. This is huge, because most contrib modules currently define their own theme hooks for the presentation of their own module-specific data concepts, which results in it being impossible for generic themes to provide theme-specific implementations of every contrib module's templates. But, if in a componentized architecture, the contrib module templates are just routing layers to core-defined components, then a generic theme could end up controlling all markup, even as various modules get installed on a site, and this was a key piece of Jacine's original vision for componentizing (see the "Your Output is not Special" section). Once we achieve such a separation, it might become natural to ask whether Twig is the best language for expressing markup-free variable routing to components, or whether something purely declarative, like YAML, would be a better fit. Hard to answer that now though, since we don't know what logic will be needed in these templates once they're all properly separated.
For components (as opposed to the non-component templates), I really like this, because getting this code out of PHP allows for client-side re-rendering via Twig.js! One thing to consider though is that each module can also implement preprocessing. But I think we can come up with a pattern by which modules could still do this via Twig files, and then we have some mechanism to aggregate all of these Twig files (that preprocess variables for a given Twig component) into the compiled Twig component.
I agree, but I think this concern applies solely to the non-component templates, since those are the only ones that deal with Drupal-bound variables. And unfortunately I don't currently see a way out of keeping this code in PHP, but perhaps it makes sense to move it from a hook_preprocess() to a
hook_view_model_alter()
. This unfortunately presents a barrier to client-side re-rendering of such a template, so it would be nice to reduce such alter implementations to their bare minimums, and do as much as is sensible in Twig-implemented preprocessing instead.Comment #32
effulgentsia CreditAttribution: effulgentsia at Acquia commentedI also want to tie this in with #8. Perhaps these variable routers could end up being Layout/Panels configurations. But that's pure speculation at this point: seems to me like there's a bunch of steps that need to happen here before broaching that, though experimenting with such speculation could yield some great insights.
Comment #33
catchThe node-specific information in node.html.twig is removable though. The reason it's still there is because we haven't completely ported node rendering from Drupal 4-6 hard-coded variables to display modes etc., but if we finish that then it'll be a three-line template or so.
Comment #34
cosmicdreams CreditAttribution: cosmicdreams commentedThanks @crell for the shoutout in #20.
Yes, I've been listening and reading these comments and am growing more and more excited about the prospect that this mechanism could mean great things for Web Components. To understand why, first a definition.
Web components are exactly the kind of components we're talking about here, just (eventually) supported natively by the browser. Specifically, "Web Components" as defined by http://webcomponents.org/ are a set of standards that define the browsers ability to create, register, and package templates and an API for interacting with them. It seems pretty clear to me, given the momentum of this effort that one day the browser could be our themeing layer. If that sounds fantastical it's because it is.
If that future was imminent, this effort to component-ize Drupal would be vital. We would need to be able to define the structure, appearance, behaviors of components. We would need to build a set of components that have well define responsibilities. We would need to understand how to use these components together so that we can build more complex components / features / pages / etc.
My current thinking about Web Component's interaction with a system like a CMS is that they benefit from two different kinds of data deliveries.
1. Initial data that establishes state / content
2. Responses to interactive behaviors.
But the main thing I want to say is that I'm excited that we're talking about this effort.
Comment #35
RainbowArrayOne of the things I've been doing the last few weeks is doing some refresher looks at Ember, Angular and React. React in particular exemplifies a component-based model, but it seems like there's a general movement away from MVC and towards components.
With React, a component can take in props (static data) or state (data that might change). I think that's exactly the sort of thing you're talking about, cosmicdreams.
One of the things I'm having a hard time getting my head around is how to keep a next-gen theming system DRY.
In particular, if we want to have more robust JS interactions in core, there are a lot of potential areas of duplication.
Let's just say for the sake of argument we're able to make use of Twig.js to reuse templates on the server-side as well as the client-side. That would be one element of keeping things DRY, but that alone is not all that we need.
Somehow we're sending data into component templates. In our current render system that involves setting up render arrays and preprocess hooks to allow for alterations to the data that ultimately gets sent into templates. Even if we come up with a new PHP-based component render system that we open source and make available to other PHP projects (as well as ourselves), how do we duplicate that logic on the client side?
The render system is in the business of taking structured Drupal data and turning it into markup. There's a lot of logic that goes on to make that happen.
In theory you could put a whole ton of logic into a template, and depend on that for transforming raw Drupal data whether it comes from a REST endpoint or internally through Drupal. I'm not sure I'm keen on that: putting some display logic in templates to make markup decisions seems fine by me. But data transformation logic? No, not a fan of that in a template.
So if you don't want that, and you want to be able to share templates between the server side and client side, so that you can do good things like virtual DOM diffing for quick page updates, that is almost assuredly going to mean duplication of logic in both PHP and JS.
This is a little bit off topic for this thread, but it's just something I've been mulling over, and it seems somewhat connected. I don't really have good answers on how to solve this right now.
Comment #36
Wim LeersThanks for all the insightful comments! There’s lots of great ideas, thinking, critiques and so on in there. I deliberately avoided commenting here for weeks (although I read every single comment shortly after posting), to avoid steering the discussions in a certain direction.
I’m happy to see that everybody is at least “+1 for concept”. That makes this issue/patch a great starting point :)
To make sense of the ~9000 words, ~50000 characters or ~45 minutes of reading that you all have posted since #2, I:
If you want to, you can ignore all that and just look at the 24 conclusions:
Attribute
[2]Top priorities to discuss, based on both number of times surfaced/contentiousness and how much it blocks progress/impacts direction:
Let's discuss at least those top priorities at DrupalCon New Orleans, hopefully more, but especially those.
Comment #6 (@FabianX):
Objects are a problem too: they can encapsulate logic that we cannot automatically map to JS. So they interfere with the goal of being able to re-render on the client side.
SO: What is acceptable is a generic object/class, what is not is a component-specific one.
Conclusions:
Comment #7 (@dawehner):
I really like your thinking around formatters + entity display + panels. But I can’t yet see a sane path to get there. Hopefully in some time, we will :)
Conclusions:
Comment #8 (@stevector):
Components as layout plugins? I re-read your blog post to ensure I fully grok your intent. Great post :) I remember it from Marc’s post, but it’s better to read the source: https://www.palantir.net/blog/explaining-panels-why-i-use-panels
dawehner already brought Panels up before you, and like I told him: I did not think about this.
I think postponing layout plugins (#2296423: Implement layout plugin type in core) is nice in principle, but rather pointless in practice: it will take a year at best for this to become fully usable. We shouldn’t put everything on hold for this: Display Suite/Panels/Layout plugins as they are currently designed still are a very valuable tool. Recorded as vote for 5.
include
syntax :)Conclusions:
Comment #9 (@catch):
Conclusions:
Comment #10 (@catch):
Conclusions:
Comment #11 (@larowlan):
image2
CKEditor Widget (which Drupal 8 extends) is extremely complex, because the<img>
tag can be both inline and block… and because the accompanying JS code must include logic matching the exact HTML. Nevertheless, added to conclusions.Conclusions:
Comment #12 (@jhodgdon):
Comment #13 (@davidhernandez):
Conclusions:
Comment #14 (@fago):
RE: mapping to TypedData: wouldn’t that violate the first thing you said? TypedData has the concepts of computed properties, settings, and more. How will we make that work in JS?
Don’t get me wrong, I see the attractive/interesting parts of it, but I think there’s also a pretty big risk in coupling it to TypedData.
On further thought, there’s an even bigger problem: TypedData does not require primitives, but in fact allows for arbitrarily complex nesting. See e.g.
\Drupal\Core\Entity\Plugin\DataType\EntityAdapter
. For now, added to conclusions as won’t-have.Conclusions:
Comment #15 (@fago):
Conclusions:
Comment #16 (@aleksip):
Url
and Drupalisms in Twig likeAttribute
: the number of Twig functions is extremely limited. We’d require those to be implemented both in PHP and JS. Any module providing a Twig extension that adds more Twig functions would have to also provide a JS implementation. RegardingAttribute
: no good answer to that for now. Added to conclusions.Conclusions:
Attribute
Comment #18 (@mdrummond):
RE: drillability: I’m not convinced that it is necessary, or even desirable. Drillability implies two things: 1) templates modified for the specific names of children in a particular given data structure, 2) complex data structures. Both of those are highly undesirable. I think part of the point here is that each template only receives simple data structures (i.e. respecting the limitations I proposed in the issue summary), because: A) otherwise too complex/frustrating themer experience, B) otherwise no ability to provide strict validation, and hence good error messages, C) impossible to re-render on client side.
Added to conclusion.
RE: definition of component should include required context: “ essentially the data source and any data relationships between multiple data sources” and “[…] it needs to get that data from somewhere.” I find these statements confusing, vague and contradicting to what you said earlier. My proposal is explicitly stating that each component defines its inputs, the inputs must be primitives/lists of primitives/other components, and that is it. We have a component tree and a matching variable/input tree. So, a component does not “need to get its data from somewhere”, it’s already getting it from that input tree. We don’t want components to have data-retrieval-logic: then we make them too complex again, much like today’s system.
I suspect you meant something different though, so let’s discuss that in IRC/IRL.
Perhaps you meant the code that is responsible for generating the component+input trees? Because that totally is something that is very important: without that being well-defined, it will be impossible for JS to generate or retrieve an updated input tree, which is necessary for client-side rerendering. Added to conclusion.
Conclusions:
Comment #20 (@Crell):
Conclusions:
Comment #23 (@Fabianx):
RE: drillability: see my answer to @mdrummond in #18. Recorded as vote for 18.
Won’t-have: drillability [2+1]
Comment #24 (@aleksip):
Conclusions:
Comment #25 (@legolasbo):
RE: UML/PHP class architecture proposal: I think that’s simply prematurely detailed, we will have to work towards something like that, but only through doing the work of converting complex cases will we know what the full set of requirements is, which I suspect this architecture won’t be able to fulfill.
However, thanks a lot for doing that — it’s great to be able to reference an already-made diagram for what is a possible implementation!
Conclusions:
Comment #26 (@moshe weitzman):
RE: style guide as learning and checklist tool: oh, wow, that makes so much sense! I think this is quite exciting. I don’t know why I didn’t think of that! Added to conclusions.
RE: totally agreed we need to deeply, 110% understand the consequences of removing preprocess. Indeed needs experimentation. Recorded as vote for 8.
Conclusions:
Comment #28 (@Fabianx):
Comment #30 (@JohnAlbin):
RE: “components library built in Twig that includes no Drupal-specific extensions” + skip-link example + comment example + “the more natural this feels for a front-end developer”: this sounds so very awesome! We should validate it in a deeper discussion, but I’m very excited by A) the approach, B) the fact that you have it working already.
This helps with:
component.html.twig
and map that to other components, but you made it far more acceptable — especially becausecomment.html.twig
and friends live in a separate directory:/templates
— one that even helps with BC! (9)Conclusions:
Attribute
[1+1]Comment #31 (@effulgentsia):
32 (@effulgentsia):
RE: “Perhaps these variable routers could end up being Layout/Panels configurations.” → another interesting insight! Panels & friends would generate such templates then. Recorded as vote for 5.
Conclusions:
Comment #34 (@cosmicdreams)
see my response to Crell regarding web components. It’s too early to talk about concrete things, but it’s definitely worth keeping in our heads. And I’m sure you’ll help with that :)
Comment #35 (@mdrummond):
Conclusions:
Comment #37
tkoleary CreditAttribution: tkoleary at Acquia commentedWhat Wim said
Comment #38
polynya CreditAttribution: polynya as a volunteer commentedWow, thanks for the great summary Wim. Can I add some +1s, especially for 1: component definition and 19: modules and themes can define components.
As an extension to 19, a module should be able to define the path to a folder containing components. The project I'm part of is building web pages from components defined in an external pattern library. We don't want to alter the Drupal repo just to update a component or to add a new one. As part of this, I built UI Components based on Wim's patch. We'll probably be using this until the features are in core.
I'm really disappointed I won't be at DrupalCon next week, but I hope to see you all in Dublin!
Comment #39
kalpaitch CreditAttribution: kalpaitch commentedI am working with Polynya above. In our particular use a component could define a twig template, css styles, some js behaviours or any combination of these. We have some components which do not have a twig template, obviously they wouldn't be directly renderable.
On requirement #4 Directory structure. When using an external pattern library we should not assume a directory structure, it would be nice to be able to alter these paths. This matters for things like asset paths.
On requirement #5: The YAML file specifies. We are currently integrating our own components from a pattern library and have added additional values that we feel we need:
* declaration of shared assets, images/sprites, fonts
* dependency on other components for twig, css, js or the above assets
* basic validation on variables (e.g. we feel it would be the components duty to set a character limit on say a title field)
* configuration variables which provide some options (e.g. add a class to a component dictating whether it should be a landscape or portrait variant of a component)
As you can see we are beginning to feel that the pattern library should be informing the application/Drupal about a limited set of conditions it has for the use of a component.
Thanks all for the big push on this topic.
Comment #40
heddnIt might be apt to mention https://www.drupal.org/project/components as a contrib module doing some work in this space.
Comment #41
Wim Leers#39:
The patch in #1 already includes an example of this: a theme may extend an existing component, and do nothing except add a CSS file.
You're already free to put assets anywhere in the patch in #1.
The first two bullets are already supported by the patch in #1. The third bullet is an interesting one that could indeed be valuable, we should investigate that. The last bullet would simply be another input. Or, if it's truly a variant (like
image
→image--with-rollover
, then there'd be a componentX
withX--portrait
andX--landscape
variants. Like the issue summary says: dealing with variants still is fairly undefined. That's definitely still going to happen.Comment #42
kalpaitch CreditAttribution: kalpaitch commentedNice work. I couldn't see it mentioned but namespacing of components could be useful. I can imagine an external library with no knowledge of the components in Drupal having clashes but not wanting to overwrite or extend said components.
Comment #43
Crell CreditAttribution: Crell at Platform.sh commentedI don't recall what I was thinking when I said we could auto-generate PHP classes, honestly. It's been a while. :-) Generally, though, my point is that we want as strong a type checking as possible, both for DX and for simplifying the code itself. We've tried anonymous structs, we know we hate it, so let's try typed/named structs this time. :-) (PHP lacks formal structs, so classes are the closest we've got.)
One thing to consider on the DRY/duplication question is that Rails takes the opposite approach: The latest Basecamp (co-developed with latest Rails) sends pretty much everything back to the server to be rerendered and then injected into the page. That avoids a ton of duplication. Of course, it also adds overhead, and that overhead is larger for PHP than Ruby because of the need to rebootstrap every time. That could be solved with websockets, but then we'd need a websocket PHP server. (See debates in various other issues.) In any event, the take-away I'd offer is that we don't necessarily have to have perfect 1:1 parallelism between PHP and JS. If there are some things that require going back to the server to rerender while JS can natively do much/most of it itself, I think that's fine. (For some definition of "much/most", of course.)
Comment #44
Wim Leers+100000000
Implementation details TBD, we'll figure those out.
Comment #45
Bojhan CreditAttribution: Bojhan as a volunteer commentedFascinating to see this thread. We've tried several times to build a solid component/style library for Seven but failed to do so as the technical limitations keep getting in the way. While this clearly focuses on a much larger problem, it's nice to see that it might lift some of those limitations. It might be a bit early to say that atomic categorization is the most sensible - but I think for Seven we can go with whatever default is provided and adjust accordingly.
Great work, and good to see the structured consensus building - a good example for others who wish to tackle difficult challenges.
Comment #46
RainbowArrayI took some time and reviewed Wim's mega-review as well as the original patch.
What has been developed so far seems to be geared mostly towards how to implement the lowest level of components, which I had described as data components. That's where simple variable types and strict typing would be most useful, and where the mapping of theme functions to components is likely to be most relevant.
However that is just one part of what is necessary for a component-based theming system.
There are lots of "votes" for no drillability for templates which, to be clear, tends to be somebody saying "we need drillability!" and then that getting counted as "we will not have drillability."
I think drillability means a couple things. One, an item component (let's say an event teaser) could have a list of dates and times when that event takes place. Each of those date/time items would be a data item, while the list itself would be the data component. Think of a data component as a field, and a data item as one piece of data in a multivalue field. So could there be a template for a date/time data component. Sure? But what I care about most is that event teaser component. I'd like that template to have something like (ignore whether this is correct Twig syntax, the concept is what's important.)
This would in theory allow you to still take advantage of whatever gets added in via whatever replaces preprocess, but still allow you to define the exact markup that will be used for that data component within the item component. This sort of drillability is essential because without this you will still see numerous templates in order to create the markup for one item component with multiple data components.
The other thing that is necessary is being able to nest components multiple levels deep. A page component contains multiple section templates each of which could contain combo components, component series (river of news) or individual item components, which would contain multiple data components. Component nesting is absolutely necessary.
What is also missing from this right now is where the actual context/data gets into templates, and particularly how that travels from high-level components down all the way to low-level data components. For example, let's say that in our event teaser item component has a data component that contains the event organizer name, wrapped in a link to that organizer's page on the site. The data component needs the name and a URL, but those values need to come from somewhere.
Let's the event teaser is nested in a list of events. That is is nested in a main content section for an events landing page, which is nested in an event landing page component. So where do we get that organizer name and URL? Do we pass up the need for those simple values all the way to the top of the chain? Ultimately that info is likely to live in a user object. I think it's probably useful for that event organizer data component to know that its caching will vary based on that particularly user object info, so that if that user info is updated (typo in name is fixed), then the caching for that event organizer in that teaser gets updated. Ultimately that caching info needs to bubble up through all those levels that contain that data component, all the way up to the event landing page component. So do we pass that user object all the way down to that data component, which takes care of transforming the user object into the values we need in the template? I don't know the right answer, but we need to figure that out.
Right now, we use preprocess to transform Drupal data into template variables. This patch sort of does that transformation in a Twig template, mixing that data logic in the same template as the markup and the logic to determine classes for that markup. I think we could improve this by having one Twig template that serves as the preprocess replacement, and another for the markup/classes logic. That would allow us to keep themer-friendly markup-based templates, while also having the replacement for preprocess logic in Twig, where it could be shared between client and server down the road.
Part of the benefit of preprocess is allowing multiple modules and themes in the theme tree to affect the variables for a particular template. I think we could replicate this by allowing modules/themes to register a preprocess Twig template that could add something to an Attributes object if it exists (or create one if not), or things like that. In the final component compilation, those preprocess templates could be included one after the other in a proper order in order to prep variables, while the final component template in the chain takes in those values. The patch documentation notes that variables can only be added by components: I'm not sure if that excludes changing the values of variables, but that seems a pretty valid use case.
-----
I also wanted to note that this work on components is really very closed to the work on layouts and blocks.
What I care most about for components is that one component represents one chunk of the design. This initial work focuses on custom component templates, which is great for front-end developer focused implementations. This will ultimately need to work for site builders as well. For those use cases, components will need regions where an arbitrary number of smaller components can be placed: essentially, semi-generic layouts for components.
So as the layouts and blocks initiatives proceed, we should make sure we look ahead to the future to make sure layouts and components work well together.
Comment #47
cyb_tachyon CreditAttribution: cyb_tachyon as a volunteer commentedJust noting that Progressively Decoupled Blocks (PDB) is also doing some work in this space for ingesting JS Framework-based components to Drupal Blocks (and maybe soon Twig namespace components) - we commented on the connection here: https://www.drupal.org/node/2707849
Sandbox: https://www.drupal.org/sandbox/mrjmd/2664138
Comment #48
Wim LeersFYI: Here is the link to the notes from the Theme Component Library discussion we had at DrupalCon: https://docs.google.com/document/d/1TJjvsF8NpRIW_YrP7hdb3ougnc7Kk7nZFP09... — they include the diagrams that were drawn.
Comment #49
aleksipI think that one thing closely related to this issue is how stand-alone components are developed, packaged and used. I just published a blog post containing some thoughts about stand-alone component projects.
It would be great if we could reach a consensus on how to do this and gain component interoperability not just between different Drupal themes, or between Drupal and decoupled client-side front-ends, but also between Drupal and tools like Pattern Lab.
Comment #50
cosmicdreams CreditAttribution: cosmicdreams commentedDo we have actionable steps to pursue now that we've hashed some of the details out at Drupalcon?
Comment #51
Wim LeersWe didn't hash out details at DrupalCon. The discussion was productive in that we've had even more requirements and goals stated by even more people than those who participated in this issue so far. But because of that, almost no actionable steps were identified (and in that sense, it was not productive at all). It is absolutely impossible to write up an https://www.drupal.org/core/initiative-proposal-template.
likeThe only actionable steps are basically:
i.e. bring the two together, because they mostly cover different ground.
That is also what is listed at the very bottom of that Google doc:
Comment #52
davidhernandezA good first step would be documenting a proposed API, and how a themer would actually use it for various use cases. We've talked some in person about use cases (user stories) but haven't written them down. All the wiz bang zoom here is meaningless if the TX turns to shit and we end up with a worse experience, undoing all the progress made with 8. At the end of the day, a themer/site builder, not a render system engineer, needs to work with this. I need to know how I would actually make things, how inheritance from core->module->base theme->base theme2->sub theme will work. How to override some things versus other things, etc.
Comment #53
Wim Leers+1000.
But, nothing is fully formed yet. It is impossible to fully design the API now, because we need it to work in a BC way, so no doubt that we'll need compromises. IMO the only way to achieve what you describe (and again, I agree with that) is by actually doing the work of converting parts or the entirety of Drupal core, to verify that it works in all cases, and to demonstrate/prove that the TX is in fact better.
So, for at least weeks (and probably longer), this issue will remain fairly vague.
I will be working on applying the few next steps that we did identify over the coming days.
Comment #54
Wim LeersI started working on this again a few days ago. Unfortunately, after applying the patch, I noticed several big problems… which somehow I didn't notice before due to last-minute changes, or because of changes in Drupal core itself.
interdiff-1.txt
: The pattern library (/admin/components/pattern_library
) was not rendering the components correctly. This was an easy fix.interdiff-2.txt
: The style guide (/admin/components/style_guide/bartik
etc.) was having a similar problem. The prior point/fix didn't fix it. Turns out there was a significant problem with template inheritance. i.e. #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance. That prevented even my explicit{% extends "@system/image.html.twig" %}
from working.To address that, I took on the task of actually fixing #2387069: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance. Thanks to an idea from @Fabianx combined with some research of my own, I've got a working solution for that, which uses the theme registry at Twig template compilation time to transform
extends "FOO.html.twig"
toextends ['path/to/parent/theme/FOO.html.twig', 'path/to/grandparent/theme/FOO.html.twig']
andinclude "FOO.html.twig"
toinclude ['path/to/current/theme/FOO.html.twig', 'path/to/parent/theme/FOO.html.twig']
. In other words: this transforms Twig templates at compilation time to respect our theme registry, but still uses native Twig functionality, therefore simplifies Twig debugging, but doesn't put the burden on the themer to specify every parent theme template.See #2387069-26: {% extends "foo.html.twig" %} in Twig templates does not respect theme inheritance.
Apply individual commits to recreate all of the work (my entire local branch) by applying the "individual commits" patch using
git am PATCH
! (Created usinggit format-patch origin/8.2.x..HEAD --stdout > PATCH
.)Comment #55
Wim LeersMy next steps:
Doing this first because #54 has already forced me to dive deep into Twig land, and this touches on similar areas.
Doing this second even though it's really the most important thing, because I think #54 + the first thing listed here will result in me working on this with more complete Twig knowledge in my head.
Comment #56
Wim LeersDone:
In doing so, I worked quite a bit with found a bug/limitation in Twig, for which I filed an upstream PR: https://github.com/twigphp/Twig/pull/2069.
{% include …
and{% embed …
some components. Not sure yet if this is best or not, but it means less change for sure. It seems to work out fine for John Albin, Micah Goodbolt, Phase 2 and others.Next:
controller -> render array -> render the render array
, and the rendering of the render array looks like this:render array (and its callbacks) -> Twig -> render array (and its callbacks) -> Twig -> REPEAT
. It's those (#pre_render
) callbacks that look at the configuration the site builder set up in Field UI and convert those into render array declarations. And then it's the rendering of those render array declarations that ends up calling into the appropriate Twig templates. This happens both at the "field in entity" level (field order + formatters) and the "item in field" level (field formatter settings). In other words: it's not Twig templates all the way down, and we need it to be if we're going to make this work. So we'll need Field UI's configuration changes to cause a Twig template to be generated to match that configuration. A very important consequence of this will be that themers can copy/paste this generated Twig template and use it as a starting point. So, this has the potential to also address the long-standing frustration themers have that site builders can mess up their work — if a themer provides a customized entity template, then we would effectively disable the field UI for that entity type, to signal that this is not possible. (Yes, lots of details to be figured out, but hopefully that sounds exciting at a high level!)#component
stuff in render arrays, for example. If we go forward with the plan of "presenters" as John explained on this issue and for example Micah Goodbolt has observed many people discover independently (which is always a good sign), then that means the V1 vision I explained in the issue summary no longer holds: roughly speaking, it would not be render arrays that know about components, it would only be Twig "presenter" templates.I'm starting to think that the best way to make this first leap forward is to help refine Zen + the
components
module, and then introduce acomponentized_bartik
theme in 8.3 (or a completely different theme), which would then serve as an example. Because it already is a steady improvement for the TX. That would not be a solution for , but it would reduce the surface area of the render & theme system, and that's why I think that could be a great first step:This would be the first time that Drupal's front-end experience is actually improving rapidly. And it'd be completely opt-in.
The most important caveat would be that we need to be careful we don't paint ourselves in a corner that wouldn't allow the bigger goals from being achieved.
Comment #57
Fabianx CreditAttribution: Fabianx as a volunteer commented+1 to #57. I do think this would be great to add components in this step-by-step fashion.
Comment #58
Wim LeersJust discussed #56 in the weekly component-based rendering meeting that happened today.
There's resounding support for what I proposed in #56:
markconroy and lauriii expressed strong interest in actually helping make this happen. (Nobody opposed — +1s from: markconroy, lauriii, davidhernandez, cyb.tachyon, prestonso.)
And as it happens, lauriii was already working on an anyway! lauriii was happy to slightly modify that initiative proposal to ensure that it doesn't just demonstrate best practices at large, but also specifically those for using components. initiative proposal
mdrummond then mentioned that mherchel was also very interested in helping out with a new theme.
Conclusion: it looks like we'll get a
initiative for 8.3, and as part of that, we'll provide guidance on how to do component-based themeing in D8.polonya todl me he's interested in helping me out with . He's already created a "UI components" sandbox module based on the patch in this issue. So it only makes sense to collaborate with him on bringing some of the things in this patch to Zen, so people can already start to benefit from some of the work we've done here.
Comment #59
JacineRe: #2702061-56: Unify & simplify render & theme system: component-based rendering (enables pattern library, style guides, interface previews, client-side re-rendering), Next.2
Long-standing frustration is pretty much an understatement, but yes! ;)
In my failed quest to respond to this thread earlier, this was one of the issues I started to write up. I also started to take some rough notes, on things I think can improve the situation, and started on a comp containing a limited version of the manage display screen. I see this issue as a big a barrier to ending up with a good result here. It's also one of those issues that causes conflict/divide within the front-end Drupal community. We have:
Bottom line: The situation is not ideal, or even really acceptable in any of the above cases. By trying to be all things to all people, we have created this mess. It would be beneficial to all involved to stop ignoring the fact that there is a huge disconnect between UI Config (Manage Display screens) and templates, and that they invalidate each other causing a terrible user experience on all sides. It appears we are finally willing to admit that it is hindering our plans to improve Drupal in general. Yay, that's the first step toward recovery. LOL.
My rough comp is attached (PDF), and my notes are in this gist, which I've pasted below:
Comment #60
Fabianx CreditAttribution: Fabianx as a volunteer commentedJacines post (which is great), remembered me that I worked on an MVP for a two-way communication with the theme for layouts in core at DrupalCon and discussed that with several people:
https://docs.google.com/document/d/1JGmMwColdzXrJUqjKX1jkwbwHTd9eHKd2GS_...
What the document does not reflect is that there are several things mixed together (theme suggestions as layout finder, layout plugins (which already exist), etc.).
I am leaving it here anyway. I do think even Jacines part could be combined with my approach:
Technically the easiest implementation is to create a {ui_configuration} twig tag, e.g.
Instead of:
it would be:
The advantage of an explicit element is that two way communication gets easier as this would be compiled to:
So it is as simple as loading the template and calling getConfiguration() on it to communicate with the UI.
Note: I am not debating whether or not the approach to set the variables in this way or those variable is a good idea or not, I am showing how a technical implementation could look.
Edit, this would look for the UI code something like this in pseudo-code:
Comment #61
cosmicdreams CreditAttribution: cosmicdreams commentedIn this possible future where the template has told Drupal that it will be control over the display of fields for an entity type:
1. Does this control need to be as granular as having control on a display mode level?
2. We should probably provide a message in the UI informing the administrator of the specific file that has control over the display.
3. Provide logic to conditionally control ordering but not formatting and vice versa
Comment #62
cosmicdreams CreditAttribution: cosmicdreams commentedAlso, would this diminish the need for modules like fences or other modules that attempt to "clean" the markup of fields?
Comment #63
jonathanshawThere's also the den of controversy that is #2289619: Add a new framework base theme to Drupal core. Should this be born in mind as a future possibility to allow for?
Comment #64
Andre-B+1 on that. similar to how views states overridden template files.
+1 there need to be ways to not abandon the field ui completely / make it even harder for devs / site builders to figure out where and how output was rendered. Also those template overrides should have a way and best practice to still be extendable (adding new fields). They probably should also respond to deleting fields without breaking the twig layout. This will require thorough examples and best practices to keep as much flexibility as possible.
Comment #65
tkoleary CreditAttribution: tkoleary at Acquia commentedSomething that I think is getting lost amidst the technical "fog of war" here are the "people problems."
Jacine hinted at this with her three personas and other comments have touched on the themer vs. site builder experience, but I still think that there are some mis-matched priorities underlying many of the comments.
On the one hand there are those who are advocating for flexibility for the themer, or the ability to override, or neutralize the ability of site builders to control display in certain instances. In other words this priority:
On the other hand there are those who are working to introduce more standardization in the theme layer, I think with the idea that it will lead to more configurability and put more control over display back in the hands of site builders. In other words this priority:
The problem here I think is that both the site builder and the themer are at the service of the site owner who is the ultimate customer, particularly if we assume, as I think we can in most cases, that the site builder's task is—like the themer's—time constrained to the period around launch, whereas the site owner must manage the site and make updates on an on-going basis. Additionally one could argue that since code precedes configuration, priority should be:
But I think in most cases this is false, since site builder and themer are often working in tandem and can pass things back and forth, suggesting the most common case is:
If we agree that this is the case then it changes the way we think about things a bit. What it means is that, whatever route the themer and the site builder take, the "system" that they produce must serve the site owner who is their proximate user. We can assume (notwithstanding exceptions) that:
There are many possible directions this might suggest, but before we go there, do people feel that the above assumptions are generally true?
Comment #66
RainbowArrayOne of the things I brought up at DrupalCon NOLA, and which I went over in detail in a recent presentation I gave about component-based theming (http://2016.tcdrupal.org/session/wont-you-take-me-chunk-y-town-component...), is that in the long term, new and improved site builder tools to work with components are going to be important.
One way to look at a component is that it's a series of slots that can accept data. Each slot can either have a 1:1 ratio, where it accepts an individual piece of data (with requirement on what sort of data is expected for that slot), or a 1 to many ratio, where it can accept multiple pieces of data.
A 1 to many slot would be sort of like a region. In a section component, a 1 to many slot might contain multiple item components. In a page component, a 1 to many slot might contain multiple section components. In an item component, a 1 to many slot might contain multiple data components (essentially fields).
With a 1 to many slot, a themer should not expect to control the order of the components within it. If that level of control is necessary, create a 1:1 to slot for an individual piece of data. That can set expectations for site builders that some slots on a component have controls on how things are ordered; others, not so much.
With a 1:1 slot, a themer would be free to move those around within the component's markup as necessary.
With a 1:many slot where a site builder is controlling the order, I like the idea of Drupal generating a Twig template file behind the scenes that contains a set of include statements specifying that order. If that auto-generated Twig file could be included within the main component template file, that would simplify the process of using that component's markup in external sources like client-side JS and pattern libraries.
One thing that Panels does to help site builders is allowing for the inclusion of an admin layout CSS file in order to give basic layout styling within the admin view to help show how various layout regions relate to each other. Allowing some sort of component layout admin CSS file could do the same thing, helping to set expectations for site builders.
Integration of components with site builder tools might be a longer term goal, but I think that's going to be a more reliable way to communicate expectations with site builders what they can do with the UI in terms of making changes to the appearance.
Comment #67
Wim LeersSo, as I suggested in #56, and as was confirmed by others in #57+#58, we're going to let this issue rest for a while. We're shifting our focus to #2759849: Create a new user-facing core theme as part of a Drupal demo instead. Hence marking this issue .
This issue at a high level
The high-level direction I set for this issue has garnered wide support. But as soon as we start to look at more specific technical details or more specific goals, there are dozens of strong opinions. It's already extremely hard to unify & simplify something that has grown organically for about a decade (it has become clear after posting this issue and working on it more that even all the cache tag work I did in Drupal 8 core was laughably trivial and tiny in comparison). But add to that the requirement that we cannot break BC, yet we must provide more simplicity, plus dozens of opinions/goals to unify… that makes this issue very hard to move forward.
So, instead, let's focus on #2759849: Create a new user-facing core theme as part of a Drupal demo for Drupal >=8.3.x core and https://www.drupal.org/project/components for Drupal >=8.1.x contrib.
This issue at a low level
The low-level direction I set for this issue has gotten lots of "yays", but over time it's become clear that the precise technical plan I've written out in the issue summary will not work. I started out with "make everything a component", where "component" means "make all
#type
and#theme
stuff#componen
, and let those components be much more strict and defined without any PHP code". I still think this is one of the most doable conversion processes that yields tangible benefits.However, since then it's become clear that this is not ambitious enough: it still doesn't solve the problem that catch pointed out in #9 (which we tried to address during the conversion to Twig in Drupal 8, but we failed) — @mdrummond also voiced a similar concern ( ). @JohnAlbin posted a most excellent providing his thinking and experimentation around this. In his approach, we don't even need
#component
; components would live solely in Twig + accompanying metadata file . His solution for getting data into those Twig-based components: , which "route" or "map" Drupal data into Twig components. This is most elegant. And better yet, at least two other teams independently came up with the same approach! See https://micah.codes/a-new-design-system-architecture/. Perhaps best of all: this doesn't require any big API additions or changes!So, lots of good reasons to get much more experience with "presenters" in #2759849: Create a new user-facing core theme as part of a Drupal demo, before we continue this issue/patch. Because presenters are key to maximize reuse of components across different "Drupal thingies": different presenters can use the same components easily. And components can easily be written in a Drupal-independent manner.
In other words: onwards in #2759849: Create a new user-facing core theme as part of a Drupal demo!
Reading & watching
In the #components channel in the "Drupaltwig" slack, I've been given many interesting links by people providing feedback. Recommended reading/watching material for sure. For your sake, I'm including that whole list plus my brief thoughts on them:
The first two provide convincing arguments supporting this direction. The latter I would consider a hack that's only useful/interesting in a very narrow context, I don't see how it's generically useful (it's flawed, as the project page itself already points out) — although it does show how things hidden behind #pre_render can be a problem.
I think this may end up be essential if we want to have a simpler rendering system that's ready for the future. If we can "bubble up" our data requirements much like we already bubble up assets and cacheability metadata (but those happen *during* rendering, this would have to happen *before* rendering), then interesting possibilities open up.
I think we should totally look into bringing that to #2759849: Create a new user-facing core theme as part of a Drupal demo, I'll take that on.
My status
I'm currently working on things that must happen before 8.2.x is frozen: features for
editor.module
,ckeditor.module
andrest.module
. I'm also working on closing the few issues thatbig_pipe.module
has, so that it can be either marked non-experimental, or at least beta-level stability in Drupal 8.2.Then starting at the beginning of August… I'll be AFK for about a month (getting married and honeymoon!). When I get back, hopefully #2759849: Create a new user-facing core theme as part of a Drupal demo will already have started doing some work, or has at least defined a design direction, so that I can play the role of there.
In other words, this issue is put on hold a bit. I will try to contribute https://www.drupal.org/project/components in the coming weeks, I definitely will when I get back, and I encourage everyone here to do the same!
Comment #68
Wim LeersComment #69
Wim LeersSome people told me that the first Medium link is not being linkified in my comment. Turns out this is either a D7 core bug, or a bug on d.o. Filed an issue: #2765681: URLs not linkified (D7 core bug? Works fine in D8).
Comment #71
webchickSo, watching how #2759849: Create a new user-facing core theme as part of a Drupal demo has spun completely off the rails to arguing about technical implementation details, I'm really wondering if it's a good idea to postpone this issue (which we know we need) on that issue (which is "nice to have" by comparison). We know front-end developers find Drupal difficult to work with because our front-end code is not componentized enough. I have never heard of anyone complaining that they refuse to use Drupal because Bartik looks dated. (That target audience is going to replace the default theme anyway.)
The rationale for needing a firm target to try and work out the details here makes a ton of sense to me. But then why not take one of our existing themes (either Seven or Bartik) and compontentize it? Then a) we can start on it more or less immediately, b) we know when we're done, and c) we're not pushing another 6+ months process to find a designer (without a budget) in front.
Thoughts?
Comment #72
catchYeah it also seems odd to postpone this. It's one thing to stop working on it actively, but there's no hard dependency.
Additionally, I think there's still a lot of mileage in #1804614: [meta] Consolidate theme functions and properly use theme suggestions in core for minor releases. The less unique theme hooks core is using, the less things have to be considered for porting to components (this was my original hope for the Twig conversion too, but it ended up being quicker to do lots of ports in the end).
Comment #73
Bojhan CreditAttribution: Bojhan as a volunteer commentedComment #74
davidhernandezI think it was postponed on the belief that it is hard to decide what kind of system we want without experiment on an end result first. You don't know what tools you need until you try to build the ship. It is a bit of a chicken and egg.
Comment #75
Lukas von Blarer@davidhernandez true. But @webchick's suggestion to use an existing core theme as a proof of concept seems to be the perfect solution to this, right?
Comment #76
cosmicdreams CreditAttribution: cosmicdreams commentedSeems like a good idea, as long as we acknowledge that whatever theme is made in the future may not actually reuse these higher level components or may need lower level components to diversify. There might be a lot of rework. I still think the exercise will be educational as this is one of the things we want to investigate: how much reuse of components we will achieve.
If we analyze an administrative theme like Seven, we should also acknowledge that the while there is some commonality of components with user-facing themes there a large difference of use and therefore a large potential for describing components that user-facing themes may not use.
All I'm really trying to say is that I think this effort will be great, if seen as a warm-up or practice run of the work we will need to do to itemize, categorize, and describe the components that the new theme will need to provide. If we focus on the process rather than the results we may be able to get this process down for when we need to do it for the theme.
Comment #77
RainbowArrayWrote up a blog post with some thoughts I've been having on this subject: https://www.marcdrummond.com/posts/2016/09/26/why-base-drupal-theme-syst...
Comment #78
Gábor HojtsyComment #79
cyb_tachyon CreditAttribution: cyb_tachyon as a volunteer commentedI've been working through all of the concerns and issues we've been discussing, along with doing a couple more new enterprise-level Drupal 8 component-based builds.
This touches on something I've commented on in # 2707849: Component Libraries >> Allow a component library to define asset libraries.
I think I've got a good "idea sketch" of a prototype to build that could be a good solution for most usage cases, that doesn't bind us to using something like Paragraphs etc. in the back-end build. I spending some of my personal time working on a possible presentation with a prototype to show.
Some discussion points:
The resulting Components contrib Feature Proposal issue is here: #2848222: Add an editing interface for Component Block Presenters
Comment #80
lauriiiComment #81
Wim LeersComment #82
btopro CreditAttribution: btopro commented+1 to web component / theme components for use in libraries. Web components would have much larger inter-op with other libraries. Good progression list toward custom element interop -- https://custom-elements-everywhere.com
Comment #83
vijaycs85We have implemented a solution based on this issue which is live. Check out Druponent for more details.
Comment #84
btopro CreditAttribution: btopro commented@vijaycs85 would be interesting to see capability difference / DX workflows between Webcomponents module and Druponent (beyond D7 vs D8 stability). WC module has highly desirable developer workflow patterns and makes drupal theming enjoyable (by not engaging in it nearly as much). Also has support for Display modes / Display suite as well as wysiwyg template and a 1-page app routing methodology.
Biggest difference at a glance is that Druponent needs to define assets while webcomponents module discovers assets put in places it's looking for and then registers them as entities automatically by parsing the files.
Comment #85
vijaycs85@btopro main difference between WC and Druponent is, Druponent doesn't integrate components with Drupal fields/formatters/widgets and so yes, not much out of box there. Instead, each component become rendered element (and we have layouts too). The work done on this issue come in place to discover the components and register them as render element or layout.
Here is the workflow:
1. Build components on a front-end framework you like(Sample components we have in Druponent are pure ES6, CSS and twig files. no framework at all).
2. Make sure each component has [component-name].json or YAML file(current one has JSON and gets converted to YAML), so that drupal knows the variables in twig file.
3. Set of gulp tasks to move the components to right dir, convert JSON->YML, process & minify SCSS, minify JS etc
Currently all component assets (CSS/JS) are compiled as one file, but we could really make each of them as dynamic libraries and can be attached as part of individual components. We didn't really need on our project.
Now, the part why didn't we integrate the components with drupal-ly stuff:
Mostly because we don't want to tie up with an agreement of content modal. We had 3 custom enties:
1. Content (Content entity) + Content type (Config entity): Bundled entities for content with paragraphs for editorial content.
2. Component (Content entity) + Component type (Config entity): one-to-one mapping with UI components(match with machine name).
3. Mapping entity (Config entity): Mapping of individual fields of each content to relative components.
This way, it's free have many-to-many between content and components.
Finally, use page node and panelizer to add components pages.
Surely it's not for all sites. We can use any content modeling in the CMS that works for the teams and org managing content.
Comment #86
aleksipI have opened a related issue for discussion: #2916453: Decide on committing to and adopting W3C Web Components standards
Comment #87
RainbowArrayI think it's worth mentioning for these reading the latest messages that this issue was *not* about web components, but about modernizing Drupal's theme system to be based around design components rather than being so data-centric. A "web component" (as the W3C defines it) certainly could be one output of a revised theme system, but there could also be standard markup output using Drupal's library system to attach CSS/JS.
Even though the word "components" is used here, I think it's important to keep that distinction clear if we ever pick up work on this issue again.
Comment #88
dasjojust for reference, pmelab wrote a blog post mentioning this idea
https://www.amazeelabs.com/en/blog/dont-push-it-using-graphql-twig
Comment #89
askibinski CreditAttribution: askibinski as a volunteer and at iO commentedAlso, there is a lot of work being done by in contrib with ui_patterns. Dries actually tweeted about this recently.
See @ademarco's presentation at DrupalCon Vienna on the topic. Especially the part starting at 41:08 mentioning ideas like twig annotations and decoupling from drupal (and why do we even would need a hook_theme anymore).
Comment #90
donquixote CreditAttribution: donquixote commentedFor me the biggest problem was always if multiple modules try to alter the same render array, and also the structure morphing from form array to render array.
If the "aspect-oriented" part is taken out of the picture, things already get more pleasant.
@Wim Leers (#36):
I think this contradiction only applies with a narrow understanding of "component", where a component is a self-contained visual element.
For me, a "component" can also be a wrapper, or "decorator" if you want. Another type would be a "list format".
Or perhaps we don't need to think of a different type of component, rather a different way to use it and to feed its parameters.
Any component with the signature of a single-region layout can be used as a wrapper, wrapper/decorator, where rendered content from a previous component (in this case the children) is piped in.
I think it makes sense for the render system to natively support this "output piping", without making it the responsibility of the theme hook to render its own children.
Perhaps there could be better ways to do this than '#theme_wrappers'.
Currently '#theme_wrappers' is the only way to pipe the rendered output to another theme hook.
One benefit is that '#theme_wrappers' can be merged in with element type defaults.
Of course one problem is that it shares element properties with the actual '#theme' hook.
I could imagine interesting alternatives to make this possible, while isolating the properties space. E.g. something like this:
$element['#pipe'][] = ['#theme' => 'container', '#attributes' => $container_attributes];
But in case of e.g. 'form_element', isolating the properties space is actually _not_ what we want.
(I apologize if my perspective is a bit tainted by Drupal 7)
Comment #91
fgmCan't believe I only just discovered this issue 4 years later. Anyway, I've been wondering about #theme_wrappers : it seems the issues that keep being mentioned about them being required relate to information in child components/elements, for which WebComponents and other tools like VueJS have the notion of (single, multiple, named) "slots" ; yet only #66 by mdrummond seems to use the word more or less in that sense.
Isn't it something that should be considered in the summary ?
Comment #92
anruetherComment #93
catchComment #94
donquixote CreditAttribution: donquixote commented@fgm (#91)
I had a look at the respective doc page in vue.js.
A "slot" has a similar role as a "region" in a Drupal layout, or a "field" in a ui_patterns pattern, or a "variable" in a theme hook.
The cool thing is the syntax that allows one template to "capture" html in a variable, which is then sent to another template.
In the examples mentioned there, the calling template explicitly says which other template should be included.
The '#theme_wrappers' focuses on a different part of the problem. It is about how data and rendered html should flow through the Drupal render system, between templates or theme hooks.
Instead of one template that directly includes another template, we have two templates or theme hooks that know nothing of each other, and a layer on top that sends the data from one to the other.
E.g. in case of a form element, one template is responsible for rendering the specific form element, the other template then adds a wrapper around that element.
For this, the "slots" syntax to capture html in variables would not give us anything.
Comment #95
donquixote CreditAttribution: donquixote commentedBtw, the problem of "use the output of another element" is already solved for some element types like "link":
The
Drupal\Core\Utility\LinkGenerator::generate()
will render the '#title', if it is a render element.There could be an alternative element type 'link_wrapper' which would instead use a '#post_render' or '#theme_wrappers' to wrap the child elements:
Which of those render element structures is the better DX?
In the second example it is more obvious what is a render element and what isn't:
Everything that is a render element is under a key without '#'.
We can analyse the complete tree structure just be following the keys without '#' prefix.
In the first example, some properties contain render elements, others just text.
Also, I like the "inversion of control" spirit in the second example: The render system passes rendered html to the wrapper, the wrapper does not need to have its own reference to the renderer object.
Comment #96
DuaelFrAs a Drupal trainer, the thing I'm asked every time about those render arrays is "How do we know the keys we can use in the array?"
For me, that's the main point of this system: discoverability. That's even more true for forms now that we have lost the big documentation table that existed in D7 and prior.
Another pain point for trainees is to understand #theme vs #type and how to choose and discover them. Even with the appropriate tips and tools, they often think they have to recreate the wheel because they didn't find the thing they though about. That's kind of fun but I specifically incorporated a point about making unordered lists in my trainings because they never find the item_list theme key.
I feel like a solution would be to use data objects instead of arrays. They would replace Render Elements and would be the only entry point of the render array (#theme would be internal). Each layer of render could be an object with a common interface to manage the tree and the cache (getParent, addChild, setChildren, getChildren, addPreprocess, setCacheTags, etc.), and with custom, but declared, properties to define the data needed by the object (setHeaderType, setMaxDepth, or whatever).
We would probably need to define a lot more to mimic the actual capacities of the render arrays but I think it would be better for readability and even alteration. I'm sure it would be more verbose and I'm afraid it would need quite some code to maintain a proper BC layer, though.
Comment #97
catch@DuaelFr there's actually a proposal here which is close to what you're suggesting. Still using arrays for the final data structure but adding classes to define them in the first place. #2316941: Use the builder pattern to make it easier to create render arrays - that step can be done without any bc layer and might make the next step easier to think about.
Comment #98
Chi CreditAttribution: Chi commentedI think builder would make sense for immutable data. It's quite often that render arrays need to be altered.
Comment #99
cyb_tachyon CreditAttribution: cyb_tachyon as a volunteer and at Red Hat commentedWe're already onboard with most of this at Red Hat, and probably a step or two ahead in some areas.
We do disagree with #3 YAML metadata though: Most web component & Twig libraries we've encountered use JSON Schema, an already popular and well-understood format for discovering and mapping data to component slots.
Here's just one example from Microsoft: https://fast.design
We are using https://www.drupal.org/project/patternkit , which we plan to contribute to Drupal 10 core as an enhancement to the core library system.
I'll avoid updating the issue description for now to foster clearer communication, but at some point we should hop on a call, copy it to a Google Doc / Collaborative platform, and hash through what next steps look like.
Comment #100
pdureau CreditAttribution: pdureau at Smile commentedAs mentioned by @askibinski on comment #89, https://www.drupal.org/project/ui_patterns is a great start to rethink the Render API, because they done a lot of thinks right.
Easy declarations
Each render element is a plugin which can be generate by:
More info: https://ui-patterns.readthedocs.io/en/8.x-1.x/content/patterns-definitio...
No need for Element plugins as PHP classes. No need for hook_theme.
A clean uniform structure
Instead of the usual mess, ui_patterns render elements still use a nested PHP array structure, but very clean and very predictable, with those keys :
Example (with #type key removed):
A leaner proposal where #fields are element children (the children of an element array are those key/value pairs whose key does not start with a '#') :
Site building readiness
Such a well defined structure allow some mapping with other plugins types :
This direct mapping between render elements and site building is very powerful and efficient, it has been 3 years me and my team do all projects like that, and we have streamlined our workload a lot.
Comment #101
joachim CreditAttribution: joachim as a volunteer commented> As a Drupal trainer, the thing I'm asked every time about those render arrays is "How do we know the keys we can use in the array?"
That is a documentation problem, and it's a regression from D7 when that information was in the api.php file documentation for each theme_foo() function. I'm pretty sure there's an issue for it.
Comment #102
nedjoIn the Component Schema module I've started to map out an approach to components modelled on core's config schema API using the typed data API.
Component Schema allows any module or theme to define components and their variables. Bulma Components is a sample implementation.
Component Schema defines a number of schema types that other modules or themes can extend. For example, Bulma Components defines a set of extended types that are used across multiple components in a format that will be familiar to anyone who's used the core config schema API.
Component Schema doesn't include any admin UI integration directly but does include a submodule that derives UI Patterns from defined components and their variables.
Comment #103
geek-merlin@nedjo #102:
Wow, this should be the way to go, rock on! 💪
In cssvars.module => cssvars_color, i **dynamically** created config schemas, and leveraged my configelement.module to **auto-generate the config UI**. Maybe that helps.