[plan] Add layouts to Drupal

(Just adding some markup for scannability.)

Trying to clarify the plan for layout plugin per @tim.plunkett.

@tim.plunkett explained that #2795833: [plan] Add layouts to entity displays (both form and view) is a separate idea, and the experimental module prototype for that may depend on and will validate this API.

#2834019: Remove the "layout__" prefixing of theme hooks is currently in the "should have" list. If this is only with respect to committing the module to core as experimental, then I'm ok with that. But, per #2834019-3: Remove the "layout__" prefixing of theme hooks, I believe this is a "must have" for the module becoming non-experimental.

The initial patch for this initiative has been reviewed and committed with ample community input, so based on that, I would consider this initiative plan RTBC.

I've discussed this plan in depth with @tim.plunkett. It reflects my feedback and addresses my earlier questions. Based on that, I'm also adding my signoff.

@catch, @alexpott, @effulgentsia, and I have also discussed the framework and release management especially for the first and third steps with their followups. The two parts of the second step probably need additional architectural review. Edit: Per @tim.plunkett, the page layout part will need a lot of ideation work still. OTOH the field (content) layouts already has an RTBC initial patch. :)

Also, this is definitely at the plan stage.

For product managers and UX maintainers particularly: Note that the issue for field (content) layouts has its own sub-plan already: #2795833: [plan] Add layouts to entity displays (both form and view) So if there are questions about the specifics for that, UI prototype, etc. we should probably discuss on that plan. (This issue is the high-level roadmap for the core initiative overall.)

This is awesome.

All the talk about layouts seems very desktop-centric. Wondering how mobile-first responsive design fits in, if anywhere.

I see a lot of talk about left/right, one column/two column/three column, as well as other cool and complex layouts. For example, https://www.drupal.org/node/2660124 will show the icons as only one version of the layout, however that layout may look different on mobile, or have any number of variations.

In our practice, mostly using panels, we've shied away from using terms like two-column, left sidebar, etc, in favor of primary, secondary, etc, so site users are thinking more about importance than layout. I'll admit that this does nothing to help a person visualize where a piece of content will be placed and how it'll look depending on the size of the browser window.

Keep going! I'm just trying to wrap my head on how best to use this, and I'm sure you've all discussed these things in your internal discussions. I'd love to hear your thought on resolving some of these issues. If there's another issue somewhere that talks about this, please point me in the right direction.

Thanks,

•• matt

@mattlt
when it comes to desktop & mobile layouts it is/will be upto the layout plugin to work through that. ie: Bootstrap Layouts fills that use case (layouts built from responsive grid systems)

Tim and I moved a couple issues to #2795833: [plan] Add layouts to entity displays (both form and view) that are in scope for that module, but not specifically blockers for Layout API itself being considered stable.

It also occurred to me that we might not want to mark Layout API as stable until the implementations validating it are beta, so that if we discover new additions or API changes that are necessary based on those implementations, Field Layout improvements can still happen in patch releases and are not pushed to the next minor. I see it going something like this:

1. Layout API alpha (done)
2. Field Layout alpha
3. Contrib implementations alpha
4. Layout API beta
5. Contrib implementations beta
6. Field Layout beta
7. Layout API stable
8. Contrib implementations stable
9. Field Layout stable
10. Updates to Standard, if any.

Any number of these steps could happen within a minor, though I'd imagine 6 on will happen after 8.3.0 so they can include feedback and testing from the experimental module process.

Adding a note about that too.

We discussed this on the core committer call today, and all agreed with the plan laid out here. I'm in favor of moving forward here, with Tim as the initiative lead as he’s been the one driving putting the plan together. Very excited to see this move forward (officially). Thanks everyone involved creating this plan and contributing to the layout initiative so far. As a next step, maybe Tim can create a patch making him the initiative lead?

Adding info on the contrib implementations we are tracking to validate the API.

Adding #2848549: Refine the layout selection UI to the "Should".

Looked at this today with @japerry and @DyanneNova at PNWDS.

So the problem is we currently have distros such as Lightning or Thunder or whatever that are using Panels for layout functionality. At the moment, those distros are pulling in https://www.drupal.org/project/layout_plugin which, although the version number shows as alpha, has already been vetted by both Display Suite and Panels APIs, and is used in production on ~15K sites.

Now, 8.3.0 is poised to ship with Layout Discovery module as experimental, which is actively discouraged from real-world use. Display Suite and soon Panels have a -dev branch with compatibility with Layout Discovery, which in theory helps validate the API, but only if people actually use them. https://www.drupal.org/project/usage/ds as of today shows literally six users of the 8.x-3.x-dev release (which uses the core stuff), compared to 7600+ users of the latest 8.x-2.6 release (which uses Layout Plugin).

Which puts these distros, as well as core, in a pretty awful position. Do they rely on a deprecated/abandoned contrib module for the next 6 months until 8.4.0, when we will hopefully receive enough independent verification to ensure that we can elevate Layout Discovery to stable? And as the core development team, how can we in good consciousness elevate something from experimental to stable if the real-world usage is absolutely pitiful? This puts the entire ecosystem at risk, and actively harms overall Drupal 8 adoption. :\ And as a result, we're starting to see a fraction in the ecosystem around people who are using Paragraphs for this use (which is meant to be a field structure system, not a layout system), and one of the big points of putting this stuff in core in D8 was to avoid such fractions.

So what are our options here? Over the next month (the sooner the better, obviously), if Panels does a similar thing and starts a dev branch that utilizes the core stuff, and all tests are passing, can we elevate Layout Discovery to stable during RC? (This seems to be the only must-have in the list left, at least within core's purview.) Or, are we simply resigned to foisting 6 months of scary ambiguity among sites that require layout functionality?

Where "Panels starts a dev branch that goes against the core stuff" means *using* the core stuff? Or moving away from it?

Oops, sorry. I meant using the core stuff. Edited to clarify.

Thanks!

Well, I'm not sure "use this alpha that 15K other people, two major contributed projects, and several distributions are also using in production" vs. "use this alpha that has been refactored from the original and no one else has used in production yet" is the same level of risk, like, at all. :)

Which puts these distros, as well as core, in a pretty awful position. Do they rely on a deprecated/abandoned contrib module for the next 6 months until 8.4.0, when we will hopefully receive enough independent verification to ensure that we can elevate Layout Discovery to stable?

Actually, right now, the primary blocker to Layout Discovery being marked beta is that Panels has not opened a development branch. The primary blocker to Layout Discovery being RC will probably be Panels just confirming that it's good enough for them. Most of this module's roadmap is already done. Unlike Field Layout, it doesn't provide UI or anything. Panels is the independent verification we need.

Also, I already outlined this in #30 above.

Ok, just told Jakob that. ;) Stay tuned.

Display Suite and soon Panels have a -dev branch with compatibility with Layout Discovery, which in theory helps validate the API, but only if people actually use them.

From a (semi) maintainer point of view of DS.

It might be important to notice that the 8.x-2.x branch of DS will /not/ work at all with core 8.3.x, which we already have put on the project page of DS itself. The main reason for this is not layout plugin in core itself, but due to subtle changes in entity displays and Field UI. Making 8.x-2.x of DS work with core 8.3.x BC wise and using layout plugin from contrib was an option, but in the end, it was kind of natural to start a new branch which uses core layout plugin and works with the Field UI/entity display changes in 8.3.x. This might be a little adventurous, otoh, given the little difference of layout plugin in core vs contrib, I'm confident that this won't be a problem at all for users when they have upgrade core. (core has a more interesting update anyway in core.extension for the profile for instance). And to be fair, imho, layout plugin is kind of beta ready anyway (at least for DS, can't tell for panels))

I don't remember having to make enormous changes in the code of DS (thanks to initial patches from Tim), so as soon as panels starts a branch on core layout plugin, we can deprecate the contrib version of it.

I know there's an issue to coordinate releases for several modules, I might ping Alex on that next week when I'm in london.

Put another way, this API is being added to core primarily to support DS, Panels, and future core implementations all at once. We're unlikely to majorly disrupt/break the contrib projects we're trying to support because that would defeat the purpose. :) We will break if they need us to for their stuff to be supported.

@xjm that's a good summary :) DS is using layout plugin since ages (I really don't remember anymore at which point we started using it). The last major major change for DS (#2612040: Allow DS to use Panels layouts and vice-versa (using the new layout_plugin API for settings)) happened in november 2015 after #2490680: Allow layouts to provide settings got committed in layout plugin, a week before we tagged a stable release of DS, two hours before core 8.0.0 ..

Now, I think I see the bigger picture/problem too: you can use Panels and DS in the same installation and then things are problematic since both can't be enabled right ? Which would potentially mean that we have to revert our branch of DS (or make 8.x-2.x work with 8.3.x) in case panels doesn't have a branch that depends on core ?

(bit late now, will think about it some more tomorrow)

tl&dr: we're working on it.

Thanks for bringing this up, webchick.

We've been discussing this in the weekly Scotch meetings for a while. Our immediate goals were a) getting the time-sensitive improvements into core that were needed, b) stabilizing the five contrib modules focused around 8.2.x compatibility (CTools, Panels, Page Manager, Layout Plugin, Panelizer). We've suffered from a lack of contributors who understand the APIs, so the few people who do have been focused on the core improvements plus their day jobs. We've identified items in each project's queue that need to be done prior to them being marked "stable", now we just need time & people to work on them. Once all five are out with stable releases we can deprecate Layout Plugin and create new branches for the other four to focus around 8.3.x compatibility and the new layout APIs in core.

So, in summary, the plan is:

• Get stable releases of the five modules with a strict line of compatibility with 8.2.x.
• Once all five are out, deprecate Layout Plugin and create new branches of the other four.
• Update the four modules to work with 8.3.x and core's layout API.
• As always, more contributors who understand D8's APIs (and the Panels APIs) would be appreciated.

Thanks everyone!

@DamienMcKenna:

To be honest, I'm not sure how realistic that plan is. 8.3.0 will be released in a bit more than a month and after that, 8.2.x no longer matters.

I've been involved a bit with Page Manger and progress there has been very slow, there were only a handful of commits since last April and a single new alpha release. And there are quite a few challenges left to resolve there IMHO before we can call that stable. And that's just one of 5 that you plan to get to stable in one month?

Why not just add a version dependency on 8.3.0 on panels and do a new release that uses the core layout discovery module? DS did a new major branch, but they also had other conflicting changes. If the layout discovery and the layouts themself are compatible, then I think it is fine to do that without a new major version and actually easier pre-stable.

That helps core and all users in two ways:

1) As mentioned above, the main blocker for layout_discovery in core getting (more) stable is panels using it and confirming that it works. If we do this now, then we can possibly make it beta in 8.3.x already which makes sure that there will be no API changes in 8.4.x and it hopefully can become stable there.

2) With DS already being updated and only compatible with 8.x-3.x, users will not be able to use DS and panels in 8.3.x.

Btw, I recently moved my custom widgets projects (https://www.drupal.org/project/widget) to drupal.org and ported it from using the very, very early layout.module (https://github.com/frega/layout). We have been using widget in NP8 since 2014 (http://www.md-systems.ch/en/blog/md-systems/2014/06/04/layout-for-drupal-8) in NP8 and it was kind-of-public on github. I actually have to change the templates in our case, but I hope it will not be too problematic for our projects. We don't actually use DS or panels right now in NP8, but I still plan on making this switch as part of the 8.3 update and will start to test this soon (I already tested 8.3.0-beta1 and it looks pretty good).

Looking from maintenance of few projects it's really not clear why lauout_plugin module can't implement service overrider to prevent collisions with core's discovery?
It makes total sense to have layout_plugin shim for migration to 8.3

Is there any real blockers in "ctools&co" to be compatible in-month with 8.3 core?

If yes it is separate blocker of 8.3 adoption because core's module experimental and the same alpha as layout plugin module

Also it's not clear now to contribute to panels&ctools because issues hangs for months (++ #51) and layouts are the least question...
So maybe better focus oo core layouts instead of solving compatibility of 8.3 & layout plugin

For the next stage of the layouts in core initiative, targeting 8.5+, here are the notes from our discussion at the Layouts sprint last week!

https://docs.google.com/document/d/1rQukpyKFzeTIcVoRmCG6xTono9O2nhVxrzNS...

TL;DR: Work on the "I want to lay out node/5 differently" use case + a nice UI for content authors.

@webchick regarding this:

Entity-based focus is still the most important (Panelizer in core). The ability to override node/5. (Only on content types for which layouts are enabled.)
...
Populate layout with fields, content, blocks, etc. [Broadest definition of "content"]
...
Should be able to add entire other entity, Powered by Drupal block, body field.

Wouldn't it be easier to create a single multi-value field using https://www.drupal.org/project/dynamic_entity_reference and then use Layout Plugin to place referenced entities in regions? Need to be able to set region per delta. Revisions and translation is out-of-the-box.

I have updated the documentation page for layout plugin to help with the 8.3.x transition: https://www.drupal.org/node/2619128

Also note API documentation for layout_discovery now available: https://api.drupal.org/api/drupal/8.3.x/search/layout_discovery

I think #57 has product manager review aplenty, so removing that tag.

Responsive design support for layouts (e.g., breakpoint integration/support) has come up frequently. According Tim it's out of scope for the initiative (at least per https://events.drupal.org/baltimore2017/sessions/continuing-saga-layouts...).

Seems important and oft-requested. I personally wouldn't be too keen on using content layouts if they didn't respond the way the rest of Drupal does out of the box. Were the handful of default layouts added to core not responsive? If they weren't I was seriously not paying attention when I reviewed that patch. :P

At the least, I'd want a could-have, maybe a should-have. Maybe some user testing on Mobile and confirmation that the current state is okay from a product perspective?

Couple things from the Layouts session at DrupalCon:

This is a "mental note" on contextual filter stuff for Tim).

Also should stuff about the idea for the landing page builder on these roadmaps?

And revisioning stuff and data model for the stuff as discussed WRT Workflow. That's a big thing (when initiatives have an overlap, one of them has more technical debt).

This is kinda out of left-field, and I'm out of my depth, but has anyone seen this idea by Adrian83? He's created a mockup for it.

As a lowly end user I really hope that one day I can just put any field in any region for any context. And that includes the title.

(And I'd love it the other way around too: put any block in the middle of any form, so I can put pictures of cats wearing glasses all through the forms).

Keep up the good work :-)

An update from a core committer meeting we had on July 20, where we checked the status of all experimental modules.

For Layout Plugin, all API-affecting sub-issues are complete, leaving only the step to mark the module as stable:

#2834025: Mark Layout Discovery as a stable module

(Per #2834025-18: Mark Layout Discovery as a stable module in the above issue, this is currently blocked on documentation in hook_help(). The module passes the core testing gate and has already had thorough architectural review. The usability and accessibility gates do not apply since Layout Discovery is API-only.)

Stabilizing this module is important, since a stable version of both Panels and Display Suite contributed modules depend on this functionality, and the API should be treated as stable to avoid disruption to the Layouts ecosystem.

We're seeking reviews from maintainers of those modules, as well as testing of the upgrade path (as noted at #2833976-7: How Panels, Display suite, Layout Plugin, Layout Discovery and core get to a stable layout plugin in core): "According to @tim.plunkett, Panels and Panelizer have complete upgrade paths and upgrade path tests. Tim wasn't sure about Display Suite. Folks could help out here by installing earlier versions of the modules and configuring them, and then testing what happens when they update them and core."

It has been a few months now but I successfully upgraded a site that used DS from 8.2 to 8.3 without an issue.

I've done a bit of discovery on how layout templates suggestions are handled, and I found some inconsistency that should be fixed (or at least documented).

Research can be found in #2862683: 'base hook' key prevents template suggestions from working from #26 to #29 and it covers both Display Suite and Panels.
TL;DR suggestions templates are not loaded (experienced with Display Suite but I think is a core issue) unless the layout and its template definition don't pre-pend respectively 'layout__' and 'layout--' (or using '\Drupal\ds\Plugin\DsLayout' as layout class if the layout is consumed by DS). Without these twig debug comments suggest to the user the wrong template files names.

To replicate:
- install DS
- create a custom layout following Layout drupal official documentation "How to register layouts" and its base template file.
- Use the custom layout in a context with suggestions, i.e. a node or paragraph display mode.
- Alter and use the custom layout template file in order to use a suggestion from the twig debug comments.
- Clean the cache and reload your page making use of the suggestion.
- The suggestion is ignored and the base template is loaded instead.

As posted in #2834025-27: Mark Layout Discovery as a stable module, the module is considered stable now.

There are still several should- and could-have issues open, including the issue linked in #66: #2862683: 'base hook' key prevents template suggestions from working that is still postponed for more info and a related issue in #2886187: Document the regex in Registry::postProcessExtension() used to find preprocess functions was filed.

Then there's #61 and #62 which are also unaddressed. Maybe they belong on a new/separate roadmap?

Re #61

Were the handful of default layouts added to core not responsive? If they weren't I was seriously not paying attention when I reviewed that patch. :P

Currently the default layouts use one column for all their regions until @media screen and (min-width: 40em) is reached, then the horizontal columns css kick in .

I think #2919372: Layouts do not respect RTL blocks both Layout Discovery and Layout Builder from being stable, since the existing block system is deliberately direction-agnostic and styling rules also need to respect RTL. I'm not sure if this roadmap is still the right place to raise that, but I've added it as a remaining must-have (and moved a few more completed issues on down).

Tagging for an issue summary update since the current summary is based on the old roadmap, before the direction was changed to focus on #2905922: Implementation issue for Layout Builder. Per @tim.plunkett, #2795833: [plan] Add layouts to entity displays (both form and view) will take a new direction based on the Layout Builder's approach, so all that needs to be updated. :)

Tentatively