[Policy] [Plan] Drupal 9 and 10 markup and CSS backwards compatibility

Part of the issue is that Classy and core version of templates are very different right now. If you create a new base theme for Seven and Bartik it would have to be a copy of Classy, and still very different from core. This might create a lot more overhead than we already have. Changes to core template have to have backwards compatibility in Stable and then also be copied to "Unstable". We might start getting tired of this.

The disadvantage to the current scenario is that when 9 is branched and Stable is deleted, we have to do some rebuilding on Classy, Seven and Bartik. That is more work later, but less work now. Also assuming we even end up having Bartik and Seven in 9. Maybe we'll decide to rebuild/replace them anyway.

At this point my vote is for #1, I think Bartik and Seven are very different and trying to optimize to keep them more in common doesn't make sense to me. I might be missing something though.

Thanks for creating the issue @lauriii!

There are very few templates that are shared between backend(admin) and frontend(default) themes. I'm also voting for #1. We made our bed, we should sleep in it;)

I think Bartik and Seven are their own things and that was the purpose of making them @internal. To me what we decided around DrupalCon Barcelona is that they are more like their own "products" that are part of the greater Drupal product/framework/whatever. So following that reasoning it should be up to those themes how close/far they want to be from Classy (or Stable's or core's) markup, how much they want to change/improve, whatever. Seven and Bartik should in general be free to use core frameworks and APIs as they wish to accomplish their goals. In general I think it'd be totally fine if some template got overridden in Bartik and only ever lived in Bartik and I don't think that's a waste. We already have that happening and it's fine, Bartik is being Bartik. Same goes for Seven.

I'm not sure Classy would "want" whatever changes/improvements happen in Bartik and Seven either, even a hypothetical Classy for D9. Trying to keep these themes closely tied to Classy or some other base theme seems to me like it will just slow down improvements and changes to them for a possible future win.

If we see some changes in Bartik or Seven that we decide fit with the direction and philosophy of the core markup there's nothing stopping us from porting over those changes (markup for example) into core. We also discussed the idea of improving Classy in contrib and I think similarly whoever is working on that can keep an eye on what's happening with Bartik and Seven as part of that work and try to port over things that are actually generic and for the 80% use case, not for the special Bartik and/or Seven cases.

That's kind of long but I really think this would be a good topic to discuss on a future Twig call :)

May i suggested that we dont have a "unstable stable" its gonna be a huge cluster & will make people unsecure of what happens with the markup

But yes we need to be able to move forward & changes that are done in core should become available for future versions so heres my suggestion.
We have both stable & classy in contrib so lets use them.

contrib:
https://www.drupal.org/project/classy
https://www.drupal.org/project/stable

When Drupal8.1, 8.2 etc comes out we should do a release in contrib that reflects the new things that have been fixied in core.

The hoops of maintainability around these things are getting crazy to think about. I feel like we should postpone this on the composer changes and then pull the themes out of core so we can version them. We are trying to have our cake and eat it too by trying to treat them like part of the api, but knowing they are really separate libraries/components/projects/whatever.

I asked about basically this in Slack a couple of weeks ago out of my despair trying to fix markup bugs in core and hitting the deadlock case:

- Bug fixes must include tests.
- Tests run against classy.
- Thou shalt not touch classy (or stable).
- Deadlock.

Neither @lauriii nor @xjm mentioned this issue, they encouraged me to open an issue about it, so I created a somewhat duplicate meta, #3031198: [META] Add classy_dev and minor-branch theme snapshots to allow fixing markup bugs within minor releases without front-end disruption. Then, last night @lauriii pointed me here. ;) That issue contains my proposal, and a working patch, and shows how we can unblock the deadlock. I don't want to be accused of "hijacking" any more issues, so I'm not going to mark #3031198 duplicate and convert this issue into my proposal. Comments over there welcome.

Cheers,
-Derek

@lauriii re: #25:

To be able to change markup and CSS without breaking themes (and without having Stable), we will disallow partial overrides of components.

For everyone not at the discussion at DrupalCon, it's not at all clear how the conclusion "we will disallow partial overrides of components" addresses the premise ("To be able to change markup and CSS without breaking themes..."). Can someone explain how forcing someone to override markup, CSS + JS together makes this any better? Without the benefit of the discussion, it doesn't seem like that solves any problems, but creates some others:

- If I'm developing a custom theme with a core base theme, and instead of defining even more specific CSS to override duplicate styles from the base theme, I often want to disable the base theme's CSS component, but am happy to keep using the template and/or JS associated with it. I believe you're saying this would no longer be possible.

- Ditto: I might just want to swap out the twig template to change some markup, but am happy to keep using the related CSS.

Meanwhile, I don't know how this actually solves the "problem" that our current BC policy is trying to fix. For example:
- Let's say we have some core templates for form elements that produce buggy output (because we do). ;)
- Custom theme extends a base theme.
Either:
A) Custom theme overrides the template for the form element (to fix the core bug) so core fixing the bug doesn't break their theme.
B) Custom theme does *not* override the template from core, so core fixing the bug should fix, not break, their theme.

I don't see how "partial" overrides changes any of this. I still don't really understand our current policy, but I also don't know what this proposal does to make anything better.

Please explain. :)

Thanks!
-Derek

I often want to disable the base theme's CSS component, but am happy to keep using the template and/or JS associated with it. I believe you're saying this would no longer be possible.

- Ditto: I might just want to swap out the twig template to change some markup, but am happy to keep using the related CSS.

Probably, yes. We're mostly just talking about Stable here, so its debatable if we need to move it away from using libraries since it won't ever change, but we'd want to encourage the habit of not separating the template from the css/js so we might enforce it.

You will have to bring the css/js with it as a bundle if we attach from the template. If you use the starter kit this won't matter, because it will already be there. And the official recommendation will be to use the starter kit.

- Let's say we have some core templates for form elements that produce buggy output (because we do). ;)
- Custom theme extends a base theme.

The only base theme you would be permitted to use is the new version of Stable. Stable templates+css+js would not change, preserving BC when you do a minor version update. Module stuff can change, but won't affect you.

The difference is changes will be permitted in module templates and incorporated into the starter kit theme. (We currently can't change Classy, which is the issue) Any new themes created using it will have the updates. The core updates will have release notes documenting the changes so people who built themes on the older versions of the starter kit can incorporate changes at their leisure. The starter kit would be a copy of all templates, css, and js from core. We affectively eliminate the chain of module-stable-classy-theme. Your theme can maintain BC, because it is frozen at its time of creation. The starter kit will not rely on Stable. If you use Stable, you will not see updates.

Moving the attachment of css/js to the template serves the purpose of locking the template to the css/js in its present location in the module/theme. If you want to override the template, you have to bring the css/js with it. Doing so lets you maintain BC, because if we change css/js (and the template to accommodate, or vice versa) we won't end up with template+css/js combinations out of sync.

Essentially, the theme you create based on the starter kit should serve as its own BC layer from the moment you create it. Changes within modules would still need to make changes in a BC fashion, but that mostly involves just not deleting existing variables.

So three scenarios:
1. Use starter kit - built in BC, you see nothing from core when you look at your source code.
2. Use Stable - inherited BC, you get no updates.
3. Base theme = false - you get all updates, and we make you copy css+js to your theme when you override a template so as we change things your markup can't get out of sync with the css/js that belongs to the version of it you have.

In the example of fixing a core form template. We change the core template+css+js. Copy update to the starter kit. Any new themes get the update. We enforce you copying css/js with the template so can't get an updated template without the updated css/js, and vice versa. We should be able to then change most things we'd want to in core while eliminating most of the avenues that lead to a core change being unexpectedly noticed by your theme.

What I'm trying to think about now are the automated tests. We've had a problem because they use Classy, so we're eliminating Classy, and I presume would use the most up to date version of the starter kit. But that doesn't help us ensure Stable doesn't break. But, I don't think we are doing that now since we don't have theme specific test coverage.

Thanks to everyone who contributed to this already! 🙏This is hard.

Steps 1 and 2 in the IS (#3050374: Create Drupal 9 stable theme and #3050378: [meta] Replace Classy with a starterkit theme) sound rather daunting. Step 3 in the IS seems wrong to me, but I may be misunderstanding what problem it's solving. I asked for a clarification in #3050386-5: Allow loading CSS and JavaScript directly from templates.

Reading the rest of the issue, and summarizing:

1. #2 through #12 seem to be centered on interactions between Classy/Stable/Bartik/Seven.
2. #20 mentions pulling these themes out of core and into their own repos with their own history with composer pulling it in. Nearly a year after that comment, that unfortunately still sounds like a distant future.
3. Both #20 and #22 talk about how complex it has become to make changes.
4. #24 and later start to consider this actively in the context of Drupal 9, which is happening soonish now.
   

   The group recognized the downside of not being able to provide security fixes or bug fixes for themes built using this approach.

   That's indeed important, and that exact argument is the reason we're not marking back end code final today: because only if extending is allowed, security fixes are inherited automatically. The same principle applies here.

   We believe it isn't big of a change from the status quo since […]

   That's interesting! How confident are we about our perspectives of "most themes" and "commonly used"?

   To be able to change markup and CSS without breaking themes (and without having Stable), we will disallow partial overrides of components. This can be achieved by tying CSS and JavaScript to a specific instance of a template. This means that if a template is overridden, this will require themes to also override the CSS and JavaScript.

   I don't understand this. Can you give a concrete example? I'm probably missing something.

   Ah, @dww asks the same in #26 :)

   #27 tries to explain it. Let me call out the paragraph that stands out to me:

   Moving the attachment of css/js to the template serves the purpose of locking the template to the css/js in its present location in the module/theme. If you want to override the template, you have to bring the css/js with it.

   I do see how attaching asset libraries in Twig templates makes it more self-contained and hence more easily overridable as a unit (template+asset library). I'm in favor of dropping #attached support from preprocess functions.

   But it does not quite convince me of the need for individual CSS/JS attachments, rather than the asset library attachments which we already have today.

I don't understand this. Can you give a concrete example? I'm probably missing something

I tried addressing that a bit in #27. Or do you mean a code example?

One that occurs to be often is Classy's node template. We have this:

{{ attach_library('classy/node') }}
<article{{ attributes.addClass(classes) }}>

If you copy this template to your theme, that library attachment is still active and working. We can't touch Classy's template and CSS because even if you are overriding the template, you may not be overriding what is in the library. Changing the template and CSS (like removing a wrapper div or changing/removing/adding a class name) means the version of the template you have can get out of sync with what is in the library in core (Classy).

Also, by continuing to use libraries in core, we invite people to use them, and thus are limited in changing them in the future. I could attach any one of Classy's libraries in my templates, if I feel its relevant.

If Classy didn't have libraries and we change that to:

{{ attach_css('css/node.css') }}
<article{{ attributes.addClass(classes) }}>

and we leave that path relative to the theme, when you copy the template we force you to copy that css, make your own to replace, or remove the attachment line. Any of those options keeps you BC safe.

Parenting to beta1 requirements issue as it is listed in the plan as a must-have by then.

Okay, here's a status update on this issue!

As @lauriii mentions, I evaluated that it was too disruptive to remove Stable and Classy from core for 9.0.0, given the short timeline (even from back in June).

• They need to be deprecated first according to the continuous upgrade path, and in order to deprecate them, the replacements also need to be available first.
• Furthermore, since most themes and therefore most sites depend on at least Stable, moving it to contrib isn't really a good choice -- it would still be a part of the core usecase for 9.0.0, and would still need to be supported even if wasn't "supported".

So, my recommendation was to deprecate them during the Drupal 9 cycle for removal prior to 10.0.0.

Regarding the specific proposals for the next version of Stable and for Starterkit:

Stable

The proposal to create a new Stable theme for each major release of Drupal makes a ton of sense to me, and is a lot better than complicated and not-well-thought-out scenarios I'd imagined involving a @deprecated equivalent in individual templates and CSS files with some override magic something something. (Because it's all exposed as public API and how would you even get around the naming requirements? And how do you deprecate one rule in the CSS cascade independently? I am clearly not a theme developer...) A new base theme built from the latest module markup as of the branching point is much cleaner.

I was concerned about stable meaning one thing in 8.9 and a different thing in 9.0, so giving each new stable theme a new name, like stable9, stable10, etc. is explicit and straightforward, without too many drawbacks.

Starterkit

I'm less sure about the Starterkit proposal.

• My main concern was actually that there simply wasn't enough time before 9.0.0 to develop the feature, and we can't afford to have features as release blockers.
• As Lauri mentions, I was also concerned that we would no longer be able to ship security fixes for preprocess, markup, and JavaScript for the generated themes. If they will be generated to depend on stable9, there might be some mitigation, but I'm not sure and it's something we should at least discuss on that issue.

That said, though, Starterkit deserves to be evaluated on its own, independently of the major release cycle, and it could be added to core in any minor (even experimentally) once we've sorted through all the considerations. At that time, if appropriate, we could deprecate Classy.

Roadmap for Drupal 9

@lauriii, @Gábor Hojtsy, and I discussed what intermediate steps we could take in Drupal 9 to improve the theme maintenance situation and enable future innovation while still providing a continuous upgrade path for themes. We came up with the following requirements:

Prior to 8.8.0-beta1

1. Deprecate stable as the default base theme, and instead require themes to always specify their base theme. This is important so that in each major version themes can explicitly adopt the new stable base theme when they're ready to, without any unexpected BC break and with clear advanced warning. Since this is a new deprecation, it needed to be completed by 8.8.0-beta1.

   The issue was resolved in #3065545: Deprecate base theme fallback to Stable.
    

2. Deprecate the testing profile dependency on Classy. Many tests rely on Classy's markup. During Drupal 8's Twig and "Consensus Banana" initiatives, Classy was added as the default base theme in the testing profile, in order to save time fixing test failures where the tests had relied on theme markup. (See #2350823: Use the Classy theme in the Testing profile for the history there.) This was intended as a temporary fix until tests that were incorrectly coupled to markup could be adjusted to not rely on said markup.

   Unfortunately, there wasn't much progress made on that decoupling, so we still have had Classy as the testing profile dependency for Drupal 8's entire production lifetime. For that reason, it's too disruptive to just change the testing profile theme without warning. (Individual tests are not considered API, but test base classes and the testing API itself are, and I think the testing profile is included in that list.)

   So, we deprecated Classy as the default testing theme in #2352949: Deprecate using Classy as the default theme for the 'testing' profile. This way, module owners know to update their tests to use a different theme (or to not rely on theme markup). We have a followup in #3083275: [meta] Update tests that rely on Classy to not rely on it anymore to just stop relying on Classy markup (or ideally any markup) in general, but since individual tests aren't API, this can be done over a long time and does not need to happen before 8.8.0 or before 9.0.0.
    

Prior to 9.0.0-beta1

1. Create a stable9 theme that uses the latest markup from modules, and commit it to both 9.0.0 (as the recommended base theme) and 8.9.0 (as the replacement for stable. This is the most important step as we basically have no way of providing a continuous upgrade path for theme APIs without it. Once stable9 is available, stable can be deprecated for removal before 10.0.0. This needs to have an issue filed.
    

2. Decouple the core internal themes from Classy. The internal core themes include Seven, Bartik, and Umami, and may also soon include Claro. These themes allow their markup and CSS to change in minor releases even though Classy doesn't. Issues need to be filed for each of the themes.

   According to @lauriii, these themes will be changed to depend on Stark instead, which means they will always get the latest markup from modules. It's my understanding that this should reduce some of the overhead of fixing the theme in the module and all the themes, and also avoid the problem where sometimes a bug gets fixed in Bartik only, or Seven only if it's in an admin UI, without taking into account that all those core themes need to work properly.

   The flip side (per @lauriii) is this means that those themes will also need to be adjusted if module markup changes break them, but it sounds like it's the preferred direction overall. Plus, if these themes are decoupled from Classy before 9.0.0, it means that Classy can be deprecated cleanly once Starterkit is available.

   I recommend we start with Seven, since that's the most important to polish before 9.0.0-beta1 -- it will be the theme that's used on the most actual sites as of the day 9.0.0 ships. Once that issue is done, it can serve as a model to do the same steps with the other internal themes.
    

3. If possible, recruit maintainers for Stable (8) and Classy (as well as other themes). Right now, there aren't really active maintainers for any of the core themes, which means most of the maintenance and review burden for all of core's markup, templates, CSS, and themes is falling on @lauriii, who is our only current Frontend Framework Manager. This is not sustainable, and adding even more core themes that need to be maintained in Drupal 9 (from 4 to up to 7) will only add to the maintenance issue.

During Drupal 9's production lifetime

1. Explore the Starterkit proposal. #3050378: [meta] Replace Classy with a starterkit theme is the issue for that. #3082655: Specify the $defaultTheme property in all functional tests and #3083275: [meta] Update tests that rely on Classy to not rely on it anymore will need to be on the roadmap as must-have blockers to deprecating Classy. (From a technical debt proposal, we definitely don't want to be stuck maintaining both.)
    

2. #3050386: Allow loading CSS and JavaScript directly from templates. Like Starterkit, this seems like an independent feature/API addition that can be added in any minor release. I'm not able to evaluate the technical implications of the proposal, but this is where it fits on the roadmap as I see it. :)
    

3. Determine when stable10 should be created, and when to deprecate stable9. @lauriii would prefer to only maintain one version of Stable per major release, but we do need to provide a continuous upgrade path where the replacement is available before we deprecate, and switching to a new base theme is a time-consuming step for sites that could add a lot of technical debt to their upgrades. For this reason, my recommendation was to ship stableN and the deprecated stableN-1 in each major release, then move stableN-1 into contrib in major version N+1. This might be less overhead than it seems right now, because we're likely looking at a mid- to late-2022 release date for 10.0.0 due to other dependencies. However, we have at least a couple years to settle this policy, so I suggest we file a followup issue for it now and discuss it as a should-have for 10.0.0-beta1. (I'm a release manager; my job is to think ahead.) :)

I may have missed some things but hopefully @lauriii and others can look for anything that's missing. If we agree, we can update the IS with this plan, and proceed with the work we need to do in 9.0.x so that themes can have their own sensible backwards compatibility and continuous ugprade path policies in the future.

Thanks! And apologies it took so long to post this overview; other things kept cutting in line on my Drupal 9 readiness to-do list.

Thanks to heroic efforts by @iyyappan.govind and @Wim Leers, we've now got a green patch at #3082655-33: Specify the $defaultTheme property in all functional tests !

% egrep stark 3082655-33.patch | wc
    1429    7145   54302
% egrep classy 3082655-33.patch | wc
     178     890    6942

That's still a lot of classy, but it's better than it could be.

However, none of those classy references have a @todo comment pointing to #3083275: [meta] Update tests that rely on Classy to not rely on it anymore. Also, given 890 tests needing updated assertions, we're almost certainly going to want to turn #3083275 into a meta and split up that phase of the effort into smaller chunks (presumably by module or something).

It'd be great to get clarity from @xjm and @lauriii in that issue as to if y'all really want 890 @todo @see comments in that patch, and if so, how you want those comments to be formatted. That way we won't waste any effort on it until we're clear what else needs to happen to move that forward.

It'd also be nice to know if/how y'all would prefer to split up #3083275 so we can get that right, too. If indeed you want it by module, maybe each @see comment should point directly to the child issue of the meta? TBD. RFC. :)

Thanks!
-Derek

Re: #35:

Prior to 9.0.0-beta1
Create a stable9 theme that uses the latest markup from modules, and commit it to both 9.0.0 (as the recommended base theme) and 8.9.0 (as the replacement for stable. This is the most important step as we basically have no way of providing a continuous upgrade path for theme APIs without it. Once stable9 is available, stable can be deprecated for removal before 10.0.0. This needs to have an issue filed.

If we've got a bunch of markup-related bug fixes languishing in the queue (which we do), waiting for a time when it's "safe" to change the markup, and given that whatever state the current module markup is in will be considered "stable" for years to come when we take this step, can we also get clarity on when the right time to fix those bugs is? It'd be fantastic not to ship stable9 with a bunch of known markup bugs if we can help it. Basically, as soon as #3082655: Specify the $defaultTheme property in all functional tests lands, we can fix those bugs in the module markup, make sure their tests points to stark, leave them broken in stable + classy, and then stable9 will be more stable? Do I understand the plan?

Thanks!
-Derek

Question, can stable(8) extend from stable9 so that we don't have the duplication?

i.e. where there is no difference between stable8 and stable9, it comes from stable9

Re: #39:

Question, can stable(8) extend from stable9 so that we don't have the duplication?

I'm *really* worried about the DX + maintainability of that approach. I think of these proposed stableN themes as branches from a specific snapshot in time. If stable(8) extends stable9, every change we make between 8 and 9 has to be "backported" into stable(8). That seems really scary and error-prone. Then, when stable9 starts extending from stable10, every change between 9 and 10 has to be backported into both stable9 and stable(8). Yuck.

Disk is cheap. ;) I'd *much* rather have some duplicate templates lying around in Git, which we then *never* have to touch going forward. Once stable9 is successfully "forked" from the module templates, we're done, we never have to think about it again. Ditto stable10. These are just snapshots (presumably around the time of beta1 or something?) of the current state of the module/system templates at the time we forked them.

Meanwhile, what about stable9_0 vs. stable9_1? In #3031198: [META] Add classy_dev and minor-branch theme snapshots to allow fixing markup bugs within minor releases without front-end disruption there was much discussion of all this, and it seemed like there was general agreement from both core maintainers and front-end devs that having these snapshots for each minor branch could be really nice. Then folks can upgrade to a new stableX_Y when they're ready to benefit from any fixes/improvements since they last ported their theme. If we wait for *major* version upgrades to be doing these forks, and you're on stable9, it'd be a *long* time before you could upgrade to stable10. But maybe you have time/budget/interest in porting over to stable9_5 or something, while stable10 might still be 2 years away...

Moving to the Drupal 10 meta for the next set of requirements.

We actually want Starterkit to be added by 9.3, which will probably be the last normal "feature" minor before D10.

It mentions 9.3.0-beta1 but looks the issues are not RTBC

Changing "before 9.3.0-beta1" to "before 9.4.0-beta1" in the issue summary per the last sentence in #3246278-15: [policy] Define allowed 9.4.x/10.0.x changes.

Coming in late here as our customers depend on purchased themes to get a head start on redesigns. And they are actively planning their D10 migration. My observation is that most of the themes in the market now will break in D10 because it appears classy, et al, will go away as parent themes.
If I'm right, a theme vendor will have to refactor their theme based on an instantiation of the starterkit at that point in time. Perhaps D11 will have an improved starterkit and the D10 customer will have to update that older theme?
Pardon if this is not the right place to ask but I'm trying to give best advice for our customers, sam

If Classy is removed from core, it will be moved to contrib, so existing contrib/custom themes could keep extending it, they'll just need to declare a dependency on the Classy contrib theme.

Thank you @effulgentsia. For most of Drupal customers, it would appear that purchased themes could work if the vendors take the time to find classy in contrib to declare their dependency on it? That would presumably shake out a lot of the deprecated themes in the marketplace? That in turn would reduce the choices for them to a better-maintained subset? Is the starterkit available now, or will it be available prior to the launch of D10?

I think that even in the case that a theme vendor didn't update their composer.json file to explicitly require the Classy contrib project, that site owners could add that require statement to the site's root composer.json file, and their theme would continue to work.

StarterKit is available as of Drupal 9.3. See https://www.drupal.org/node/3206389 for details. That change record also links to the corresponding issue that worked on it, #3050384: Provide a starterkit theme in core, and the "referenced by" section in the right sidebar of that issue lists some not yet fixed issues that might matter to you if you start using it.

The policy part of this issue is done now. Implementation is happening in #3050378: [meta] Replace Classy with a starterkit theme which has its own steps + associated change records, most of which are either release blocking or should have for the Drupal 9.5 beta.

#3050386: Allow loading CSS and JavaScript directly from templates is still under discussion but can be added in a minor release and isn't closely tied to the rest of the plan.

Tried to add commit credit, a bit tricky when some of this was discussed at sprints etc. so I hope it's sufficiently accurate.

Thanks everyone.