[discussion] Theme/render system problems

AMEN!

My god yes.

Let's make it a bit clearer this is paraphrasing sentiments expressed elsewhere, and that sun's not doing a flounce.

oooh where should i start ... i can write a fucking book about all the shit thats wrong - somebody start the tape recorder ;)

Ideally the theming layer would be completely free of logic/functions and the render array stuff doesn't really help, so +1 for getting this out of our templates.

#6 sounds like #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes .

sigh @ issue title.

Actually, I'm going to go one better than a sigh, and re-title this issue to something constructive.

Then I'm also going to toss a link in here to http://drupal.org/dcoc, because this issue clearly crosses the line. Please read its contents, if you haven't before, or re-read it if you haven't read it in awhile. It embodies what makes people "stay for the community." The tone in this and other issues (as well as rants on Twitter, etc.) make me think that we could all do well with a refresher on some of the fundamental values of what brought us to participating in the community in the first place, because we need to protect that environment, both for ourselves and for future contributors.

I want to be crystal clear here that it's perfectly fine to express frustration with the way things are, and it's wonderful to gather feedback about flaws in the system. But this must be done in a way that's respectful and considerate of the work that others have done. The last thing you want to do is burn out fellow contributors who could otherwise help address some of these issues. And living by the DCOC is actually even more important the more of a well-respected contributor you are, because you are helping to set the tone for how others behave in this community.

Now. With that bit of unpleasantness out of the way, it'd be great to hear more comments in here like #6, which point at something specific and actionable that can be fixed.

Updated issue summary.

Anything I have to say is going to get me in trouble one way or another. Also, this is probably going to be a +1 fest from hell that be equally frustrating so...

/me unfollows and goes back to HTML5.

Where do we go to find designers and themers to comment on this?

I'll tweet to try to draw attention to the issue.
done. https://twitter.com/#!/kattekrab/status/151423164302360577

Themer more like to learn from REAL EXAMPLE instead READ DOCS.

a new theme:
- contains all basic templates
Yes, guys don't know how to find out html.tpl.php ..blocks.tpl.php. It demonstrates how to modify blocks / fields style..etc

- demonstrates basic usage
node--article.tpl, node--page.tpl ...etc

- how to modify $classes like variable
searching $classes in templates.php and nothing found.

- Better API Docs / More theme helper function
I don't know how to coding but I know how to copy stuff and modify it.

WP
http://codex.wordpress.org/Function_Reference/wp_list_bookmarks
Copy to theme and see what happen

Drupal
http://api.drupal.org/api/drupal/includes--theme.inc/function/theme_brea...
Copy to theme and get a lot of errors.

some theme API function are tooooo powerfully.
eg. http://api.drupal.org/api/drupal/includes--theme.inc/function/theme_table/7

Personally, I'm a developer & themer, I used some time to figure out how to use it without errors.

It would be great to get some people to help out with #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes (the issue @webchick linked to above).

I'd almost go so far as to say that it's premature to think about massive refactoring before we deal with really basic themer-experience issues like that one. (It will not solve all the world's problems, but it can make a huge difference on its own, and it's completely backportable to Drupal 7 so the benefit can be felt right away.)

Another nice feature of that issue is that it doesn't require any deep knowledge of the theme or render systems to work on; pretty much anyone who has ever modified a .tpl.php file is qualified to work on that patch. It does require a lot of elbow grease though.... I'm happy to put in some more hours myself, but the issue has been relatively inactive lately and I don't want to spend more time on it if it's not going anywhere due to lack of interest from others in actually working to fix it.

I am a big proponent of overhauling the theme system to consolidate APIs. Drupal has too many theme APIs. I don't think anyone is going to argue with this point (but please speak up if so).

I don't doubt the validity and usefulness of the current patch proposition in #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes . It is certainly worthwhile and practical to provide better documentation and outreach in general.

However, that issue is primarily reactive, that is, from a perspective of "Here's the theme system we have...how can we make this system easier to use?" It treats the symptoms and not the cause. It is resigned to the history of what has been developed.

From my point of view, the present issue here brought up by sun is primarily proactice, that is, from a perspective of "Ok, suppose we dislike the current theme system we have...what about it don't we like? Can we discover a more holistic approach to designing a theme system that will be easier to learn and easier to use? What are analogs and metrics for success?"

The theme system has grown-up piece-wise from many different ideas proposed over a series of many years and suffers in consistency and approachability. Inasmuch, this likely isn't going to be fixed shortly, or with few simple patches -- I view this as sweeping a change the scale of the Drupal 8 initiatives.

So, while issues like #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes are certainly valid, and I think it is fine to bring them up on this present issue, I don't see them as prerequisites nor dependencies of beginning to discuss the larger, systemic problems we face.

Let us at this point simply gather an inventory of the complaints and potential problems. Discovering the overarching solution will take time.

I consider myself as a themer with quite deep knowledge of how things work within the Drupal theme layer. I did express my frustration with the theme layer some time ago in this "infographic". It is not the render, hide nor php functions but it is the theme files and how things are being wrapped together (hierarchy) - eg fields, views, date and such. Sometimes it is tricky to get to the right file and override just the needed one, when there are formatters and others in the way - and finding such file does require deeper knowledge of Drupal.

Yeah, that infographic made me LOL. ;) But so describe how it would work in an ideal world.

Also, you're aware of http://drupal.org/project/devel_themer right? That's the tool I always use when I want to figure out where something on the page is coming from.

Personally, so long as there is a way for me to override anything that I need to I'm happy. With that said there are improvements that can be made (as sign so artistically pointed out).

Ultimately I agree with webchick. The only way it's going to improve to our liking is if we provide our own proposal on how it should work. I say make that recording morton, we put a list together (everyone throw in a list of their top 5 or so) and prioritize based on what would make the biggest impact. (just my 2c)

In D6, templates were simple: I prepare variables in the preprocess functions, and I print them in the templates. No function calls or data manipulation in the tpl. Great, that's elegant and easy to understand. I can look at a tpl and imagine the rendered page pretty easily.

With D7, we have a whole new layer of process() functions, so that we can do twice as much preprocessing as before -- and yet we also now have function calls and data manipulation in the tpl. :-\ As I read through a template I have to envision the data and understand what is happening to it before I can see the rendered page in my head. That means at least two passes..

I guess we need to add a postprocess() set of functions if we want squeaky-clean tpl files like in d6?

I get (I think?) that the data in preprocess() is in arrays, and the data in process() is flattened into strings(?). I don't know, really. The docs imply that. I do know that if I try to preprocess something and it's not working, I have to try processing it instead. E.g. iirc, if I want to unset the title, I have to do it in process_page() for some reason, rather than preprocess_page().

I would like to see " to enable non-developers to create themes" removed from the title of #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes -- I think it's a mistake to assume that "developers" have an easier time creating d7 themes than "non-developers." It is certainly divisive to talk that way. When I am wearing my "designer" hat, what bothers me is the ugliness in the templates, not some limitation in my ability to grok associative arrays. We aim for a certain kind of elegance in our backend code, but when we get to the front end template, we just dump whatever in there.

As I mentioned in #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes , the complexity in tpl files is bothersome if not confusing. Some variables are simple strings, some are arrays, some are arrays of arrays; some can be printed as-is, some need to go through render() first; hide() can affect some kinds of things but not others; render() is recursive (sometimes, if it needs to be), etc..

Does hide() work inside of preprocess()? Seems like I tried it once, in order to clean up my templates, but it didn't work.

If we are going to hide($content['links']) in every node template, then why have $links be inside of $content to begin with? Seems like we could eliminate hide() show() and render() altogether if we simply break up the template variables a bit more before they get to the tpl. Why can't we have <?php print $links ?> just like we have <?php print $user_picture; ?>?

Or, to go the other way, why can't I hide() the user name in $submitted?

Do people actually still do tpl-based theming now, in D7? I can hide and rearrange fields just fine using the Manage Display tab, and if they need to be wrapped into groups, Field Group module works beautifully. In practice, I've been able to avoid using hide(), show(), and render() because there are better ways to achieve those things. I imagine that's why there hasn't been a lot of activity on this thread, because themers are using tools such as Panels, Display Suite, or simply Manage Display to achieve stuff that used to be done in templates.

(Not sure if this is appropriate here since it's configuration.)

The handling of Main and Secondary menu is confusing out of the box. And then confusing when you need to set one up in your own theme. Seems like a big "GOTCHA".

Main and secondary menus can be configured in two ways:
Model 1) Set as a "feature" in the theme .info file. Then, under Structure > Menu > Settings you assign which menu is "Main" and which is "Secondary". WHOAH that is obscure.
Model 2) Place the menus as blocks. Main menu is already appearing under the disabled list of blocks, so you can place it into the appropriate region. To make a secondary menu, you add a menu, this makes a block and you can place it.

Confusion arises when using a core theme like Bartik and you want to place the main menu somewhere else, you'll get double menus. So then you need to know Main menu is also set as a theme configuration, etc.

The 2nd model, of just using "menus as block" is WAY easier to understand. People learn the block system quickly enough. Finding hidden settings in odd places is just confusing.

I'd say #19 applies to more than just blocks.

Some things you place-and-position using blocks.
Some things you place-and-position using one-off GUI settings, like Primary Menu.
Some things you place only using dedicated variables in a page.tpl.php file.

Eew.

IMO the Panels Everywhere-style "everything is a %$#^ block, just put it in a %$#@ layout!" is much more consistent and easier to learn. (OK the UI sucks, but that's a separate issue.)

I think "wait, do I even use the theme system here in the first place?" is a valid part of that question.

Thanks for posting #18 @sun. I agree with all the problems you listed.

#19 is not related to this issue and the theme/render system. That is discussed in various other issues already. See #474004: Move menu_block module functionality into core

I think @heather's feedback is totally valid and is definitely of the problem. Let's not cut this conversation off before it starts. Menu block will not solve the all theme settings issues.

I totally agree with all problems listed by @sun. I may be stating the obvious but I think the main point here — and something that I've struggled with a lot myself — is consistency. I think that regardless of the final direction that is taken it is by far most important to be consistent. Ever since starting to theme D7 sites I've found myself with the overall feeling that the whole render/theme system feels like it was only halfway implemented. Often I have thought that I had it all sussed in my head only to come across another theme/render implementation where things differ only slightly enough to make it impossible to predict outcomes. I think this is a big hinderance to uptake by non developers or even lower level developers. Even if something is complex, if it is consistent it can be learned a lot easier.

Now I realize that it's all very easy to just say 'it needs to be consistent' without any concrete examples but I think @sun has listed a sizeable portion of the issues and I will add any as I remember/come across them. Perhaps, then, if striving for consistency was just an overall goal that we can aim for and use as a yard stick to measure all future changes by we will end up with a more 'usable' result.

Node edit Form
Would theming the Node-edit form count under the present topic? Seems like it should be a simpler process. Just drop node_custom-edit into your theme folder, clear your cache, and have it work. No preprocess voodoo required.

Theming the node edit form in Drupal 7 for custom content types
http://drupal.org/node/983864#comment-3939860

Too much structural confusion for newcomers to drupal.
If I crack open a renderable, I see both #hashed and unhashed keys. Unless I have time to learn both the theme layer and dev layer, neither part make sense to me (not me, but thinking back to my early days). Not to mention, the vast and wild mixture of callbacks and datatypes that reside within the rederable. I keep watching people coming from other frameworks try to approach the Drupal theme layer the very same way they would with an MVC: attach the model or data chunks to the view, and do EVERYTHING else there. Now I understand that there are certainly some practicality issues with that, and its a very simplistic thing to say; but, shouldn't the theme layer really be that simple?

As for theme hooks, they're handy, if you know how to write them. But again, a VAST mixture of datatypes and callbacks, #hashed data and non-#hashed data. Is there any way to standardize or compartmentalize the information pushed into the renderable?
Last, but not least, its not predictable when to use render(), drupal_render(), or just plain old "print". That is all.

The issue:
I've been theming in Drupal for about three years now, and I have to say that I am using the the Drupal theming layer only in very rare cases these days. Because of its super flexibility to change/alter/render everything in too many places (template files, theme function, process, preprocess functions etc), the theming of a site quickly becomes unmaintainable.
So instead of focussing on where the Theme or Render API could be improved, I want to talk about a radically different and more simple approach.

My personal solution:
This is not a shameless promotion for Display Suite, but allow me to explain how I currently work using DS and ignoring the Drupal Theming layer, as it might show how much easier theming in Drupal core could/should be.

Let's use Nodes as an example:
- First step is to define your Content Type by adding Fields in the Drupal backend.
- Next you go to 'Manage layout' screen where you can choose a layout for your Node using one of the predefined templates, and you drag and drop your fields where you want them to be. The fact that this happens in the backend in a simple drag and drop interface is crucial. It does not make sense to let the least technical people in the development process use template files. They don't like that stuff. Choosing how content should be displayed, should be as easy as choosing which fields you want in your content.
- Surely you will want to manipulate some of the input data into more advanced examples. In that case, I simply add a custom field through the UI, where I can do simple PHP programming, or use Tokens. I agree that this is a step in the more technical direction, but at least I know exactly where I will put this field, and I know it's context, so throwing data together in one field is relatively easy.
- Typically, one Node is shown in different ways throughout one site. Drupal has View Modes for that. I set up my View Modes using DS, assign the right fields per Content Type, and done. If a developer uses Views, or custom coding or Panels, all he has to do is choose the right View Mode and done. No fiddling with fields and crying over the inconsistent HTML that creates. One centralized place, DS, decides on the output, not the module used to gather the different pieces of data.

Now this approach works for all entities, so that covers a large part of Drupal content. I only use one Module, one interface, one consistent approach. The benefits towards consistent HTML and maintainability are huge. I don't need to think about the render API, or template files etc.

The solution
My idea would be to take it back a step and try too keep things simple. Defining the layout of your content should be as easy as defining the structure of your data. Give themers a nice drag and drop UI to get the 90% right, and make it as easy to get to the last 10% using custom fields for example.
If all modules expose their output to this system, then there is no need for a zillion functions where you can get in and alter some stuff.

Hopefully this will inspire some people to go from this theory to a technical solution.

I think part of the problem is that in (partially?) merging the FormAPI render system and the theme system we've mashed together two systems that may have superficially looked similar but really aren't.

Take the form API array system. It's an array form elements where each element is a sibling. There are different types of element and functionally they are siblings too: they are each a form control (the exception being radios and checkboxes which expand, but that's more to do with the HTML for forms being a bit rubbish).

There's also nesting in the array but that's only with either fieldset elements or un-typed ones: a different class of elements used specifically for grouping. And it gets more complicated when you get to #process and #after_build and all that, but that's advanced stuff that you're free to ignore. What I'm getting at here is that the basics of FormAPI can be explained in about 15 minutes and a novice can build a simple form without much trouble.

Meanwhile, on Drupal 6, the theme system was a reasonably amiable ragtag bunch of templates and functions. There was a mix of theme components for pure HTML elements (theme_table, theme_item_list), for Drupalisms like theme_links, and for Drupal page components like page, node, and block. An entire Drupal page is made up of different tiers of those, roughly interspersed with pure HTML either in the theme functions or templates. It's not a clean or pure system, and when contrib modules come into the mix it gets even more tangled (CCK and Views!) but it worked reasonably well.

Now if we wanted to have a complete representation of the page as an array to render -- and bear in mind I am not convinced that we do -- the totality of that needs to be rewritten from scratch. You need to have a structural hierarchy of elements to make the page that's pretty much a representation of the HTML DOM, with some Drupal flavouring. It would go something like this:

-html
-- page
--- region
---- block
----- block heading
----- block div
------ who knows what inside the block

Every piece of HTML needs to be a renderable element, not just the ones like TABLE and UL that we bothered to write theme functions for because they're fiddly.

That would be a monster undertaking, and the result a monster in itself. Forms benefit from this approach because they can be abstracted to this sort of structure, but I don't think that the whole page of HTML does in a way where the benefits outweigh the costs.

What @jyve said about DS: I use a similar workflow. All because of the enormous benefits for pleasing html and maintainability of the output that is provided.
In that workflow it's really easy for anyone involved to know where the output is coming from.
When using nothing but template files and functions, things tend to get messy and maintainability is a lot harder.

I think this raises a larger point in that theming for Drupal is perhaps overdue a complete rethink.

From an integration point of view - the current theming model forces themers to bend the system to it's needs rather than bend to the content/design it is trying to render. More time is spent overriding Drupal output than actually building the sites to begin with.

This overhead is not only undesirable from a production point of view but enforces a number of limitations on the actual use of the software due to strict guidelines on markup for accessibility or a myriad of other use cases. Point being - Drupal should not touch the markup at all - it should fit around it rather than the other way round . Data and presentation need complete abstraction.

Responding to some of sun's points in #18:

3) Something like http://drupal.org/node/650016 might help, assuming the workflow actually made sense.
The lack of an obvious place for the various #types causes a lot confusion for me, due to the #process/#pre_render callbacks and added #theme/#theme_wrappers.
4) This is probably the most confusing part of the theme layer to me. I understand the results of the distinction, but not the reason for it.
5) Same as 3).
6) This is similar to 2), in that the real answer is subjective, because AFAIK it's never actually been discussed.
7) and 8) How can we make this sane? That this is still broken in core makes it impossible to do correctly in contrib.

As someone so very new to learning theming in Drupal, what I can say is tracking down elements that are crazy long just to change one tiny thing is migraine -inducing. I would love less code spit out by whatever it is spitting it out, whether it turns out to be modules or system related.

*running off to my hidy-hole now*

I don't get it. The problem. Everybody seems to agree there's something wrong (that's almost a quote), but nobody can convince me how (or what) it's wrong.

Drupal theming can be very, very hard. Ofcourse! Drupal is huge and it has to be. It's also very complex and has to be. It caters to all kinds of websites and people. That everything's possible should be a good thing!

I can't believe I'm rooting for Drupal, but apparently I am. All this time discussing about what's wrong could be spent on

* extending the docs
* removing all D5 docs and separate D6 and D7
* more examples: more simple examples, more advanced examples, more exotic examples
* explain the relevant theming features very well (and the hardly used somewhere else)

I'm a themer and I don't know half of the theming system. (Apparently not even close.) And that doesn't matter. I don't have a problem "bending it to my will".

Not only is it difficult (apparently) to define the problem, it will be much more impossible to fix it. Drupal 8 won't be simpler. You can try, but it won't.

That's my 2c and now I leave it to the pro's.

@okwari's comment is the first useful. I can agree with that. More readable var dumps.

I agree a lot with #33 by @rudiedirkx, a lot of the bitching about the theme system is just that, not an indication of problems.

My impression is that many people, including the designer types that bitch the most about markup, come from a background of cottage industry hand crafted small sites with no interactive features where it's both possible and realistic to write markup from scratch – keeping it very neat, light and tidy. Drupal in contrast is industrial mass produced production line markup, it's ugly and complicated. But the sites we build are too complicated and feature rich to keep writing markup from scratch. It may look horrible but it works and is practical.

I think most of my theming issue's actually derive from contrib modules. Core doesn't provide a feature so module-x adds that functionality but has to do some hairy stuff to make it work. You then find the undocumented hairy stuff gets in your way when you try to theme it. The same can be said for some of the more complex base themes. How do you get round this? Core can't do everything. Fiddling with the core theme system may not help.

Better docs. Yes. Apart from the hide/show stuff I have no idea how useful the Render system is. Just a suspicion.
$mostly['it']['just'][0]['seems']['to']['make']['for']['very']['long']['variable']['statements'] = 'sigh...';
I think the syntax for manipulating arrays is actually quite tricky in php, seems to trip me up a lot, practical examples are gold.

Better dev tools. @webchick mentioned theme_developer module, personal experience is that it breaks layout and javascript on a regular basis and thus is kept disabled unless its really needed. The end result is either a long trip around admin screens or a trip to the command line and drush to enable/disable it as required. That limits its usefulness.

The comments about unifying the evolved branches of theming: variable vs block vs gui settings are spot on, we've obviously been evolving various systems over time and bringing back some consistency would be good.

But mostly I think people bitch because they have to work, and there is no way around that.

adrinux: It's not just a question of "mass produced production line markup". I've not done hand-crafted markup for serious in a decade, I'm a module developer not a theme developer, and I hate the markup Drupal 7 produces. As a PHP developer, I have no idea how I'm supposed to use theme(), #theme, #theme_wrapper[s], etc. And no amount of documentation is going to make that obvious.

I might have agreed with you in Drupal 6 to an extent about "generic code requires generic markup, deal". But Drupal 7 goes too far with that, and now that we've dropped support for IE 6 and IE 7, which are what required a lot of that extra markup, we have an opportunity to clean things up all over the theme system. Let's do.

In response to @Crell, also:
Lets take, for instance, trying to theme a menu I've placed in a region using the system block manager.

First off, the general output will look like something along these lines:

<div id="some-block-delta" class="some-block some-block-region some-block-region-menu-some-menu">
  <div class="block-inner">
  <h2>Menu Block Title</h2>
  <div class="content">
    <div class="pretty-sure-theres-an-inner-class-somewhere-along-here">
    <div class="menu menu-id">
       <ul class="menu menu-id">
           <li class="menu-leaf active first">
             <a href="http://www.google.com" class="menu-link active">Holy crap, I finally got a link.</a>
           </li>
           <li class="menu-leaf menu-parent">
             <a href="http://www.google.com">Ooooh, a link!</a>
             <ul class="menu">
                <li class="menu-leaf">
                   <a href="http://yahoo.com">Is this complex enough?</a>
                </li>
             </ul>
           </li>
           <li class="menu-leaf last">
             <a href="http://www.google.com">Yay, another link.</a>
           </li>
        </ul>
    </div>
    </div>
  </div>
</div>
</div>

What ideal code looks like to a front-end developer is something like this:

  <h2 class="some-heading">Menu Block Title</h2>
  <ul class="menu-id">
      <li class="active first">
        <a href="http://www.google.com">Holy crap, I finally got a link.</a>
      </li>
      <li>
          <a href="http://www.google.com">Ooooh, a link!</a>
          <ul class="menu">
              <li class="first last">
                  <a href="http://yahoo.com">Is this complex enough?</a>
              </li>
          </ul>
     </li>
     <li class="last">
        <a href="http://www.google.com">Yay, another link.</a>
    </li>
  </ul>

And here's what it takes to get there:
* Make a new copy of block.tpl.php, throw it into my theme, and rename it to block--menu--menu-id.tpl.php (where menu-id is the machine name of the menu I want this for).
EDIT: I missed a step!
* Clean out the unwanted cruft from the block--menu--menu-id.tpl.php.
* In my theme's template.php, I need:

// ... other code

function mytheme_menu_tree__some_menu($variables) {
  return '<ul class="i-now-have-control-of-the-wrapper">' . $variables['#children'] . '</ul>';
}

function mytheme_menu_link__some_menu($variables) {
  // But, if you goof this part up, or futz with something you shouldn't,
  // It'll either go totally haywire, or break everything you know about the menu.
  $element = $variables['element'];
  $sub_menu = '';

  if ($element['#below']) {
    $sub_menu = drupal_render($element['#below']);
  }
  $output = l($element['#title'], $element['#href'], $element['#localized_options']);
  return '<li' . drupal_attributes($element['#attributes']) . '>' . $output . $sub_menu . "</li>\n";
}

// ... [And you get to repeat this step for EVERY MENU you want, blech...]

In order to just get nice, clean markup that's easier to debug, and easier for the browser to parse, we get to repeat this cycle for EVERY menu we want.
I'm not trying to say the theme system is horrid, unreliable, or terrible, its just a much more complicated than it probably needs to be.
In our company, its no wonder Front-End developers either don't want to, or don't have the time to understand Drupal. Sure, its flexible, but eeeshhh...

But mostly I think people bitch because they have to work, and there is no way around that.

Wow, what an insult. Pfft.

This.

I think in general, a concerted effort really needs to be made for the markup out of Core (and ideally Contrib as well, certainly Views and Panels, but a discussion for a different place) to be much cleaner. What I've seen of the HTML5 Initiative stuff, this has been happening a great deal, but this needs to happen all over the place.

Yes.

as someone who has to teach people this stuff - #37 - spot on. Now let me bikeshed.

The problem with the theming layer is that the flexibility has led to a lack of discoverability. The discoverability is what is killing new people. It's not that it's so hard once you figure out you need to create a function called "mytheme_menu_tree__some_menu", but *How the hell would you figure that out without digging into theme.inc, menu.inc, menu.module, etc. etc.*

Now, since I know what is in those functions, and why things are the way they are, I also know that there are no easy solutions. This stuff used to be simpler.

You used to just have to implement a couple theme functions for a menu (the list itself and each link) But then people had preg_replace in their theme functions too, which is terribly brittle because the flexibility wasn't there.

So if you have a user base which demands flexibility (which requires complexity) and a user base which demands simplicity, I think focusing on discoverability is the key. This means that maybe menu (and others) continue to suck w.r.t the various wrappers and granular overrides, but we provide people who are new tools like devel_themer, but better that will write some code for you. Or... what? better docs which demonstrate it?

I really don't know, but I do know that we got where we are because the old system was hacky and inflexible so saying we need to get to "clean" markup and "less theming functions" is probably not the panacea we're looking for.

The problem with the theming layer is that the flexibility has led to a lack of discoverability.

Wonderfully put. This is the exact problem I've been tounge-tied about when trying to get new folks ready to interact with Drupal.

This means that maybe menu (and others) continue to suck w.r.t the various wrappers and granular overrides, but we provide people who are new tools like devel_themer ... or better docs which demonstrate it?

I think the latter might be enough to help. While devel_themer is a quick and masterful solution to finding out exactly what needs to be added, it still doesn't totally help a true front-end developer. How many come to Drupal thinking "Hey, this is easy to build stuff with." and walk off with "My head hurts... Gosh, that's a lot of markup; maybe I'll just use a pre-built theme."?

I think the latter would be the better solution, and while writing this post, actually had an "aha" moment.
Maybe, and correct me if I'm crazy, but what about a "theme kickstarter" folder? It would contain nothing more than documentation, -dist versions of the .info file, and exact clones of the system provided templates? While the user can't install this theme directly, it becomes clearer that these are the steps needed to take control of Drupal's output. I realize "Stark" was introduced to D7, but even then, I keep seeing people "hack core" of it, and then turn around and go, "I'm doOOOonnne!". The .tpl.php documentation is EXCELLENT, IMHO, but, doesn't mean as much to the front-end developer.

As another thought, maybe we suggest a strategy for planning out the necessary templates for site builders to set in place, and "scaffold" the theme before handing off the site to be "themed"?

We've actually changed the normal FE->BE->REVIEW process to BE->FE->REVIEW specifically for Drupal because of some of these issues. Thoughts?

I do know that we got where we are because the old system was hacky and inflexible so saying we need to get to "clean" markup and "less theming functions" is probably not the panacea we're looking for.

Understood. But, what's needed to bridge the gap between a Front-End developer and a Back-end developer? Many devshops (not necessarily Drupal specific) don't have the manpower or time available to get someone fully familar with what needs to be in a theme to get exactly the results a client wants. The recent trend of the web has certainly suggested that a Front-Ender needs to know how to code (in some small amounts) for the system they'll be working on, or get someone to help them.

Shameless promotion of my first d.o issue #1295958: Arguments to theme as renderable array in a render array. Allow better nesting of html element. :3, basically what sun said #18 point 7.

Theme functions is oblivious about drupal renderable php array, maybe it's for the good, but making renderable php array works around it is the problem.

wjasper's aha moment is excellent.

... While the user can't install this theme directly, it becomes clearer that these are the steps needed to take control of Drupal's output.

If a new themer starts with that, all relevant options will be explained.

@Jacine

Wow, what an insult. Pfft.

When I say 'bitching' I'm talking about people that just complain. NOT those making constructive criticism. I used to think like them, maybe I've just got too used to drupal markup...

Better documentation and examples are band-aid solutions, and address the symptoms, not the cause. This issue is about the cause. This issue is not about people "bitching" who don't read the docs. I don't consider the items in #18 to be "bitching." Let's focus.

I would like to applaud @wjaspers comment #37 that actually shows *what it takes* to gain control of markup in Drupal. I had written a blog post a few months ago demonstrating a similar exercise. It sat unpublished for a while, I didn't really have a thesis to follow-up the example. Now that more people are engaged with the topic, it's probably ripe. So, if you'd like, read on: http://c4rl.ws/blog/node/2.

I would love to hear some feedback from people that have extensively used other systems that they think handle this problem area well and see what we could borrow.

At this point I feel like my brain is so deep into Drupal's system that I can't even see the forest for the trees, the only thing that even catches my attention are some of the big differences between 6 and 7.

So what systems are the leaders here, I've heard positive feedback about Expression Engine, anything else that makes writing great front end code enjoyable and straightforward?

While preparing Frontend Training for Drupal newbies, I thought of some things:

1. The main reason people need to use the theme/render system is because they want to clean up crappy default HTML or add custom classes. With the fantastic work that is being done for Drupal 8 in terms of HTML and CSS cleanup, this should be far less of an issue.

2. So let's say default markup in Drupal is fine. Adding some classes or id's here or there should be enough in that case, right? Since everything is moving to Entities, an improved Field UI could play a big role. A Field UI where it is easy to make minor changes to the HTML output (like adding classes), where you can easily select how the field content is outputed (a link, a list etc) and where you can use som predefined layouts (compare with Display Suite). This would make all those template files and their suggestions obsolete, since you now have a UI to manipulate your output.

3. The current theming system does make sense somehow, once you get to know it. The only difficulty being that it can do too much. Having theming functions and templates, and all functions linked to preparing those template files, is just overkill. With points 1 and 2 in place, could we actually simplify this by kicking out the whole system for template files (except for maybe page.tpl.php), and simply keep a good list of theming functions?

A comment I made in IRC earlier that I think is relevant here:

"The goal is not to make Drupal easy to use; it's to make Drupal easy to learn."

Complex problems often require complex solutions that take effort to use. That's OK, but learning how to use the tools should be as easy as possible. The current theme/render system fails on both counts, of course.

I'm not one for elegant prose or well constructed arguments, but here are a selection of my thoughts on theming.

tl;dr Basically, what @jyve was outlining in #27 and #49: full control of HTML from the UI, no more template files.

The Problem

Fantastic front-end developers, who know their HTML and CSS inside out find Drupal's theme system impossible to use.

Why is this a problem? Why are they trying to override the default markup, anyway?

This may seem obvious to some but isn't to others.
Some developers seem to think the front-end folks should just apply their CSS stuff to the markup that Drupal outputs. After all, they've gone to the trouble of adding loads of divs and classes…

Well, we need control over the markup for a variety of reasons (an incomplete list):

• New developments in the web. HTML5, responsive design, focus on mobile.
• Ease of debugging. (#36 @crell Ironically, I've solved many an IE6 or IE7 issue by *removing* markup…)
• Ease of maintenance. Cleaner code is easier to read. When you open FireBug and are confronted with a seemingly end mass of nested divs, you know your day isn't going to go as well as it could. When working in large teams, keeping things clean and simple makes it much easier to spot bugs that may have been introduced by others.
• The unknown thing that the client is going to request in two years time.
• Sometimes, for expediency, you just need to change the markup.

If the theme layer is going to do something useful, such as turning an array of items into an unordered list, then it should do just that and no more. No assumption that it needs a wrapping div.

A block assumes I want its title to be a h2. Hmmm. No, block system, I think *I* should decide, thanks. And no, I don't want to override or install another module.

"You can just override it."

When I've got my developer hat on, I really like Drupal 7's theme system. Working on complex sites, handling data from different sources, writing custom theme functions and outputting data as render arrays is a fantastic developer experience.

Most front-end theming tasks are not like this, however. Most involve The Dreaded Override.

As @jacine demonstrated above, and @sign's infographic so clearly explained, digging out the right picec of code to override is a horrible experience. Yes, you can use devel_themer, but then what do you end up with? A gazillion theme functions or template files.

Constantly having to override is not only tedious, but just feels wrong. It makes a fairly simple yet important task difficult. I really believe we should give the option for the other approach: to switch off all default markup, allowing the themer to build up from there (a more natural approach for a front-end developer). This may sound ridiculous, but we need Drupal to be able to output no HTML if that's what the front-end developer chooses.

In practice, we really just want some great default markup that makes sense a lot of the time.

Template files are a problem

They can be incredibly confusing. I've seen far too many themes with dozens of .tpl files, just to make little adjustments to markup. It can be a horrible mess to wade through.
There is also a big disconnect between what might be being set in the UI and what's being overridden in the theme.
They encourage bad habits. If you've ever found SQL statements in a theme, you'll know what I mean.

(Also, you could argue they're slow. theme_field() has a nice warning about using field.tpl.php.)

Overriding using theme functions not quite as bad: at least you can keep all those overrides in just one (hopefully small) file.

Onwards to great markup

We want to enable front-end developers to structure the output with ease.
I think @jyve is absolutely right: this needs to be controlled from the UI.

Out content is precious. We need to carefully wrap in in well-chosen HTML. Currently Drupal squeezes our content into one of many potential template files.

So, instead of this:
content --> theme layer --> markup/css/js

Something more like this:
content --> markup --> theme layer --> css/js

The theme layer shouldn't really be doing more than assembling the pieces together. It shouldn't be deciding how to markup the output. Humans should be deciding that.

A lot of the pieces that can help solve this are already floating around the Drupalsphere; view modes, Display Suite and Fences being good examples. The progress made in Views 3 is fantastic and really helps (an improvement there would be to have the wrapping markup off by default).

Maybe some form of panels in core could really benefit controlling overall page HTML. But then, we could possibly work out something with the block system, too.

We need to think through a consistent approach and UI, so an experienced front-end developer can start playing around with Drupal and find that he/she can easily control the markup to output what's required by the project. I think we can do this.

And finally…

Let's not forget what we are trying to do here. Fundamentally, our ultimate aim is to output sensibly marked-up content that is well structured, accessible and visually appealing across a wide range of devices. We need to make sure that our tool of choice (Drupal, yay!) is making that as easy as possible.

Just going a bit in overdrive here, but some use cases building further on #49 and #51:

1. If we have a good Field UI, then Views should be able to use that one to manag its output, removing the need for those wicked template files.

2. If a contrib module wants to give power to the users to manage the layout, they simply make their output an entity. If needed, some theming functions can be used, but template files are no longer needed.

3. There is a decent set of theming functions such as theme_item_list, theme_table etc which should stick to the very basics in HTML output. Field UI should offer these as output options. The combination of clean HTML defined by the theming function, and the ability to add classes through Field UI should do the trick.

Are you suggesting going back to the dark days of Contemplate module and storing the HTML mark-up in the database? If so - yuk.

@budda, I've had the same feedback here from dev colleagues.

I don't see it as a step back. Replacing the template system with a flexible Field UI, is a step forward for Drupal newbies and all Frontend Developers in general. The learning curve drops hugely in that case.
Is it that bad to store HTML in the database? Better than having a pile of unmaintainable template files? If so, exporting to code would be a solution, right?

Anyway, I'll be listening to other proposals for now, and shut up :)

@budda Yes, I think so!

I admit it feels odd saying this.

Use fields, content types, views, panels? Those are stored in the db, too. The db isn't such a terrible place to store things.

I think I've totally shifted my opinion on this in the past few months. I no longer view markup as code. Really, it's just metadata for content. In the use-case of HTML output (just one of the potential use-cases of Drupal output), it describes what the content is.

Feel free to expand on "Yuk" :-)

My fear of removing the template system is a fairly simple one: the next question is "how do I lay out my page without a template system".

To expand on that, having wrapper and class/id options in Field UI settings is fine and dandy, kind of like what the Fences Module does now, but that will only affect fields, not the entities that wrap the fields (i.e. wrapping for nodes). Well let's just add that displays options as well. Great, but what if I want ti display this content wrapped in an <article> here and an <aside> there? Well now it sounds like we're talking about building Display Suite into Core. What if I want to add regions to uniformly place blocks in the same place across my site? I don't have a template file to build a region, so I'm going to need a UI component to do page layout. That sounds like Panels in Core. What if I want to test and tweak the layout for an anonymous user? Now, instead of being able to use Drush and a text editor to test and make quick tweaks, I need to have two different browsers open, one logged in to tweak, one logged out to look. What if I decide I really need to add a

The db isn't such a terrible place to store things.

When front-end devs start using a CMS, they are used to seeing templates. This was my experience and I think is common. I'm a believer in templates.

Drupal has never been strictly MVC, but convolving these components any more creates harder problems to solve. The fact that there isn't clear separation on how to achieve a task is the reason we've ended up with the problems we have. (Do I override a template? hide()/render()? Use preprocess? Use process? Alter hook with new theme function? Field formatter?)

The current state of Field UI needs love, but it's doubtful any browser-based UI could ever replace what can be accomplished via code. We don't want to get away from code, we just want the code to be more consistent and approachable. Solving those problems will be better for both front-end developers and back-end developers.

This is something I've talked about before: http://www.garfieldtech.com/blog/drupal-priorities

There is a natural, inevitable, unavoidable tension between "design by configuration" and "design by theme (templates)". As sun notes in #58, design by configuration results in godaful templates, and design by theme means you have to remove, or silently break, configuration options in the UI. That is an intractable problem.

To be clear: Both design by configuration and design by theming are viable approaches! There is nothing intrinsically wrong with either mental model... as long as you pick one. You must pick one to prioritize.

No matter what we do, one of those two approaches will be a second class citizen. We have to simply accept that. One of the big problems right now is that we don't, and as a result they are both third class citizens, because we end up doing both of them badly.

I don't have a really strong opinion as to which one should be our primary focus. I do feel strongly that we need to pick one to be primary and one to be secondary, accept the implications of that, and be consistent with that decision.

And oh yeah, Contemplate-- :-)

#58 and #59 are a perfect summary of the dilemma we've been having for years within CCK and Field API. Tags configuration in the UI means ridiculously cryptic (and inefficient) templates, and overriding in a custom template means silently neutering the stuff configured in the UI ("why don't my changes get applied ? Oh, right, because I have a custom template that ignores the stuff I see in the UI".)

Stating that front end people prefer clicking through a UI rather than writing HTML in template files is a bet I've always been hesitant to make.

Possible food for thought :
- Maybe it's not an all-or-nothing issue. Would it be correct to state that keeping control over entity templates (mess with node.tpl, node-[node_type].tpl et al) is precious, whereas having to deal with field.tpl, field-[field_name].tpl to change field markup is seen as tedious ?
- Fences takes an interesting approach : ships with a collection of field templates (you can provide your own in your custom theme), and lets you configure in the UI which field uses which template.
Good mix of "html stays in templates" and "configure the output through the UI".
Con: that's an uncommon use of the theme system, arguably a hack. What this is really about is : field wrappers (div, ul/li, ol/li, comma separated...) as a new type of pluggable components, next (and orthogonal) to formatters (that take care of formatting the values within the wrapper).

I have to agree that the 'everything in the UI' is not very realistic, but I wanted to take a stand so that people could at least start to consider pro's and con's.

As I said before, the current theming system does make sence once you know it, although it might be showing off too much. I believe that yched's point is very valid in this case: Not every detail should be configurable. Some things will just be the way they are, and if default HTML is good, this should not be an issue.
The big issue is here is where you will want to strart removing flexibility and complexity.

I like the approach from Views or the Block Class modules: simply allow the user to add one custom class. Allow the very basics to be handled in the UI, and use the flexible template system for more control.
Having ten template files that only have one class as a difference is propably the most frustrating part of all those templates.

My impression is that many people, including the designer types that bitch the most about markup, come from a background of cottage industry hand crafted small sites with no interactive features where it's both possible and realistic to write markup from scratch – keeping it very neat, light and tidy. Drupal in contrast is industrial mass produced production line markup, it's ugly and complicated. But the sites we build are too complicated and feature rich to keep writing markup from scratch. It may look horrible but it works and is practical.

I believe that clean, light and semantic markup has its place in web-development and Drupal should not make us think it is not possible to achieve it without breaking the complex functionality of the very complex sites we are building for our clients. We are usually never creating HTML from the scratch, we are using frameworks for HTML and CSS already (be it custom or 3rd party like we are using base themes,...).

There are people that don't want to and don't care about the markup like some other people don't care about PHP and Drupal coding standards either they don't really care or they don't have the skills. For these the default output is fine. Then there are people that believe in standards, pushing the web forward and treating the front-end as a craft thus spending a lot of time on it.

My last experiment (just finished), bending Drupal's output to what was in static HTML5 build, went better than expected. The build was done by colleague who is not liking Drupal because he doesn't know where the divs and other wrappers/markup is coming from and usually ended up being frustrated (as we have seen in several responses above). The CSS wasn't adapted to Drupal classes at all.

I myself was frustrated few times, when it came to removing nested divs within fields, etc... (mainly in views module). Generally I would end up with a custom field template for each of the views fields, etc... This is where Fences module helped, but not for selecting the right wrapper from selectbox but setting the field settings to not wrap it at all (clean). I do not like the idea setting the wrappers, output, etc... within UI and storing it in the database, this makes it for front-end developer un-readable, un-trackable and at some point "dirty", but I do like the idea not to wrap it in a default mark-up, so I can work just with the variables within the template.

We have used several custom template hook suggestions in preprocess when it came to specific blocks positions (we are using context). So sometimes the output of the template was wrapped in aside, sometimes in article,... all of these things have its place in the design and as well the design has been built for CMS, so most of it is predictable in terms of what the structure behaviour will be.

Inconsistency in contrib and core. While lets say the node.tpl.php files are using render in most cases (fields), views requires you to know how to handle objects. Eg. printing the output of a field within fields template will look like print $fields['field_name']->content (after applying Fences not to show any markup). I am specifically talking about the fields views template, because I was doing the formatting there rather than outputting the format per field. You can always have a fallback for the newly added fields (thanks to render too) so you won't loose any flexibility.

But every system have its rules, you usually won't give your client full set of tools when it comes to really robust and complex solutions and I believe you shouldn't, instead create set of rules what they can do that is useful for their business and run some of the more complex changes through developers as a new phase of development.

So building a theme for your client where the needs are specific is possible (without loosing flexibility), but when it comes to building a contrib theme, where you want to have this flexibility for all of the modules out there, that is a problem. And I don't think we will be able to fix this if we don't have the specific installation profiles/cases.

Anyways to wrap it up:

I am up for having some level of control within UI to say:
☑ Remove markup (or even for specific cases, like just the fields,...)
but at some point this could be a configuration within .info file which templates should not be used. Certainly not for controlling what markup will be there per build mode.

I would really like to get just the raw value of a field to work with, no markup included, so I can do some display logic within template.

Finding the right theme suggestions (be it a function or template file) is a pain, but managable, if you know how to display vars['theme_hook_suggestions'] using devel in preprocess_xxx. It is more painful when it comes to contrib modules that are using rendered fields (thus using fields tpls,...) then it is hard to find.

I am also not worried about PHP within template files. People will learn how to work with arrays, even objects, or how to run loops, etc... (look at WP theming) but the problem is that too many things are involved during rendering that are not easy to track (like markup from other formatters).

And in this last experiment, I didn't have to use page_alter or to work with render arrays more in depth at all. Although as a module developer I know it is not consistent and broken/confusing/not implemented even in core (eg. forum module). And devs are struggling when to use render arrays, what elements are, etc...

The biggest problem with Drupals theming system is the way markup is added. There are tons of classes and wrappers that are "good to have" but really shouldn't be there unless it's wanted.

Forms in Drupal are notoriously difficult to theme due to the fact that they are overly complex in div wraps and such with overlapping classes (.form-item is the perfect example of this).

Rendering, say, the value of a field, instead of rendering the value + label + wrapper + wrapper for label + wrapper + for value + wrapper for both is actually difficult to do - but it should be the default behavior. I'm not saying that markup should be written from scratch all the time. Some things like odd/even or first/last classes are incredibly useful.

Field templates are also very complex, and editing them in order to improve/alter markup is both tedious and dangerous (unless you know exactly what you're doing).

As a non-coder, this is just my experience, and I hope that it proves useful to get my opinion

Having core output only the value of fields should be optional for those that want (to go through the trouble) to write their wrapping tags/classes from scratch. It shouldn't be the default though.

As it is now, for doing basic/starter theming, we ask people to take a base theme and tweak/override its css. If we default to not outputting any HTML markup, then we're asking them to do additional work. Leave aside the fact that there are *a lot* of contrib modules out there that depend on some of that markup to do their magic - we'll just break everything and ask developers to do additional work too? ...that's at the very best (make their module's features impossible to happen at worse).

So sure, we can introduce some "Theming mode" (to be in tandem with the "Maintenance mode") where everything is output as pure values, but as I said: it should not be the default mode.

As a non-coder, this is just my experience, and I hope that it proves useful to get my opinion.

You say that now, but you do carry almost 5 years of drupal experience along. How would that seem to you if it was the default when you started??

This issue thread is an incredible revelatory read, thanks wjaspers for pointing me to it. I don't have much to add but a few questions:

Should we be considering what we have to do in order unify the render() and template approaches? I've love to make / find a diagram of the workflow of a themed item's lifecycle. Perhaps seeing this diagram would help the rest of us understand the competing actions of these two approaches.

I think we can agree that there are both benefits to the template approach and the UI approach. Indeed they are for different audiences.

I believe this is not a question of one-vs-the-other, but rather how they fall into the hierarchy of the execution stack, and their relationship to core.

To me, a system like render() seems to be heavily influenced from the UI-driven approach -- we have a setting, and we simply "render" it. I don't necessarily have a problem with this approach, but I find it problematic that the assumption and emphasis on render API seems so deeply baked into how Drupal works.

Before we even hit the theme layer, we have this big render array, that knocks on the door and says "Hi, I'm a Render Array, 20 levels deep, are you ready to get down and theme?" and I often answer back, "Actually, Render Array, I'm glad you're here, but I don't really need all of these parameters, and if I wanted them, I'd use a preprocessor to get them, you know?"

So, perhaps instead of the process

[Modules assembling render arrays] => [Theme]

We could break this up a bit, so that it is more pluggable.

[Modules] => [Rendering layer] => [Theme]

The difference here is subtle -- modules will be assembling data -- but it should be up to the rendering layer to decide how that data should be constructed in order to be presented. So, functions like node_view() would have #less #of #this #stuff #in #them.

This also prevents the very real problem (brought up to be by msonnabaum) that having render() in templates themselves means we are bound to PHP as a template language.

I was trying to get a Twig theme engine working in a sandbox project and was having major architectural problems. The Drupal template system is quite difficult to work with, and could definitely use some unravelling of its spaghetti.

@RobLoach, can you elaborate what architectural problems you had?

• #1158090: Separate usage of hide(), render(), and other template features from print statements to enable non-developers to create themes
• #304486: Theme Engines as Modules
• #330159: Allow module template files to be coded using any templating engine: don't force PHPTemplate
• #304540: Disable themes when theme engine or base theme aren't available
• #1336598: Look to Joomla! for ideas to improve core theme engine
• #474684: Allow themes to declare dependencies on modules

There was one other issue to support multiple theme engines somewhere, but I'm not quite sure where it went.

@Robloach I remember Rene Bakx was working on a twig engine for D6 https://github.com/renebakx/twig-for-drupal

@Klonos:

You say that now, but you do carry almost 5 years of drupal experience along. How would that seem to you if it was the default when you started??

A very good point, and I'll try my best to explain where I'm coming from. When I started out with theming Drupal, I loved that stuff just worked out of the box. I loved that I could browse for modules and themes and just enable stuff. I didn't have to do anything but override css to get things the way I wanted.

That lasted for about one week.

As soon as I wanted to do a minor change I often ended up struggling for weeks to find the right php snippet that did what I wanted. It wasn't uncommon either, that I found loads of threads on the forums or in groups that had the same problems. Themers, new to Drupal, looking for ways to actually do some theming. Yes, you could definitely argue that I should have learned PHP instead of googling for snippets, and that is true. But should themers really have to know PHP and the whole rendering engine just to be able to produce slick looking websites? No, I don't think so.

Discussing these sort of issues with people before the initial experience of overriding the theme system of a person comfortable with HTML is as follows:

1. Open up the theme and find template files (page.tpl, node.tpl)
2. Open up the files and find them to look relatively un-intimating.
3. Make changes necessary until they come to something like $breadcrumb and see that this variable has HTML in it that they need to override.
4. They look up the docs find out about theme functions and override theme_breadcrumb.

Of course between 3 and 4 there is a lot of confusing, many errors and perhaps even a little cursing :) I believe the jump from editing templates that print out data to going further down the rabbit whole of theming to override variables printed out in templates as HTML is where we have a few problems. We are taking people away from their comfort zone of HTML to somewhere uncomfortable and confusing. What about instead of this we had something like this when dealing with breadcrumbs (using twig here because it has a nice syntax)

<ul id="breadcrumb">
   {% for link in breadcrumbs %}
      <li><a href="{{ link.href }}">{{ link.title }}</a></li>
   {% endfor %}
</ul>

Now there are various ways we could improve upon this but as a general principle I believe it to be a good one. We should try as hard as we can to have templates that contain only two thing: HTML and data. And we should try to have templates more often.

Also as a more general point discussions of themers and what they want sometimes fail because 'themers' cover such a broad variety of people. A while ago there were some user personans created for themers which try to capture the various types of people and their various needs. Bearing these differences in mind might be useful. http://groups.drupal.org/node/90849

@edward_or
Themers would die for that markup :)

OOOH! I just thought of the ultimate clicky theming system.

Here is another go at it, just to keep ideas flowing.

In this approach, the focus is on strict separation between logic and output through clean template files.

1. Power and flexibility is great, but let's keep all logic out of the template files. There are enough other places to prepare the variables. Al that should be in the template files, should be something like this (fictive template language):

<div class="[classes]">
  <h2>[title]</h2>
  [content]
  [read-more]
</div>

No render engine, and no PHP logic. This offers less flexibility, but then again, flexibility is not the only goal here.

2. If more in depth changes need to be done, then theming functions and preprocess/process functions can still be used. I would prefer to keep these processing functions in PHP (as opposed to #73), as introducing another template language there would be even too complex for beginning themers, and also force developers to learn it. Template file = for themer, variable processing = for developers.

3. In this case it would be uber-important to keep Drupal core CSS and HTML clean, so that themers won't need to go into the preprocessing stuff for every little detail.

4. More of a detail: to keep logic out of the template files, there should be a template suggestion per view mode, e.g. node-teaser.tpl.php and node-full.tpl.php. Simple and clear, right?

A trade-off of switching to a templating engine that doesn't support logic is:
* Modules that (using javascript or other purposes) explicitly target certain classnames or ID's will have no choice but to provide #prefix and #suffix wrappers around a specific template.

While this makes the approach to solving certain problems explicit and easy to document, Drupal will take a minor step backward into outputting unchange-able markup in some places. If we're willing to accept that functionality/challenge I think something like Twig would be just fine.

For example:

The output our module wants to target:

<div class="[classes]">
  <h2>[title]</h2>
  <p>I need access to...<span>this</span></p>
</div>

A themer may do this when overriding the template:

<div class="i-only-care-about-my-classes and now-the-module-cant-see-this-section">
  <h2>[title]</h2>
  <p>I need access to...<span>this</span></p>
</div>

We wind up with:

<div id=[provided-by-module] class=[alternate-solution-provided-by-module] i-cant-be-altered-by-the-themer">
  <div class="i-only-care-about-my-classes and now-the-module-cant-see-this">
    <h2>[title]</h2>
    <p>I need access to...<span>this</span></p>
  </div>
</div>

The above solutions I really don't like as it's, IMO, a step backwards from the wrapper bloat that is so hated amongst front end devs. What this proposal sounds like to me is, more or less, implementing the Smarty Template Engine as the default template engine as opposed to PHPTemplate. I'm also concerned about this:

This offers less flexibility, but then again, flexibility is not the only goal here.

When revamping the theming system, I think the thought of removing flexibility from a system that really needs to be super flexible to account for all known and unknown usecases (after all, this is a platform to build whatever the hell you want off of) is a bad move. This also only solves writing templates inside a theme and doesn't help necessarily with module provided theming (for instance, theme functions spit out HTML instead of kicking to a template). Views and Panels already do this to some extent, and you'd be hard pressed to find someone who likes the divitis this causes.

Sometimes you get a more powerful system by removing flexibility in certain places, because it gives you flexibility in others. Let's not knee-jerk assume that flexibility == "I can do anything anywhere", because that's how we got into this mess in the first place. :-)

I don't think we'll ever arrive at a consensus of theming engine, so I don't want to go too far off-track here. PHPTemplate might not seem as slick as Smarty, HAML, or mustache, but it works, and is probably the most flexible. The more one digs into a theming engine, the more it just seems like an abstraction of PHP, IMHO.

We can agree that flexibility is sometimes at odds with simplicity, and the current state of Drupal is tipped in favor of the former rather than the latter.

I think a lack of consensus is actually ok, and actually a good thing because I am hoping it will drive the idea of having a more pluggable relationship between modules and themes.

Let's face it: Core has to produce markup. But the current method of doing this is very tightly bound to the render system and thus PHPTemplate. If this relationship were more clear-cut, we could plug any sort of theming API into Drupal we'd like (see also my comment #66).

So, stated simply: modules and themes are too tightly-bound via the glue that is Render API. I think some of RobLoach's links in comment #69 highlight some other issues related to this.

I know at this point we're just collecting pain points, but here's some more information about a possible pluggable architecture: http://c4rl.ws/blog/node/4

Oh well, seems like nobody interested in my grandeus plan (so much for building momentum). :P

Ok, some said that Drupal has a dumb block system, same thing can be said about Drupal theme functions. It knows arrays as arguments, string, integer, but does the module using it knows what they are though.

So here enters Rules|contrib wrapper for Rules actions arguments. This way we can have theme functions that depends on a certain entity, or we know that the argument is an integer, and contrib modules like Panels can use it in its rendering system. BOOM. :O

So we will have something like theme_list(<entity:list>, <int>) {}.

The entry points for themers is theme functions I think, we generate html there, maybe we could also add css file there also. It's like Panels|contrib layout plugins.

Drupal rendering system also rolls around theme functions, we have #theme..., and #type that eventually leads to a theme function.

Edit: the basic idea is that theme functions are mass produced by themers, and they decide what arguments they want to accept, and Drupal made that transparent through user interface.

Crell, very very true. My concern isn't so much remove flexibility here to gain it elsewhere, but more removing flexibility for the sake of removing it (i.e. removing the flexibility of PHP in templates in favor an, IMO, a worse solution of Inception-level div wrapping, thus making a system that ultimately doesn't solve our issue, and worse yet, makes the output HTML even unwieldily than it is now. Oh, and less flexible to boot.).

These are all great ideas, and I'm really happy with the direction things are looking.

We need to clean the way our template system works. Maybe using something like Symfony Templating would help. It supports PHPTemplate out of the box, and is easily extendible to support other rendering engines.

My biggest pain point in Drupal's theme layer is the lack of consistency. Followed closely by the inability to alter markup that has already been rendered. I speculate, most of what a themer wants to do when altering default output is add/remove classes, change/rearrange tags and add/remove wrapping divs.

I really really like the idea of a render array. However, I'm often confused as to where and how to alter it, so it outputs what I want. Do I use a preprocessor, processor, pre_render, alter, write a custom theme function or circumvent it altogether by changing the template? That's a lot of options for a task as simple as adding a class to a tag.

Here are some thoughts on how the theme layer could be improved. For context, I don't consider myself a developer. I am most comfortable in Photoshop, HTML and CSS. PHP doesn't scare me too much, if the function and variable names are familiar and make sense (unless we're dealing with databases.)

1. Nothing should be rendered into markup before it hits the theme layer. This allows for themes to render content in a format other than HTML and gives them the last say in presentation.
2. Markup is made of a few simple building blocks: content, tags and attributes. These building blocks should be easily modified in a simple and consistent manner. That's good for both developers and themers.
3. Extra contextual information that is passed to a template is useful. However, it is distracting and intimidating when you just want to know what you need to change in order to add a freaking class. Think of all the stuff that is exposed when running something like dpm($vars); This information should be segregated from the render array. As a bonus dpm($vars); would resemble the actual markup much closer than it currently does.

I wanted to see what something like that might look like. Here is a link to a proof of concept. https://gist.github.com/1911705

The general idea:

1. The basic building block is the element object.
2. Elements are made up of optional properties: tag, attributes, content and wrappers. These are familiar terms that new themers can easily identify with.
3. Elements will also have a context property. This serves as a, for lack of a better term, dumping ground for all the contextual information that may be useful in altering the element. The node object, for example. It's important, this be kept segregated from the above properties as it can overwhelm newcomers.
4. Elements can be nested and wrapped like xml or html (or possibly converted directly to json?)
5. Modules and themes can create and alter elements. Templates would print them to the screen using a render function:
   <?php render($block_title); ?>
6. The context should never directly affect the rendering of the element. It should only be used to alter element objects. Essentially the render function ignores context.
7. Theme functions are just a means to construct an element object.

With a consistent render element, this opens the door for some very useful methods, such as:

<?php
$block_title->set_tag('h1');
$block_title->add_class('fancy');
$block_title->strip_classes();  // Hooray!
$block_title->strip_wrappers();  // Efff Yeah!
?>

Theme functions might look like this:

<?php
function theme_link($content = NULL, $path, $attributes = array(), $context = array()) {
  $attributes['href'] = check_plain(url($path), $context);
  return new element($content, 'a', $attributes, array(), $context);
}
?>

Some potential downsides:

Less control at the template level. Admittedly, the preservation of templates is not my end goal. While they are approachable for newcomers, I find templates give themers a false sense of control. I feel the real power of themes is in the CSS and how the output is dynamically constructed before it reaches the template. My aim is to simplify that process and make it more approachable for non-coders.

This could lead to some serious nesting of elements and the opposite of what sun shows in #58. Essentially you could end up with a block template that looks like this:
<?php render($block); ?>
Not sure of a way around this. This could possibly be mitigated with WSCI since content will be delivered in smaller chunks, but that's just a guess.

The proof of concept above does not address the rearranging of elements which, in my experience, is where templates are most useful.

Performance. Good or bad, I have no clue.

@pixelwhip - that looks an incredibly sane proposal! The best thing about it is that I can see a transition from what we have now into something more like that being possible without radical leaps and breakages.

Yes, it is most often about adding or removing classes and changing tags. Yeah, that seems like a good thing to target.
Hm, only in the most simple cases is there a single 'tag' to change. Look at the tpl files in use - many supply structure for loops and lists, others for structured content. Right now there isn't even a theme for block_title, there is just 'block.tpl.php' which has three tags that could be controlled by this system and yes, *would* take us back to the #58 example.

Dammit, I wanted to love this idea, but as soon as I'm one step into it, it's falling apart for me.

Y'know, you could probably build a theme like that right now with liberal use of the render arrays' #prefix and #suffix and throwing away a large number of tpl files!

Thanks for taking a look dman. Can you be more specific about the limitations you're seeing? I don't think I'm seeing what you're seeing.

The element object should allow for just about any structured content. The content property of an element can be a string or an array of strings and, more importantly, other elements. You can also have an element with no tag, in which case it would just render the content. Or an element with no content- just a tag with attributes, like an image for instance.

One thing I haven't fully thought through is how to best pass these elements to a template in a way that keeps the templates clean. To continue with the block example (knowing full well it will be undergoing a massive facelift), the default block template could look something like this. See block.tpl.php for reference.

<div <?php render_attributes($block); ?>>
  <?php render($block_title); ?>
  <?php render($block_content); ?>
</div>

OR

<?php open_tag($block); ?>
  <?php render($block_title); ?>
  <?php render($block_content); ?>
<?php close_tag($block); ?>

OR

<?php open_tag($block); ?>
  <?php render($block->content['block_title']); ?>
  <?php render($block->content['block_content']); ?>
<?php close_tag($block); ?>

OR

<?php render($block); ?>

There are a number of directions it could go. I personally feel, any of the above are cleaner than the existing block.tpl.php. Like you said, it may be good to try this approach in a actual theme to see how it would work. Although, I'm not sure how far I could get with this in a theme alone. It may need help from a module.

Well, in example #2 (and #4) above, you have have silently posited that where we used to have ONE block.tpl.php, we now have three templates, for block, block_title, block_content. :-} Three places where a themer has to look to find out how to change the h2 to an h3.
That's not going to win any friends.

The original block.tpl.php was at least comprehensible (with a little squinting) to folk who could real HTML. The alternative above is less so, *and* is only a slightly different way of doing what sun illustrated in #58 as a bad thing to do. There is in fact zero HTML in those examples! It's just code. Where is an HTML-coder supposed to start?

This direction is not what we can call 'cleaner' than the existing block.tpl.php. It's antiseptic. You have thrown away all the skills an HTML-expert could have used.
The existing templates use HTML for HTML stuff, and squirt code-based stuff into it. Yes that mixes the paradigm a bit, but all templating languages do that to some extent. I think that pushing it FURTHER into config and generated code and further away from anything a themer can recognise as HTML is not where we want to go. Remember the mission here.

(aside)
I've worked on (helping people escape from) a few CMSs that followed the same rainbow (MySourceMatrix, MSCMS) that decided that HTML markup was too special to be managed by mere users or themers, and built their own data structures and nested element trees where almost every HTML element was passed off to their own configurable element renderers before it came out the other end as HTML. I can appreciate that the developers at one time thought that HTML in the database was icky, and that their alternative internal representation of the documents was superior and gave them more control. But once this approach starts to scale up, it's shocking.
Already, I can agree with the designers that I work with that the Drupal system of theming is going a little too far down this path, and I think we need to pull it back and make the code look and behave MORE like real HTML, not less.

I really do see the attraction in the abstract idea of making every bit of code generic & hyper-configurable, but it's just not sane enough for even a well-skilled web developer to make sense of without elevating the learning curve out of reach.

I'd throw out an alternative prototyping suggestion that is the opposite : 100% HTML templates (NO code or markdown) that get modified via post-process manipulations to inject required classes and get the CODE to do the hard work of figuring out what goes where, not the beginner themer.
Not a serious suggestion, but one that could give Drupal a name as being intuitively easy to theme for!

There is in fact zero HTML in those examples! It's just code. Where is an HTML-coder supposed to start?

Just a quick thought: if the default markup was good, and it was easy to add classes and wrappers or change tags in the UI, just how often would we need to look at templates/functions?
Hardly ever, I guess, potentially never. When writing CSS, don't we use Firebug/Web Inspector etc, rather than looking at template files?

If I were building the perfect, all-customizable theme from scratch from the Drupal/coder side of things, then there is a lot of merit in a totally-configurable code-based theme engine.
However, many of the real-world use cases, and especially when training (even skilled) Designers, webmonkeys and theme-cutters is that they can build a working prototype in static HTML and then "convert" it into Drupal theme templates.
That has meant a small amount of copy & paste from HTML source files into Drupals template snippets. And some replacement insertions. And that just works.
It's not the only workflow, but it is a common one, and one that did work pretty well most of the time.
Themers work with files. With other desktop tools. Files can be managed through version control and released over a working site. And released as, y'know, "themes".

The alternative - building a theme from the ground up with full knowledge of how to leverage the Drupal theme system is a much more specific, non-transferrable skill. Rarer and harder.
Theme settings in database (a-la contemplate) have their place for fiddlers, but they are not fun for full release development and original from-scratch themes.
</opinion class="random">

However, many of the real-world use cases, and especially when training (even skilled) Designers, webmonkeys and theme-cutters is that they can build a working prototype in static HTML and then "convert" it into Drupal theme templates.
That has meant a small amount of copy & paste from HTML source files into Drupals template snippets. And some replacement insertions. And that just works.

Well put. This is actually a situation I've come across in my current employer, where its sometimes easier to let the front-end developer mock everything needed in a static folder, and let me pull the rest together for Drupal when I'm ready. Truly, I like this approach, because it keeps the theme dev/site-building separated from one another until components start becoming dependent on each other. This seems to go faster, and keep responsibilities properly separated. (And it gives the client a moment to really experience what they signed off on--and push back EARLY!)

Something that has crossed my mind, is since theme engines are supposed to be modular in the first place (or at least we're discussing the opportunity of making them easily swappable), is that Drupal 8 might just introduce expertise driven themeing tools. One templating engine (for instance PHPTemplate), maximizes control and requires the most finesse, and a solid knowlege of APIs and programming. On the other side, is the beginners theme engine, which is more like Twig, using nothing more than simple substitution to process content; no programming required, and no crazy logic all over the place.

I really like some of the ideas from the previous posts.

The fact that the object that is used in a template file is full of confusing data is really intimidating. If we could simply map this to logical things that reflect the structure of a template file, that would be sweet. For a block for example:

Title
Title_attributes
Content
Content_attributes
Attributes

I would however not go as far as to take the HTML out of the template files, since that would not lower the learning curve for themers. They are template files, so let's keep as much markup in there as possible.

I'm quite sure that some options could just be taken out of the template files, to make them easier to read. Let's look at block.tpl.php as an example.

<div id="<?php print $block_html_id; ?>" class="<?php print $classes; ?>"<?php print $attributes; ?>>

  <?php print render($title_prefix); ?>
<?php if ($block->subject): ?>
  <h2<?php print $title_attributes; ?>><?php print $block->subject ?></h2>
<?php endif;?>
  <?php print render($title_suffix); ?>

  <div class="content"<?php print $content_attributes; ?>>
    <?php print $content ?>
  </div>
</div>

The title_prefix and title_suffix for me are typical examples of advanced cases that might be useful once in two years, but let's keep the default simple.
Some naming is also confusing: $title_attributes and $block->subject for example (title vs subject).
Wether we're working with a node, or block or whatever should be transparent in a template file. No $block->title, but $title. We are in block.tpl.php so the system should now.
By combining all attributes into one variable, we can also keep it cleaner. Combine this with the option to add classes through the UI (Block Class, Views, Field UI) and that should not be an issue.
My proposal in this specific case:

<div <?php print $attributes; ?>>
  <?php if ($title): ?>
    <h2<?php print $title_attributes; ?>><?php print $title ?></h2>
  <?php endif;?>
  <div <?php print $content_attributes; ?>>
    <?php print $content ?>
  </div>
</div>

The combination of really simpel arrays that arrive in the template files, and readable template files seem like the way forward to me.
All advanced changes can still be done in preprocess functions.

Fully agreed that a themer should not have to deal with arrays and objects. Printable things should be pre-cooked & exposed for use.
To *me*, dereferencing an array is as natural as breathing, but the difference between $block->title, $block['title'], & just $block_title is a huge cognitive leap for those that don't naturally think in those terms. We can round off that corner for them. (I do a lot of training)

Seeing as we are blue-sky spitballing here, I'll refer to my Manifesto against markdown - in favor of (X)HTML(5) + microformats+RDFa

Any "alternative" templating syntax attempts to be a "simplification" of what we do with phptemplate. And almost immediately starts adding the same complexity using yet another syntax. A syntax with its own learning curve, rules to read, lack of IDE parsers or validators, and a smaller audience.
Seeing as anything *but* phptemplate invokes yet another post-process parsing procedure, there is a processing overhead (which we can probably eat) but there is a *cognitive* overhead that I really don't want to see. I'm sure that "twig" or "fictive" ;-) token-based replacements for phptemplate have their own specialized strengths, but they have unnecessary learning overheads. The perfect-case example can sometimes look cleaner than PHP (I personally see zero visual win in #73 for example) but when they are called on to do what phptemplate does daily (see a views template) they are even clunkier.

Drupal *used* to support a number of templating engines (they are probably still alive somewhere) but we congregated on phptemplate because we could do everything we needed there, and the documentation and manual didn't have have big chunks of "but if you are using Smarty then do this" clauses in it any more.

So : no more new templating languages. Please.

But.
We can take jyve's (or block.tpl.php) examples and rip out the PHP as much as possible (still need some "if" there) and get the machine to do the rest.

<div typeof="drupal:block">
  <h2 typeof="drupal:block-title"><?= $title ?></h2>
  <div typeof="drupal:block-content">
    <?= $content ?>
  </div>
</div>

And write a post-processor that is smart enough to parse the elements, realize what they need to do, and insert the rest for the themer.
Our theme func can look at those "typof" tags and figure out if there are attributes to add into it. And yes, even change the element type on the fly if needed!
I'm sure you can see the pseudo-code in your head for this process already. (PS: Don't use regexp)

You'll notice I've drunk the RDFa Kool-aid too much. (If you know me, you know that happened to me long ago) but RDFa is part of core and we can make it meaningful (aka semantic) for anyone who opens these files later. It could be guerrilla upskilling to best-practices for our designer buddies.

I've not thought this all the way through yet ... but I've been writing DOM-based input filters for a while now, and ... they work and feel better. To me.
I think a similar approach can work for templating.

Just throwing this out there, there's gonna be some pitfalls - but maybe it'll jive with someone.

_{Edit: our magic post-processor can even remove empty elements for us. Don't need the PHP if}

My biggest grief as a themer is that I have to make an installation profile, why can't I make simple layout plugins and let site builders choose from those layouts I gave, and also give me the data I need. Which was the point I made at #81. ヽ(｀Д´)ﾉ

selfnote: it's not onion heads, but close enough 0.0b.

My biggest grief as a themer is that I have to make an installation profile...

I think you have a different idea of an "installation profile". An installation profile is used to build a new site in a few clicks or with the help of something like features. Unless I don't see something.

why can't I make simple layout plugins and let site builders choose from those layouts I gave

This is what Panels offers currently, and site builders can select from the panel configurations that are enabled. There is a module called Panels everywhere, that lets you take over the system entirely with Panels, but it takes a LOT of finesse to make it work exactly right.

It might be in our interest, to put something like Panels, or a variant thereof in Drupal 8 core.

I'll be posting a blog post later this week (as will Dries) on this very topic, but in short, "Yes we're going to try to put something panels like into core".

Eclipse

Going back to what Yched wrote in #60, I think the need for altering markup through the UI was born from the fact that altering the markup through php in the theme layer is so tedious.

All of our front-enders have recently taken a huge liking to display suite (basically from the day we started using it, after the learning curve was done with). The first thing that comes to mind would be: "Great! Let's go the config way then. Themers really like DS!". But, and this is a HUGE but, DS was born out of the fact that the template files are too hard to handle in a lot of cases.

Just like a lot of people have mentioned, altering classes and wrappers is very hard work in a lot of cases, and DS makes this easy. If it would be easy to do through template files - would themers still want DS? Personally, I don't think so.

Ps. I don't have anything against display suite, it was just an example. There are ofcourse other modules that do similar things.

@EclipseGc Will be interested to see the post, please provide a link on this issue when ready. :)

Something I've forgotten to mention, (and run into for many clients) is the inherent bad behavior of Drupal core (presently) to attach unlike objects to a page view without any regard for real re-usability. As useful as these may seem out of the box, they're a huge pain point for reconfiguring a page's behavior.

Take, for instance, the comment stream attached to a node. Unless I'm using views, or know exactly how the theme layer works, they are always attached to hook_node_view(). The same goes for attaching node teasers to taxonomy terms.

It seems silly to keep making modules for ridiculous workarounds for things that should logically be separated from one another.

Since Drupal 8 will introduce context-driven "smart" blocks, I don't think this will be a big issue; but figured this was worth mentioning.

Typically, the code where you'll see this behavior looks like this: Why not detach this entirely? How on earth

function taxonomy_term_page($term) {
...
if ($nids = taxonomy_select_nodes($term->tid, TRUE, variable_get('default_nodes_main', 10))) {
     $nodes = node_load_multiple($nids);
     $build += node_view_multiple($nodes);
     $build['pager'] = array(
       '#theme' => 'pager',
       '#weight' => 5,
      );
   }
   else {
     $build['no_content'] = array(
       '#prefix' => '<p>',
       '#markup' => t('There is currently no content classified with this term.'),
       '#suffix' => '</p>',
     );
   }
...

JFTR, two posts mentioned @ #95:

* http://krisandju.e-webindustries.com/blog/drupal-8-blocks-layouts-everyw...
* http://buytaert.net/layouts-for-drupal-8

Thanks all, for participating in this discussion so far. Some great stuff being uncovered here. I started creating issues tagged with consistency and d8dtx (themer experience) to capture some of what I think is actionable from what's been mentioned. Please add more as you see fit. The ones I added don't yet include the meaty topic of how the theme system should change once the Layouts work referenced in #99 is in core. That, I think, requires quite a bit more thought. I hope I have a chance to meet some of you in Denver next week and chat about that. If any of you feel like attending the core conversation that John and I are leading on the theme system, I look forward to meeting you there.

Would love to attend the core conversation, but unfortunately, I won't be in Denver.
Will it be recorded, do you know?

Yes, we plan to record Core Conversations.

@effulgentsia Thanks for breaking these down into digestible issues. I'll be training all day Monday, but would definitely like to sprint on this in Denver. Anyone else? It may feel like a waste of time until we know exactly how the theme system is going to work with the layout initiative, but any steps toward consistency should make any needed transition easier.

Totally, however it would be nice to sprint on this on a day that isn't Friday. That day is already packed with sprints.

I'd just like to note, for the record, that this post details what I think is wrong with current theme system: http://jacine.net/post/19652705220/theme-system

I wonder if we have someone between us which has more access to solutions to this problem from other frameworks, cms, languages etc.

I'm not that experienced, but there are problems out there which are solved over and over again, just because you do not check solutions outside of your scope of expertise.

I mean there is java, perl, python, ruby, asp each with top 10 frameworks / cms and therefore well grown or prefered solutions to this problem. Reinventing the wheel is not always the best solution imho. And do not say that drupal is superspecial. As far as I understand Jacines and suns post, we *just* look for a solution to limit the places stuff will be created, then altered and ultimatly rendered. This sounds like a common problem - not just one from drupal.

http://en.wikipedia.org/wiki/Comparison_of_web_application_frameworks

ps: for the record, I hate the render arrays more and more. a print_r() on a variable in drupal is a firefox output nightmare.

foobar3000: Have a look at #1499460: [meta] New theme system, where there is strong support for building a new theme system around the Twig template engine. It's not a drop in, surely, but it's a lot less Drupal-specific stuff.

I would like to add ideas here. Maybe someone is still reading.

As far as I see my drupal development. There is the CORE, then Community Modules, which I need to get functionality I need for my projects and finally, there are MY modules which add stuff I need to make my project special and custom to my needs. Sometimes MY modules are modules from the community I slightly extend.

So there are 3 Blocks of components sitting on each other. MY STUFF -> COMMUNITY -> CORE.

In Magento (PHP Shop System) All templates are hierachycly ordered. BASE / DEFAULT and THEME. The theme system is ordered so that files from BASE can be overriden by files in DEFAULT if they have the same name and the same place in the filesystem. The same applys for THEME, which can be override DEFAULT files. The great thing is that this ladder of templates has always the fallback to the BASE template files.

In BASE are all template files the magento core needs to display a functional basic shop and and the admin area. The DEFAULT is mostly used by extensions and the THEME by customisation.

As far as I see it and understand Janice correct the data itself could be just data on its way to the browser until the ultimate end when the rendering is performed. When every data structure is a LIST, then the wrapper / list decorators could apply nicely.

The flow of data could be the following (abstract)

* CORE: creates some data to print
* CORE: sets the list-theme to *table*
* COMMUNITY: adding interesting and usefull data
* COMMUNITY: processing data (for example switching the list-theme to *collapsible-table*)
* USER: adding interesing and useful data
* USER: processing data (removing and fetching some data)
* THEME: rendering the data array with the given themes for this item

Every step the order of the modules is ordered by weight. It all reminds me of the decorators concept.

The rendering itself (putting xml/html on the data) should be happen at the end. I guess thats what everyone wants. The USER should be alwayys the person who should be resonsible for the look of the outcome of the data. HE likes the suggestions of the CORE and COMMUNITY, but should have an easy way to alter the output.

Not sure if that all makes sense, a user with a graph tool could be useful.

Ok, as this is a "say what's on your mind" thread, I'll do exactly that :)

The myth of logic-free HTML

#6 (JurriaanRoelofs)

Ideally the theming layer would be completely free of logic/functions

I'm just picking this quote to say what I wanted to say anyway :)
I had this conversation with the person I work with. He is very fond of smarty (built into CiviCRM), and separation of logic and display.
I generally agree with the separation of business logic vs display logic, BUT:

We are building dynamic websites, with dynamic html.
Dynamically assembled attributes, zebra striping, grid layouts, or fancy table stuff with dynamic rowspan and colspan.
All of this is logic. Not "business logic", but it is logic.
And it is getting more, not less.

The "I am just the html guy" person can provide us with some html wireframes, but ultimately we want someone with a sense of logic and mechanics stuff, to turn this into a theme.

The question is, what is the ratio of logic vs html?
If we have more html than logic, or the logic is just some simple if/else, then a template is the preferable thing to do.
If most of the logic can be processed before the template, or moved into a helper, then let's go for that: Preprocess + template + helper.
If however, if the HTML is really highly dynamic, then IMO it is better off in a function or method, than in a template. E.g. for menu rendering.

Theme/layout should not be static/global-scope

Most theme overrides, e.g. for theme('menu_link'), will (by default) indiscriminately override the rendering of all menus. No matter if it is a dropdown menu, a sidebar menu, a sitemap..
We choose a theme somewhere in hook_init(), and then everything we do is dominated by this decision.

On the other hand, the theme function itself is ridiculously fine-grained (in menu rendering): theme() is called up to 3x for every menu item.

There are much more practical systems in Drupal: E.g. with Display suite or with panels, you can create custom layouts. These do not "override everything", but instead can be used just for specific node types and view modes.

Menus should work the same:
Instead of saying, "this is the html of a menu link", a theme should say "please render the top nav with layout X, the sidebar menu with layout Y, the sitemap with layout Z", and then provide these three layouts.
A "menu layout" would be a group of different functions, methods or templates, that generate and assemble pieces of html.

It would seem natural that a "menu layout" is a PHP class, where the methods create the html snippets. But then we need to say goodbye to the romantic idea that every piece of html needs to be in a template, and strictly separated from any logic. A bunch of micro templates + preprocess logic + dynamically assembled tag attributes, is hardly a better DX than just having it all in a PHP class - including the html.

Just to add my two cents. This has probably been covered before but from a front-end development perspective Drupal leaves a LOT to be desired.

I've now built 5 Drupal sites (one in D6, four in D7). The last one I've been working on, I've ended up using JS (yes, JS!) to add markup to core forms. WTF? I searched and searched for "How tos" but to no avail, the project needed finishing so I added JS. This is ridiculous (and won't degrade gracefully). I'm a front-end developer with umpteen years of experience, good research skills, perfectly willing and able to use template.php and alter the .tpl files and I'm using Jquery to add markup. For crying out loud!

From my perspective, there are two quite basic problems:

1. The way Drupal's theming works needs to be properly DOCUMENTED. We can talk and talk about how it should or should not be changed but if I don't have a complete grasp on how it actually works after building 5 Drupal themes from scratch, how can I comment properly? Yes, yes, I know, it's boring doing the docs bit, I've written enough of them over the years. Finding bits of replies on forums, little snippets from dark corners of the web and trial and error has added about half again to my development time for my current project, which I have taken the financial hit for (it's a good portfolio piece so I'm not "bitching", just sharing my experience). The documentation I've found is often so abstracted that it is practically useless in the real front end development world. For instance, today I can't find the way to alter the description beneath the password fields on the register form (I've successfully altered other stuff though). Eh? Why aren't the descriptions for core form fields exposed in the GUI? Trying to work out the voodoo that is template.php is ludicrous for simple tasks. It seems to me that power and flexibility are driving the development of Drupal, while the ability to do simple tasks within core take a back seat. It feels to me like someone who has a grasp on usability/front-end development within a CMS (if only I had the time) needs to take a good long look at some pretty basic stuff. It's arrogant to assume that those descriptions are a catch-all for any site. I want control.

2. There needs to be some way to force CONSISTENCY in any future development for/of Drupal. I have come across a vast array of inconsistencies which are just mind-numbing (and on a bad day, angering). I can alter some bits of forms in template.php but not others? I have to hard code a link to a downloadable document into the user profile .tpl rather than there being a way to simply add it via the core GUI for that page? I can't add some oh-so-simple instructional markup to the top of the register form in the GUI? Lets not even start on the CSS - suffice to say my stylesheet is approaching 500 lines with a lot of repetition and pointless messing about. As someone mentioned, I've styled .form-item almost 30 times!

Further to this (don't worry, almost done) why-oh-why are there ANY CSS styles attached to any part of a module or core when a theme is present? What is the point? I spend hours and hours filching through wrapper after wrapper to find that some other stylesheet is breaking my carefully constructed HTML/CSS (which works perfectly cross-browser when I initially build it). This is just maddening. Unless I'm missing something the core/modules can't be seen without a theme so why is there default CSS being read for my custom theme? I can live with the wrappers (although the "bloat" is sometimes just horrible) and I can see why they are there. But default CSS, really?

Anyway, I have been really frustrated at times with my current project so I apologise if this all comes across as a rant. Drupal could easily be the best open-source CMS out there but it lacks a guiding hand or a definite trajectory to really make it shine. Simplify the theming process by documenting how to do it (which would in turn highlight inconsistencies in it) and make sure the core (for starters) works in a consistent and logical way.

Keep on trucking kids,

Phil.

Title changed to something less hilarious albeit more descriptive. Removed explanation.

there are a lot of comments like #18 in this issue can be inserted into summary
so tagging

looks like this is related to #1696786-160: Integrate Twig into core: Implementation issue so added related into summary

Updated issue summary.

Updated issue summary.

Since the related meta is closed and Twig has been added to core, I'm pretty sure we can close out this issue.

This issue was prematurely closed. Twig has nothing to do with this issue, despite having hijacked the last 1/3 of it. Twig is simply a template engine, that's all. It is essentially the last step in the process and thus, not involved with how we provide the data to templates.

This issue is about the underlying theme systems (i.e. registry, theme hooks, etc). I am attaching related issues (which provide consensus and solutions to similar issues discussed here), however are rather specific in nature of what they're trying to accomplish.

This issue should remain open as a general discussion. How we approach the "theme system" in the future (9.x) matters. Many of the sentiments expressed by @sun (#18) and @Jacine (#105) are still valid, especially the ones regarding current render array and Form API co-existence (which is still probably about 80% of the issue at hand IMO).

Some of the issues raised here can be done in 8.x, postponing a discussion until 9.x means there'd be no time to do any of it.

[ redundant rant removed ] - see related issues

Thanks @yoroy! I hadn't come across the above issue.

For anyone who stumbles onto this issue like me:
Unify & simplify render & theme system: component-based rendering (enables pattern library, style guides, interface previews, client-side re-rendering) is a great move forward and a solid proposal, Create a new user-facing core theme is currently required for progress.

Adding a new path forwards with a proposal from the Red Hat Web Patterns working group.

Decouple Drupal’s Theming Layer into a REST Service

Closing due to lack of any recent activity and in favor of #2821399: [meta] Theme System Modernization Initiative.