Allow Twig templates to use front matter for metadata support

Well then you lose the nice support for HTML editing and validation in your IDE of choice. There's never going to be a time when these aren't XSS filter probably because changing that results in a change to the security model so /shrug.

However I can see the benefit for doing this to extend the pattern to other templates.

OTOH maybe we should propose an upstream change to support template metadata like this is Twig in some completely better way.

OTOH maybe we should propose an upstream change to support template metadata like this is Twig in some completely better way.

I've gone ahead and opened https://github.com/twigphp/Twig/issues/3082

Perhaps we should postpone until that is either resolved/closed?

:D

The suggestion is to implement one more twig loader https://github.com/twigphp/Twig/issues/3082#issuecomment-507992951

Well, that sucks :(

I hate when people just casually dismiss a carefully thought out proposal like that; why even bother.

---

I personally would prefer to do C.

This could have the added benefit of fatal-ing if it cannot parse whatever we decide to use moving forward.

The problem with A & B is that it can be syntactically correct (HTML is valid), but could still have an incorrect tag name or attribute name used.

I'd rather avoid this altogether and just assume a block of content at the beginning of a template is some sort of key/value system (1:1) in the form of JSON or YAML.

This way, if there really is something wrong, we'll know.

In fact, I suppose we may as well use a semi-industry "standard" like front matter blocks instead. The only reason I was proposing creating the {@ tag in Twig is so that it didn't clash with existing tags/nomenclature. I'd rather not subclass all of Twig just to get that working.

In fact, we can easily implement an existing 3rd-party library: https://github.com/webuni/front-matter

Which, coincidentally, also contains a Twig loader we can just pop in:
https://github.com/webuni/front-matter/blob/master/src/Twig/FrontMatterL...

We will likely need to add another method into our custom TwigEnvironment class to extract this from the loader, but it seems doable.

I'm still not entirely convinced this hackish approach is going to be very performant though; which is why I was hoping to at least get the Twig project's involvement.

Sigh.

+1 to #11. If we change this we should change this to doing it in a better way and in a way that can be leverage by other things.

but right now we don't have to load the Twig files at all to enable the plugin manager can get the meta-data

This isn't really true as get_meta_tags() takes a filename argument. This implies that it is loading the file to parse the entire thing for <meta> tags.

Regardless of the fact that this loading is obfuscated by PHP and likely using more C based functions to speed it up, it's still having to parse the entire document in an attempt to find unique <meta> HTML tags and their attributes.

If we switched to a front matter approach, we can simply extract just the beginning of a template and not have to worry about the rest (or rather let Twig worry about the rest).

If we change this we should change this to doing it in a better way and in a way that can be leverage by other things.

Yes, which is why I'm cautious of introducing this new <meta> paradigm into Twig templates now (just for the sake of Help topics).

Not fixing this now will only introduce confusion and complications in the future, let alone having to provide BC support (which will be even more of a performance hit).

Better title

Here's the simplest/easiest implementation I mentioned above using the existing https://github.com/webuni/front-matter library.

It essentially wraps @twig.loader (chained Twig loader) and allows all the metadata to be captured automatically (regardless of other loaders).

Note: I'm aware this patch will fail because I didn't even remotely touch the tests.

I did manually step through this and the UI to make sure everything works though.

If someone wants to fix the tests, please go for it (not really my area).

From what I gather, it will likely need to be converted to a kernel test now that the Twig service is needed in the discovery.

+++ b/core/modules/help_topics/src/HelpTopicDiscovery.php
@@ -115,27 +126,15 @@ public function findAll() {
+          $data['label'] = new TranslatableMarkup($data['label']);

FYI, it would also be nice to simply get rid of this and just use !translatable YAML tags instead #2088371: YAML discovery incompatible with translations, e.g.

---
label: !translatable "My title"
---

Front matter is YAML, that's the stuff in between the --- block at the top.

1. +++ b/core/core.services.yml
   @@ -1624,6 +1624,12 @@ services:
   +  front_matter.twig_loader:
   +    class: Webuni\FrontMatter\Twig\FrontMatterLoader
   +    arguments: ['@front_matter', '@twig.loader']
   

   How come we can't register this like the other twig loaders?

     twig.loader.filesystem:
       class: Drupal\Core\Template\Loader\FilesystemLoader
       # We use '.' instead of '@app.root' as the path for non-namespaced template
       # files so that they match the relative paths of templates loaded via the
       # theme registry or via Twig namespaces.
       arguments: ['.', '@module_handler', '@theme_handler']
       tags:
         - { name: twig.loader, priority: 100 }
   

   ie. with tags?

2. +++ b/core/modules/help_topics/help_topics.api.php
   @@ -18,7 +18,7 @@
   -  $info['example.help_topic']['top_level'] = TRUE;
   +  $info['example.help_topic']['topLevel'] = TRUE;
   

   The change of top_level to topLevel is out-of-scope. And if it is in scope because the FrontMatter doesn't support _ in yaml keys then that'd be a downer and it shouldn't have an opinion about that.

3. We need some idea of the cost of adding this to ever parse of a twig file.

Also adding new libraries requires a heap of work and is usually done in a separate issue. See #3036210: Add justinrainbow/json-schema as a composer dependency so JSON:API can use it to validate its responses for an example of the work that needs to be done.

The good news webuni/frontmatter that it adds no new dependencies...

    "require": {
        "php": "^5.6|^7.0",
        "symfony/yaml": "^2.3|^3.0|^4.0"
    },

So that's one headache gone. But that lack of a security policy, code-of-conduct, contributor guidelines etc make it tricky.

+++ b/composer.json
@@ -5,7 +5,8 @@
+        "webuni/front-matter": "^1.1"

Needs to be added to core/composer.json and not the root.

1. #20.1 - Because it's not really a "loader" per se and more of a wrapper of loaders to parse the source code and extract the front matter (YAML) at the beginning. That's why it takes a $loader parameter in its constructor.
2. #20.2 - Agreed. I thought about this right before I uploaded the patch, but decided I'd rather get this POC patch out the door in response to #15. I mostly did this because I really dislike using snake case for new code and see it merely as an antiquated holdover from hooks. Regardless, I should have never touched it.
3. #20.3 - Yes, we definitely need some profiling done. That being said, there's fairly little overhead added here.
4. #21 - Not really. It's already defining a Twig Environment. We could easily add the loader to that to extract the metadata. If we can utilize !translatable YAML tags, it could be as simple as walking through the YAML array and detecting when a TranslatableMarkup object is found. Then the key/value can be used.
5. #22 - I'm actually thinking we should just implement our own wrapper instead of using the 3rd party library. This library does way more than is needed IMO and could be optimized even more like only regex-ing the source if the code starts with ---\n using something less expensive like strpos.

FWIW, I perhaps should have said with the above patch that this was more of an initial POC to show how easily implemented this could actually be. I was less concerned with a full-on proper patch. If this feels like more of the direction we'd like to go, I can be a little more "core gates" aware.

Also created a JetBrains issue for this:

https://youtrack.jetbrains.com/issue/WI-47589

Here's another POC without using any library and just creating our own wrapper.

This patch contains changes that will allow it to work in conjunction with #3068483: Automatically map !translate YAML tag to TranslatableMarkup.

Leave as CNW as the help_topic tests still need to be fixed.

My local, new ddev setup, is having issues getting the testing to work.

If someone will please just take a quick look at them, that would be fantastic :D

I'm still see this new syntax a bit complicated.
Yes, time showed that people can make typos in templates so topics need more test coverage and probably a bit of validation while discovery

Front-matter looks overkill in terms of making code complicated (too much abstraction for simple job), plus what #22 said about extra dependency.

So not clear where this code could be reused except help topics - please update summary to point how this extra dependency could be useful

Front-matter looks overkill in terms of making code complicated (too much abstraction for simple job), plus what #22 said about extra dependency.

It's really not... it's just inlining what would already be a standalone YAML file(s). Also, the latest patch no longer adds an extra dependency.

So not clear where this code could be reused except help topics

Actually, it has the potential to be used by a lot of things. This is/has been something the FE has needed for a while; we simply didn't really know (exactly) what it was until now I think.

I've updated the IS.

---

I think this has morphed from just help_topics though, so I'll create a new issue later that breaks out the help_topic stuff from this issue/patch.

Ok, here's a proper patch that focuses just on the front matter implementation.

edit: Oh and this also includes a fix to any error line numbers that would be offset due to front matter :D

I also finally got my local to work, so this includes some tests as well.

I'll create a new issue shortly for help_topic changes that will depend on this.

Ping

Overall I'm +1 to adding support for front matter in core. As mentioned in the issue summary, there's already at least a few good use cases where we could use this.

1. +++ b/core/lib/Drupal/Core/Template/FrontMatterSource.php
   @@ -0,0 +1,195 @@
   + * Class FrontMatterSource.
   
   +++ b/core/lib/Drupal/Core/Template/Loader/FrontMatterWrapper.php
   @@ -0,0 +1,106 @@
   + * Class FrontMatterWrapper.
   

   Nit: We should add real one-line summaries here. Using the class name as a summary doesn't provide any extra information and makes refactoring harder.

2. +++ b/core/lib/Drupal/Core/Template/FrontMatterSource.php
   @@ -0,0 +1,195 @@
   +   * Wraps a callback to intercept any \Twig_Error and correct its line number.
   ...
   +  public static function fixErrorLineNumber(callable $callback) {
   

   This method could benefit from some extra documentation. We should explain why this is needed.

3. +++ b/core/lib/Drupal/Core/Template/Loader/FrontMatterWrapper.php
   @@ -0,0 +1,106 @@
   +  public function getSource($name) {
   +    // @codingStandardsIgnoreLine (symfony deprecation).
   +    @trigger_error(sprintf('Calling "getSource" on "%s" is deprecated since Twig 1.27. Use getSourceContext() instead.', get_class($this)), E_USER_DEPRECATED);
   

   How do we make sure this method gets removed after we upgrade to Twig 2?

4. +++ b/core/lib/Drupal/Core/Template/FrontMatterSource.php
   @@ -0,0 +1,195 @@
   +  protected $lineOffset = 0;
   ...
   +  public function getLineOffset() {
   

   Where is this being used?

5. +++ b/core/lib/Drupal/Core/Template/FrontMatterSource.php
   @@ -0,0 +1,195 @@
   +  public static function hasFrontMatter($code) {
   ...
   +  public static function parseFrontMatter($code) {
   

   Are these methods public on purpose?

I was pointed here by @lauriii based on #3050386: Allow loading CSS and JavaScript directly from templates, which I've previously reviewed.

1. @jhodgdon Thanks for the excellent issue summary! 👏 Makes it much easier to understand what the goals and context are :) It makes me tentatively enthusiastic that this may make #3050386: Allow loading CSS and JavaScript directly from templates unnecessary 🥳
2. @markcarver I am saddened by the terse response you got from the Twig maintainers. I feel sorry for you :( I thought you articulated it well. You even provided multiple use cases!
3. #16: Using https://github.com/webuni/front-matter looks very interesting!
4. #26: You really weren't kidding when you said we could easily write our own: this patch is 27.38 KB, #16 was 24.82 KB.
5. #30: Actually, it has the potential to be used by a lot of things. This is/has been something the FE has needed for a while; we simply didn't really know (exactly) what it was until now I think. 👍 — see point 1 :)
6. #31: Thanks for distilling it down to the essence!
7. #34: Glad to see you also like it :)

Patch review:

1. I'm not reviewing details, only trying to validate architecture.
2. I think it'd be valuable to roll a patch for #3050386: Allow loading CSS and JavaScript directly from templates on top of this one to help validate this proposal.

3. +++ b/core/core.services.yml
   @@ -1629,6 +1629,13 @@ services:
   +  # This Twig loader should not apply the "twig.loader" tag as it is a wrapper
   +  # around the normal chained twig.loader which collects other registered
   +  # Twig Loaders using that tag.
   +  twig.loader.front_matter_wrapper:
   

   A crucial bit of information that's missing here is that we were told by the Twig maintainers that this is how we should implement it, because they don't want to add it to Twig itself. I think explaining that briefly and a link to the GitHub comment where they said that would be very helpful.

   That also addresses #20.1.

I already did that #3069109: Replace help_topic meta tags with front matter.

Regarding layouts, it would just be taking existing *.layouts.yml definitions and moving them to their respective .html.twig files. Would need to modify the plugin manager to merge in the metadata like help topics does as well.

Here's a new patch that addresses #34 and #35.

After thinking about this a bit, I decided to take a decorator approach. This adds a bit more flexibility and actually makes a little more sense to do it this way.

I also went ahead and created #3075093: Remove/replace deprecated Twig loader interfaces/methods as a follow-up for Twig 2.x stuff in 9.x.

Nice, thanks @markcarver! Any chance you could also do:

I think it'd be valuable to roll a patch for #3050386: Allow loading CSS and JavaScript directly from templates on top of this one to help validate this proposal.

? 🙏

I replied there: #3050386-14: Allow loading CSS and JavaScript directly from templates.

This proposal's use cases have already been validated with #3069109: Replace help_topic meta tags with front matter (which is what this issue started out as).

I could create some issues for layouts as that seems to be the next logical step for utilizing front matter in core.

I think it would make sense to also introduce more of a generic "TemplateDiscovery" mechanism that plugin managers, like help_topics and layouts, can both use.

I've replaced the existing use cases in the IS with references to actual issues.

edit: I've also created #3075427: Create TemplateDiscovery for plugin managers to use and subsequently #3075432: Inline .layouts.yml definitions with each template

1. +++ b/core/lib/Drupal/Core/Template/FrontMatterSourceDecorator.php
   @@ -0,0 +1,85 @@
   +      $this->metadata = (array) Yaml::decode($yaml);
   

   We should move the decoding part outside of a __construct to ensure that we can write unit tests for this code in the future.

2. +++ b/core/lib/Drupal/Core/Template/SourceDecorator.php
   @@ -0,0 +1,50 @@
   +class SourceDecorator extends Source {
   

   Twig\Source has been documented as @final. Looking at the Twig code base it has been already marked as final using the PHP syntax.

Besides this, we should ensure that all tagged tasks get done.

#46.1: unit test what exactly? It already has unit tests to ensure that the metadata parsed is returned properly.

#46.2:
I don't know how I could have missed that 😱

Probably because my IDE allowed me to extend it as there was no PHP syntax preventing me from doing so. I don't recall actually looking at the doc block of that class now that I think about it.

I'm getting really frustrated that Symfony puts "final" on everything and makes all their properties private. It makes extending its code next to impossible. I get that is the point, but it's also laughably ironic since that's basically what they said we should do.

How can we extend something that is, at the same time, not extensible? Seriously!

Should create our own Source interfaces that basically do the same thing... creating a menagerie of boilerplate code that has to convert to proper "Twig\Source" objects and back again?

This is exactly what I was trying to avoid in the first place and why I opened the Twig issue. This isn't DX friendly nor performant in any way.

Should we try to re-open that issue and force them to eat their own dogfood (i.e. you tell us to extend but then finalize every bit of code)?

Grr, very frustrating especially since this was already green.

I've made another issue, I guess we'll see how it goes...

https://github.com/twigphp/Twig/issues/3141

Clearly Symfony/Twig doesn't want to even remotely entertain the idea of relaxing their control over... anything.

It's a real shame and completely unnecessary to lock it down the way they have.

So instead of something streamlined and performant, we'll have to implement a completely decoupled approach for this.

I'm not posting an interdiff as this is basically a whole new approach (again) to this issue.

This is also utilizing the undocumented {% line \d+ %} approach that was suggested by Symfony/Twig.

Granted, this is likely nowhere near as performant as I'd like it to be, but considering that Symfony/Twig is basically thumbing their noses at this, there's really not much else that can be done.

1. +++ b/core/lib/Drupal/Component/Utility/FrontMatter.php
   @@ -0,0 +1,155 @@
   +  /**
   +   * A serializer.
   +   *
   +   * @var string|\Drupal\Component\Serialization\SerializationInterface
   +   */
   +  protected $serializer;
   ...
   +   * @param string $serializer
   +   *   A class that implements
   +   *   \Drupal\Component\Serialization\SerializationInterface.
   ...
   +   * @param string $serializer
   +   *   A class that implements
   +   *   \Drupal\Component\Serialization\SerializationInterface.
   

   This is always a string. It's a class name that is a subclass of \Drupal\Component\Serialization\SerializationInterface but it's never an object (which this implies it is).

   Hmmm... actually I guess with the current code it could be an object. Perhaps the @param for ::construct and ::load need changing from string to string|\Drupal\Component\Serialization\SerializationInterface.

2. +++ b/core/lib/Drupal/Component/Utility/FrontMatter.php
   @@ -0,0 +1,155 @@
   +  /**
   +   * The source.
   +   *
   +   * @var string
   +   */
   +  protected $source;
   ...
   +   * @param string|\Twig_source $source
   +   *   A string source.
   

   Hmmm... should we support Twig_source here - or should we require that the caller does $source->getCode() first. I think it might be nice to have no mention of twig in this class.

3. +++ b/core/lib/Drupal/Component/Utility/FrontMatter.php
   @@ -0,0 +1,155 @@
   +  public function parse() {
   

   I don't think this needs to be public API. The other getters seem really nice rather then returning an array with hardcoded keys.

4. +++ b/core/tests/Drupal/KernelTests/Core/Theme/FrontMatterTest.php
   @@ -0,0 +1,198 @@
   +  public function testCorrectedFrontMatterLineNumber(array $yaml, $errorLineNumber) {
   

   Great to see this tested.

5. +++ b/core/lib/Drupal/Core/Template/TwigEnvironment.php
   @@ -101,6 +103,33 @@ public function __construct($root, CacheBackendInterface $cache, $twig_extension
   +    // Note: always use \Drupal\Core\Serialization\Yaml here instead of the
   +    // "serializer.yaml" service. This allows the core serializer to utilize
   +    // core related functionality which isn't available as the standalone
   +    // component based serializer.
   +    $frontMatter = FrontMatter::load($source, Yaml::class);
   

   If the yaml fails to parse what should happen here? I think we need to convert the \Drupal\Component\Serialization\Exception\InvalidDataTypeException into something useful so the developer can find the broken twig file.

6. I think we need unit tests of the FrontMatter class outside of twig testing
7. I think we need tests where the yaml in the front matter is broken.

#51.1: I mostly did this for IDE typing. But yes, it could theoretically be an instance of a serializer as well (i.e. service). I'll change the types for __construct and load.

#51.2: Yes, I was warry of this as well. I agree Twig shouldn't be referenced here. Given that 8.8.x is now PHP 7+ we can typehint this as a string and force it on the implementation side.

#51.3: Whoop, agreed, it should be protected.

#51.5: It's intentionally ignoring any exceptions in FrontMatter. Originally, it would have been caught by Twig (as it was part of the source object). Now that this is decoupled, we'll need to wrap it.

#51.6: Yes, now that this is decoupled, it needs its own test.

#51.7: Agreed, this will address #51.5.

Ok, this addresses all of #51.

1. +++ b/core/lib/Drupal/Component/Utility/FrontMatter.php
   @@ -45,13 +45,13 @@ class FrontMatter {
   -   * @param string $serializer
   +   * @param string|\Drupal\Component\Serialization\SerializationInterface $serializer
   

   I think we should only support strings... less complexity in tests and we can have a primitive typehint which is nice.

2. +++ b/core/lib/Drupal/Core/Template/Exception/FrontMatterParseException.php
   @@ -0,0 +1,37 @@
   +    // Extract the line number from the serializer error (if possible).
   +    // Add 1 to account for the opening front matter separator (---).
   +    if ($previous) {
   +      preg_match('/line:?\s?(\d+)/i', $previous->getMessage(), $matches);
   +      if (!empty($matches[1])) {
   +        $line = (int) $matches[1] + 1;
   +      }
   +    }
   

   This looks neat. Potentially a little fragile - relying on the exception message parsing.

3. +++ b/core/tests/Drupal/Tests/Component/Utility/FrontMatterTest.php
   @@ -0,0 +1,163 @@
   +   * @expectedException \InvalidArgumentException
   +   * @expectedExceptionMessage The $serializer parameter must reference a class that implements \Drupal\Component\Serialization\SerializationInterface.
   ...
   +   * @expectedException \Drupal\Component\Serialization\Exception\InvalidDataTypeException
   

   A while back we switched from the annotations to ->expectException...

4. The test coverage is looking more and more complete. That's great.

#54.1: I'm a little frustrated that serializers can be either. While core may use static serializers, contrib/custom may use objects. I'm not sure what restricting this actually gains us other than one less test (which isn't that hard to implement).

#54.2: Oh it's very fragile... but there's really not much that can be done here as the line number that's returned in the exception isn't correct and the YAML messaging is rather vague at best.

#54.3: Ah, ok.

Addresses #54.

Re #54.1: I went ahead and made the change regardless. I just want to get this in.

After discussing this a bit with @chx in slack, we were able to simplify the regular expression and make it less fragile.

He also said there was no need for premature optimization by checking the beginning of the contents using substr, just use preg_match and be done with it.

Yes because optimizing for speed at template compilation seems a bit pointless.

Or during plugin discovery. All this stuff will get cached anyway and whatever very minor perf bump this may bring I don't think outweighs the massive benefit this ability brings us overall in the long run.

If I may nitpick a little more...

1. +++ b/core/lib/Drupal/Component/Utility/FrontMatter.php
   @@ -0,0 +1,162 @@
   +      throw new \InvalidArgumentException('The $serializer parameter must reference a class that implements \Drupal\Component\Serialization\SerializationInterface.');
   

   I would use assert here but of course that's just me, if we have similar elsewhere then keep it. Especially now that we are PHP 7.

2. +++ b/core/lib/Drupal/Component/Utility/FrontMatter.php
   @@ -0,0 +1,162 @@
   +      $this->parsed['line'] = count(preg_split('/\r\n|\n/', $matches[1])) + 3;
   

   We already established to use \R for linebreaks instead of \r\n|\n. https://www.pcre.org/original/doc/html/pcrepattern.html#newlineseq it's short for (?>\r\n|\n|\x0b|\f|\r|\x85)

   Also take note https://www.php.net/preg_match_all "Returns the number of full pattern matches (which might be zero), or FALSE if an error occurred" thus you can replace the count(preg_split()) with preg_match_all. $this->parsed['line'] = preg_match_all('/\R/', $matches[1]) + 3

It seems like a Twig template could have another '---' line in it (someone might want that as a separator, although I am sure they should use an HR tag instead!).

This regex works with other --- separators, both in the front matter YAML and in the content:

https://regex101.com/r/Oe4BuC/1

This is because it only matches the separators that are on new lines (by themselves) and only the first found front matter "block".

My other concern about the regexp is that we have standards for Twig templates that say that they should have @file tags at the top inside comment blocks (although we are not currently doing that for help topic Twig templates).

This standard will need to tweaked to mention that the exception to this is the inclusion of front matter metadata for the reason following.

It seems like it would be better for the @file comment block to be first in the Twig file, before the front matter (if any)?

No. That defeats the purpose of front matter then because this would essentially be an injection of YAML somewhere (arbitrarily) inside the contents of a file.

Front matter is intended to be a fusion of what could be two separate files, but with the convenience of a single file: static metadata + original file contents.

The former essentially being prepended to the file in question using the wrapping --- separators to denote the content inside it as YAML metatdata and anything after that as the contents of the actual "file".

That being said, the standard of keeping @file at the top of the "contents" section is still relevant as it technically belongs to the file itself, not the front matter metadata.

So in the case of a Twig template, it should look something like this:

---
key: "value"
---
{#
  @file This template's DOXYGEN comment.
#}
<div ...>

In fact, I would imagine that the API module would simply need to run the contents of a file through a similar class proposed here that splits the front matter data from the content before parsing it through the API module. Then just create a new "Front Matter" section or something for these files.

Also, the coding standards automation tool would likely need to detect front matter data as well so it doesn't send out false positives.

The regexp would get the "some Twig stuff" as part of the front matter I think? And fail parsing.

No, it doesn't.

https://regex101.com/r/Oe4BuC/2

My concern about the front matter going before the @file comment is that we normally have a standard saying the @file comes first , and it is confusing DX if it's "well not always".

And it should still come first, as part of the "content" portion of a front matter infused file.

I don't think it's that complicated to just expand the documentation to state that front matter is considered separate, yet inlined, YAML data. Coding/documentation standards apply to the content portion of any file that is prepended with front matter.

This addresses #60 and #65.

Modified the regex just a tad to deal with empty sections and added tests.

Also, decided to rename the "code" property/method to "content" as it could be anything... not just code.

This just consolidates and cleans up the tests and code a bit more.

Re: documentation. Where exactly should this be done? In code and somewhere like https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Render%21...?

Or should a new sub-topic be created in https://www.drupal.org/docs/8/theming/twig?

I also went ahead and created a CR: https://www.drupal.org/node/3087114

Another use case would be signatures. Not sure why Twig doesn't provide a mechanism for it but I think it doesn't. I copied out an include from a template of ours, this include statement with literally a dozen arguments occurs 13 times so adding in place documentation and enforcing would be quite welcome.

I still think this could get in before the first alpha.

This is basically just a simple addition to provide support for it. It doesn't really impact much.

Here's a patch that adds some simple documentation. We can expand on it more as time goes on.

I still think this could get in before the first alpha.

Sorry, to clarify. I think we should get this in before the first alpha. That way more people can test this during alpha and we can continue to improve this moving forward with real-world use cases (see IS).

@jhodgdon, does this satisfy the documentation requirement?

#75.a: Fixed, this was just a c&p from the Jekyll website FWIW

#75.b: Fixed, it's a string because serializers implement static methods. I originally supported class instances, but it was suggested in #54 to make it more simple and just use the primative string typehint.

#75.c: Fixed

#75.d: Fixed, renamed to FrontMatter::create() (naming things--). FWIW, #3069109: Replace help_topic meta tags with front matter made it in as "load". I decided to keep it for consistancy sake, but I suppose its better to change this now to the correct terminology considering that the help_topics implmentation is internal. It's fine.

#75.e: Fixed

#75.f: Fixed, shame that it doesn't. I really think it should have the @throw in there so I just c&p the entire upstream doc block and reformatted it.

+++ b/core/lib/Drupal/Core/Template/TwigEnvironment.php
@@ -101,6 +105,53 @@ public function __construct($root, CacheBackendInterface $cache, $twig_extension
+      if (($line = $frontMatter->getLine()) > 1) {
+        $content = "{% line $line %}$content";
+      }
+      $source = new \Twig_Source($content, $source->getName(), $source->getPath());

I think we only need to create a new source when if (($line = $frontMatter->getLine()) > 1) { is TRUE - right?

Addresses #77 and #78.

Just realized that this can't be in Drupal\Component\Utility because of its dependency on the Drupal\Component\Serialization component.

This has to be its own standalone Drupal\Component\FrontMatter component.

Also including the appropriate meta files other components have.

Just wanted to ask that if front matter support in Twig templates is added to Drupal, is the idea to eventually deprecate .layouts.yml and .libraries.yml files? Or would they remain as an alternative way to add metadata?

The response from Symfony developers to @markcarver's proposal is saddening, but I would still like to think that interoperability between projects is a goal worth pursuing. And that it would be great to come up with non Drupal-specific solutions to this and other Twig/frontend issues whenever possible.

I see a huge value in being able to create pure Twig templates free of drupalisms. This makes it easier for HTML/Twig developers not familiar with Drupal to write templates, and also makes it easier to use the same templates in tools like Pattern Lab, or even on other platforms like WordPress and Symfony apps, for example. It is this possibility to use Twig templates in other projects that makes them different from PHP files, where annotations do not make the files any more Drupal-specific than they already are.

If Symfony does not want to support metadata in Twig templates, I personally don't see a huge problem in continuing to use separate metadata files like .layouts.yml and .libraries.yml. A dream of mine for several years now has been to have a generic solution for component metadata that could be supported by several projects. Many requirements are not Drupal-specific, and even the Drupal-specific ones could perhaps be just added as an extension.

I personally don't see a huge problem in continuing to use separate metadata files like .layouts.yml and .libraries.yml.

+1. I like the way yaml definitions work in core. I would event propose remove plugin annotations in favour of yaml discovery. Of course that's a matter of personal preferences.

If it is essential to have metadata in the Twig files themselves, has adding it inside {# Twig comments #} already been considered? This would keep the templates portable, unlike when using front matter.

@jhodgdon Thanks, that does makes sense to me also. It is the front matter format that worries me the most, if Symfony won't add support for it in Twig. That would mean that even though front matter is used elsewhere, in Twig templates it would be a drupalism, and the templates would not work as-is in tools like Pattern Lab for example. A comments-based approach just like in PHP files would be better at least in that sense.

Yes, if layouts need both meta-data and a Twig template, it seems like they should use template-based discovery with the YAML for the plugin in the template.

Layouts may also have associated libraries and PHP class which would be tricky to embed in Twig template.

To me, it makes sense that if a plugin needs a class, you should put the meta-information for the plugin in annotation in the class doc header. If the plugin needs a Twig template, you should put the meta-information for the plugin in Front Matter at the top of the Twig template. If the plugin only needs textual meta-information, you should put the meta-information into a stand-alone YAML file.

That means we will have at least three different ways to define meta-information which will make plugin system even more complicated.

@jhodgdon I had a look at that issue, and it seems that only HTML comments were discussed, not Twig comments? And from the examples provided Jekyll uses Liquid templates, Hugo uses Go Templates, and Grav doesn't seem to use front matter in Twig templates, only in markdown files? So even if front matter is consistently used with --- delimiters, this would not seem to be the case with Twig files, and would cause problems if the Twig files were used anywhere else than Drupal.

I can see the confusion with how this issue started (attempting to streamline this with Twig).

However, as I did more research, the more it became clear that Twig was just a medium for providing the metadata.

Ultimately, Twig comments are not an industry standard for inlining metadata, whereas front matter is.

Also, front matter can be used with any document, where as Twig comments would be restricted to just Twig templates.

That's why this has morphed into a stand-alone component, with a very minor Twig integration.

I don't think we need to rehash this.

#81.a: Because it's an Exception, which already has a "line" property/getter. As the comment says, it returns the correct source line (not the exceptions). I really don't know how else to say that.

#81.b: Because, just like the Front Matter line offset, it adds to the current line number (1) to account for the opening --- separator.

#81.c: The only other class this does this is Drupal\Core\Site\Settings and even that doesn't make sense. I'd rather we didn't muddy the utility namespace with this as it's not actually part of that component (which are separate composer packages).

---

I don't have this issue on my local right now, I'll have to wait until I get back next week to address any follow-ups... but the above seems like rather trivial nitpicks.

It's just hard to know what some people will understand and what some won't. To me, it makes perfect sense within the context of the classes, but I get how it can still be confusing :D

Like I said, I won't be able to get to the above until next week. If you feel like making these small changes to help move this issue along while I'm traveling and recuperating, then please go ahead :D

No point in us making this a thing.

I have nothing against front matter in general or using it in Drupal-specific files. It is great that Drupal will have a stand-alone component providing support for front matter for all kinds of use cases.

However I've understood that this issue is specifically about allowing Twig templates to use front matter? Although front matter is an industry standard for adding metadata, using it does not automatically keep the files interoperable.

As front matter is not supported by core Twig, adding it to Twig files would make them non-standard. This is especially problematic because other systems will just display the front matter, which is not a desired outcome.

The point I wanted to make is that since Drupal Twig files can be used (and are actively being used) in systems beside Drupal, adding front matter to Twig files means that this interoperability will be lost, unless support is added to those systems as well.

If the decision is to do this anyway, so be it. I just thought I should inform that it might cause genuine problems for some. I wish I could have raised this concern earlier, but I only learned of plans to add front matter to Twig files from @lauriii's session at DrupalCon Amsterdam this week.

Edit: Ok, I see now that the component is being added in this same issue? I would say that the issue title is slightly misleading, although I am guilty of not reading everything before commenting. Just to be clear, I have no concerns about the component itself or using front matter in Drupal-specific files.

That's odd, didn't mean to change the status.

As front matter is not supported by core Twig, adding it to Twig files would make them non-standard.

A decision that was made for us. I tried convincing them to support a "native" way to do metadata in Twig templates. They have made it quite clear they don't care about this issue. They even said that we should implement this in core ourselves (albeit, with suggestions that are near impossible to due given their finality control over their classes). So I'm not sure how their recommendation becomes "non-standard" when it's the only solution we have here.

This is especially problematic because other systems will just display the front matter, which is not a desired outcome.

I have an issue with this. Primarily with the implication that this adds responsibility to core where it now must somehow support other systems for the templates it owns/consumes. No, if another system chooses to consume templates intended for Drupal, then the responsibility of supporting this will need to be taken into account.

If the decision is to do this anyway, so be it. I just thought I should inform that it might cause genuine problems for some.

I think you're confusing the issue here. This is an opt-in feature. Not all Twig templates in core are going to have front matter. Only specific templates (help_topics, layouts, themes that have to support multiple versions of templates, etc.) would actually benefit from this.

Given that these sub-systems actually require Drupal (and not intended to be used outside of Drupal), this really isn't an issue.

The benefits this gives the theme system is enormous and certainly outweigh any minor inconvenience this my cause 3rd party consumption of Drupal intended templates.

Regardless, that is why this is being made into a standalone component. One that can easily be used by other software as needed.

So I'm not sure how their recommendation becomes "non-standard" when it's the only solution we have here.

I did not mean the solution, just the files themselves. By non-standard I meant files that do not work correctly with just Twig core. I know we have Drupal-specific filters and functions too, but using them is not required.

I have an issue with this.

All I am suggesting is that whenever possible we should try to think about interoperability. Pattern Lab for example is a very popular tool used by Drupal front-enders to develop templates for Drupal.

I think you're confusing the issue here. This is an opt-in feature.

Whether this will remain an opt-in feature or not was, in effect, my first question in this issue. In his DrupalCon session @lauriii mentioned that Twig front matter is being considered as a general solution to avoid BC breaking changes in themes.

Given that these sub-systems actually require Drupal

Basically any Twig template could be developed in an external tool that does not understand Drupal-specific things.

The benefits this gives the theme system is enormous and certainly outweigh any minor inconvenience

I am not so certain about the benefits outweighing those of interoperable Twig templates. Also not so certain about the level of inconvenience. A standalone component can help in adding support to other PHP-based projects, however this support would still have to be added by someone to every other project. There is also a lot of innovation going on related to using Drupal Twig templates in JavaScript.

If we don't give any consideration to how other projects interoperate with Drupal, and shift resposibility to them, I don't see how that is different from the response we got from Symfony.

@markcarver Twig integration in the current patch is based on extending Environment::compileSource(). If one wanted to add identical support for Twig front matter to other projects, how would you recommend doing it? A sincere question, since I would like to try it out with Pattern Lab.

For example Pattern Lab currently allows access to and even setting the Environment object used, but I don't think there is a way to change the class it uses to originally create and initialize an Environment. And I believe an Environment decorator class would not work, because compileSource() is called internally? So it seems to me that the current Environment access/plugin type of approach seems impossible, and instead changes to the core project would be needed.

I'm asking this here because if we decide to create a non-standard Twig file variant, hopefully we'd at least make it as easy as possible to support it in other projects.

If one wanted to add identical support for Twig front matter to other projects, how would you recommend doing it?

File an issue with said project.

So it seems to me that the current Environment access/plugin type of approach seems impossible, and instead changes to the core project would be needed.

Despite my better judgment, I went ahead and took a look at the following:

https://github.com/pattern-lab/patternengine-php-twig/blob/edeb27a56a7a3...

So, in its current state, yes... it would be considered "impossible" due to the fact that they're hardcoding this \Twig_Environment instance with no way for it to be overridden/replaced.

While this issue is adding support for this in Drupal core, in reality, contrib could even do this by replacing the entire Twig instance (service) to be via container alterations. The fact that this kind of extensibility isn't present in another system just goes to show how different they really are.

That being said, that project is open-source. Filing a PR to add support for this wouldn't be that difficult once this issue is in. It would require the drupal/core-frontmatter component (which this issue is creating). So until this issue makes it in, you really won't be able to do anything (unless you decided to go another 3rd party route like was previously done earlier in the issue).

I'm asking this here because if we decide to create a non-standard Twig file variant, hopefully we'd at least make it as easy as possible to support it in other projects.

As stated above, it would just be a matter of creating simple PR in that project to support it.

In the case of Pattern Lab's Twig implementation, I would go even as far as allowing the environment class to be overridden via config (which isn't currently possible). If no config is present, then it defaults to a custom one that could implement this. It would be up to whatever/whomever overrides this Environment class to also extend from the custom pattern lab implementation so they don't lose the Front Matter support.

The current implementation is "as easy as possible", primarily due to the way Twig has locked down a lot of its classes/methods/properties/APIs. It leaves very little room in when and where this can be implemented and there's nothing I can do about that. Symfony's attitude towards/interpretation of PHP is very different from Drupal's.

Overall, it isn't very complex. It's just sub-classing the Twig environment and then semi-overriding the methods needed (like this patch is doing).

Also, please stop saying "non-standard". This isn't changing Twig.

Front matter is just a fusion of YAML and whatever the original document was. It prepends itself to the original document; allowing it to simply live in the same file for easier maintainability.

As I said above, the Twig templates using Front Matter in Drupal core are likely not going to make their way into Pattern Lab.

If, for whatever reason they do, then support for Front Matter can easily be added to that project.

You're making this a far bigger deal than it really has to be.

Code changes. Features are implemented. Adaptation occurs. It's the lifecycle of open-source.

---

Getting back to the issue at hand... setting CNW and assigning to myself to implement #81.

Missed one part of making this a new component.

@markcarver Thanks for your reply!

they're hardcoding this \Twig_Environment instance with no way for it to be overridden/replaced.

Actually the \Twig_Environment can be replaced by plugins in Pattern Lab, but then the current state with possible added extensions etc. would be lost. And it seems to me that a decorator class can't be used because of the internal compileSource() call?

I would go even as far as allowing the environment class to be overridden via config (which isn't currently possible).

Overall, it isn't very complex. It's just sub-classing the Twig environment and then semi-overriding the methods needed (like this patch is doing).

This would work, but there would be a problem if two or more plugins required their own subclass, so it doesn't seem like an ideal solution.

Also, please stop saying "non-standard". This isn't changing Twig.

It isn't changing Twig, but it is changing the Twig file, making it work incorrectly with any tool or system that doesn't support front matter in Twig files.

As I said above, the Twig templates using Front Matter in Drupal core are likely not going to make their way into Pattern Lab.

Did you have a look at the DrupalCon session linked in my comment? I might have misunderstood, but I got the impression that front matter is being considered for all Twig files in the future?

If, for whatever reason they do, then support for Front Matter can easily be added to that project.

I'm trying to find a good and easy way to add it to Pattern Lab. I am a member of the Pattern Lab developer team and the first person to fully integrate Pattern Lab and Drupal. Help and advice from the Drupal project would be much appreciated, and would benefit the many Drupal front-enders who are actively using Pattern Lab for Drupal Twig development.

You're making this a far bigger deal than it really has to be.

I am sorry if I have come across as impolite or a nuisance, that was never my intention. I am only hoping to offer a perspective on what this change can cause off the island. And I am already trying to adapt to it. This would indeed be a great chance to prove that Twig front matter support can be easily added.

But I do want to say that we should not be upset by Symfony's responses to us if our own responses to other projects are not that different.

>
Another question, are we requiring a PHP version that supports this now?

Nope. That's 9.0. You are correct. public function getSourceLine(): int is not supported by D8.

Nevermind, I am wrong, we are on PHP 7.0.8 now, I mixed this up (we had a feature discussion with larowlan the other day but that was a 7.1 feature). #3053363: Remove support for PHP 5 in Drupal 8.8 So yeah, it's a new PHP feature but supported.

Actually the \Twig_Environment can be replaced by plugins in Pattern Lab, but then the current state with possible added extensions etc. would be lost.

I didn't see that anywhere, but then again, I really don't know Pattern Lab that well.

And it seems to me that a decorator class can't be used because of the internal compileSource() call?

Then that is a bad decorator. The point of a decorator is that it implements the same type of interface of the object it's decorating. The method in question isn't internal, it's public. It should be able to provide the same type of method, do its thing (like it is here) and then just pass along to the decorated environment.

This would work, but there would be a problem if two or more plugins required their own subclass, so it doesn't seem like an ideal solution.

If it's truly so common that a \Twig_Environment instance is overridden/customized, then a decorator pattern does seem the way to go for Pattern Lab.

It isn't changing Twig, but it is changing the Twig file, making it work incorrectly with any tool or system that doesn't support front matter in Twig files.

As I said before, if this hypothetical "tool or system" encounters Twig templates that are using front matter, it will be able to implement this component once it's in. This, of course, is on the assumption that its Twig templates encounter front matter at all.

Did you have a look at the DrupalCon session linked in my comment? I might have misunderstood, but I got the impression that front matter is being considered for all Twig files in the future?

Yes, I was actually there. And yes, you're misunderstanding this issue.

I think it's important to state the following: all this issue actually does is simply allow the possibility for templates to associate metadata with their template for sub-systems that would actually use it (i.e. help topics, layouts, etc.).

This means that said metadata has to exist, have a real use-case for it to exist and have some sub-system that would actually consume the metadata associated with the template.

Why would "all Twig files" have front matter? What would that even be if that were the case? This isn't adding front matter to every Twig template, it's just allowing the possibility.

Help and advice from the Drupal project would be much appreciated, and would benefit the many Drupal front-enders who are actively using Pattern Lab for Drupal Twig development.

I'd be happy to help implement this for Pattern Lab, but as I stated before... I think it may be overkill. It's highly unlikely Pattern Lab (or the Drupal [sub-]themes that integrate with it will actually ever encounter it.

This is based on the assumption that Pattern Lab is primarily focused on the individual components. Things like help_topics templates aren't filled with components, but rather the verbiage for that topic (maybe some minor inline formatting tags).

In the case of layouts, it would still have to support the YAML based file definitions, so a sub-theme could easily use the deprecated format and simply not use front matter at all.

I am sorry if I have come across as impolite or a nuisance, that was never my intention. I am only hoping to offer a perspective on what this change can cause off the island. And I am already trying to adapt to it. This would indeed be a great chance to prove that Twig front matter support can be easily added.

Not at all, I do value the feedback. I'm just not convinced that it is the issue you have been making it out to be. It's also more of a Pattern Lab issue that needs to be discussed on how best to implement/support it, if or when it ever needs to be. I think there was confusion on both parts, but I think we're now at a place where we understand each other.

But I do want to say that we should not be upset by Symfony's responses to us if our own responses to other projects are not that different.

Actually, they're quite different. Symfony basically said nope and has really not provided any help whatsoever. Their components and code are highly controlled and managed, to the degree that makes them practically impossible to sub-class, decorate or extend. Basically: use it their way or not at all.

Whereas, in Drupal, nearly everything is alterable, extensible, overrideable. In fact, even after this goes in, you could still replace the normal Twig service with your own and completely remove this front matter support. The class introduced here is extensible, not final or private.

Even if you don't want to actually "support" front matter, you could still implement the parser to strip it from any Twig template. There are lots of options, it's just a matter of figuring out what to do and creating a PR to implement it.

Ultimately, I think any further questions or discussion surrounding Pattern Lab should be done in an issue for that project. I don't think that hashing it all here now in this issue is the best place.

I understand the initial concern, but I've thought it over and it really isn't as big of an issue that it was originally made out to be.

---

Re: public function getSourceLine(): int

I discussed this with @chx in slack and ultimately I think this should come out for right now for a few reasons:

1. Core doesn't use return type hints currently anywhere else in code, despite the PHP version being sufficient in >=8.8.0.
2. The patch is technically already inconsistent (in this regard). Either all methods should use return type hints or this one should be removed, I'm opting for the latter.
3. Ultimately, this was just a mistake added by my IDE that I overlooked when switching environments one day.
4. I don't want to rock any boats, would just like to get this in.

However, I have not reviewed the composer.json, or the license and readme files (not familiar with the requirements for Drupal\Component namespace). If someone can point me to docs on our policies with respect to that, I would be happy to review. If there are no docs on that, then someone familiar with those policies will need to review.

I too would like to know if there's some documentation around this. I couldn't find any.

That being said, I just looked at the other existing components and saw all these files were technically checked into the repo:

https://git.drupalcode.org/project/drupal/tree/8.8.x/core/lib/Drupal/Com...

I c&p them and just modified them for use with the FrontMatter component.

edit: The closest I found was the IS of #1826054: [Meta] Expose Drupal Components outside of Drupal

Then that is a bad decorator.

What I mean is that although I can provide the same method in the decorator, and pass calls to the decorated object, this does not help when the method is called internally by the decorated object itself. In the patch the method is overriden in a subclass.

If it's truly so common that a \Twig_Environment instance is overridden/customized

I do not know how common it is, probably not very common, but I would not want to be forced to knowingly make a bad architectural decision.

The best option I can think of is adding front matter support to the current Environment initialized by Pattern Lab core. But this would mean adding a feature used only by Drupal to Pattern Lab core, and Pattern Lab is not a Drupal-specific tool.

As I said before, if this hypothetical "tool or system" encounters Twig templates

This isn't hypothetical, I mean any software that somehow supports Twig templates. Like JetBrains tools, for example, for which you have opened a Twig front matter support request. GitHub statistics alone show that 146,345 repositories currently depend on Twig.

I understand Symfony's strict position on breaking changes very well. They have the whole Twig ecosystem to consider, of which Drupal and its ecosystem is only a part. I agree with their position that adding front matter to Twig files is a breaking change, and strongly believe that Drupal should also treat it as such.

all this issue actually does is simply allow the possibility for templates to associate metadata with their template

Allowing the possibility is a hugely important decision as it could have major consequences in contrib, unless getTemplateMetadata() is marked as internal. And this possibility has already opened doors to discussion of wider adoption in core.

It's highly unlikely Pattern Lab (or the Drupal [sub-]themes that integrate with it will actually ever encounter it.

Pattern Lab can encounter any Twig file used in Drupal, including help topics files.

Actually, they're quite different.

What you are referring to is not what I meant, but never mind, this is not directly relevant to this issue.

I think any further questions or discussion surrounding Pattern Lab should be done in an issue for that project. I don't think that hashing it all here now in this issue is the best place.

I know it is unconventional to discuss another project's technical decisions in a Drupal issue, but I am doing this to provide just one example of what this change can cause in Drupal's ecosystem.

Anyway, I think I have now made my concerns clear enough. I have a high regard for our core committers, and trust them to make a considered decision, whatever it is.

This isn't hypothetical, I mean any software that somehow supports Twig templates. Like JetBrains tools, for example, for which you have opened a Twig front matter support request. GitHub statistics alone show that 146,345 repositories currently depend on Twig.

And I suppose that every one of those repositories is consuming Twig templates that are designed for Drupal? No.

Personally, I've never liked using "stars" as some sort of metric for how good a project is. It's arbitrary.

That being said, compare https://github.com/pattern-lab/patternlab-php-core which just has 42 stars. (edit: I know you used the "Used by" metic for Twig, but there isn't even an equivalent for Pattern Lab here... this is the closest comparison).

I get that components are the "next big thing". Unfortunately, Pattern Lab is just one implementation amongst many.

As with most "component" based implementations right now, each have their own approach and methodology.

Is Drupal core suppose to support all of them, even the relatively minor ones? No.

Nevermind the fact that any real web component-based solution that's adopted will likely require static HTML and not even support Twig (so there's that).

In regards to JetBrains: in that issue, there are links to other requests to support front matter, not just in Twig or Markdown, but also HAML and ERB files. To quote them:

YAML front matter looks like a common thing and perhaps it can be implemented in general for all the languages.

Front matter may have started out for just Jekyll markdown files, but it's highly portable and can easily be used/applied to other files and languages as well.

This is why it has become the industry standard (not "non-standard") for how one injects metadata into files.

I agree with their position that adding front matter to Twig files is a breaking change, and strongly believe that Drupal should also treat it as such.

Again, we're not changing Twig. The original issue I opened proposed a new Twig syntax which, I suppose, could be considered a BC by some (specifically the only two people who commented).

However, the reality is that this has NEVER been about BC because it would only ever work with newer Twig templates where any new addition to syntax/files would be supported; it's an addition, not a "change". The fact that the tests for all these patches in this issue since #41+ have passed is proof of that.

It's clear you have several issues with this issue. I'm sorry that I cannot seem to alieve your concerns about... well, anything. To me, it honestly feels like you just want to argue for argument's sake.

All I can tell you is that this is probably one of the most important additions to the Drupal theme system since... well Twig itself. It opens the door for so much down the road. Stuff that's actually needed for Drupal... not 3rd party consumers.

If you want to open an issue with Pattern Lab to continue discussing this topic and how it may be implemented, I'd be more than happy to provide assistance to and guidance so that it aligns with what we're doing here.

---

@jhodgdon, awesome :D I'll get on it.

😁

Great work here:

I suggested another upstream approach FWIW:

https://github.com/twigphp/Twig/issues/3141#issuecomment-549611111

Let’s see what they say to that.

Will give the patch a quick glance over now.

—

Edit: First of all as theme a ubsystem maintainer I do agree with using front-matter as it nicely combines YAML with Twig in one file.

I can see the problems with the templates but for anything not specific we can always do:

—-
Frontmatter: value
—-
{{ include(‘foo.html.twig’) }}

Which is a very nice way to combine re-usable templates with front-matter, so not worried there.

I do think what might need some work is how a template can be self-aware of its front matter (expose in context?) and that ideally we can get the front matter from the compiled template via a function call.

A way forward might be to mark the way of getting front-matter @internal for now as I neither wish to derail nor delay this important issue. And the component can well stand on its own.

I still think it’s possible with my outlined approach to get the desired effect of having the front matter included in the compiled template.

Not sure if it’s needed for our use cases though, I am just trying to outline a potential solution.

I do think what might need some work is how a template can be self-aware of its front matter (expose in context?) and that ideally we can get the front matter from the compiled template via a function call.

Interesting. So you're saying to instead of just removing it from the template, convert it into a twig tag/function? Perhaps it'd be easier to just set a custom local variable instead, that way it's also accessible in the template, e.g.:

---
key: 'value'
---
<div>{{ frontmatter.key }}</div>

Converted right before compilation to:

{% set frontmatter = {
  'key': 'value'
} %}
<div>{{ frontmatter.key }}</div>

This also brings up the need for having this metadata accessible in preprocess functions. However, given the scope of that endeavor, I've been hesitant to even bring it up in fear of derailing this issue. The primary cause of concern is the fact that theme engines in Drupal only really do one thing, procedural and only invoked towards the end of the theme render process.

(edit) Here are the two issues I created as follow-ups:

#3092385: Supply front matter metadata to preprocess functions
#3092387: Convert front matter metadata into a custom variable in Twig templates

A way forward might be to mark the way of getting front-matter @internal for now as I neither wish to derail nor delay this important issue.

I thought about this quite a bit already actually. I'm assuming you're referring to the added Twig method public function getTemplateMetadata($name). If that's the case, I don't see this method signature changing. Primarily due to the confines of Twig itself. There's no real way to inject or store this metadata on any of the objects it provides unless we wish to create our own Template class and use base_template_class (edit: something I'm really not looking forward to doing. A custom Source object, on the other hand, would be more appropriate IMO).

As #3069109: Replace help_topic meta tags with front matter showed in the HelpTopicDiscovery implementation, the Twig environment isn't even being used. It's just loading the files right there and consuming the front matter data:

$front_matter = FrontMatter::load(file_get_contents($file), Yaml::class)->getData();

I suspect we'll likely do something very similar in #3075427: Create TemplateDiscovery for plugin managers to use.

This added Twig method is more for convenience than anything (if already using the Twig service). All anyone really needs to know is the file path of the template in question and it handles the rest.

In theory, the component itself is primarily what matters. If we decide to deprecate TwigEnvironment::getTemplateMetadata() in the future because it just isn't being used like we think it will (which is very possible), I'm OK with that. However, I don't think it will change or expand to anything more than what it is currently.

Okay, so I spoke with @aleksip and he has some valid concerns:

- We are doing the same BC break (as minot as it might be) as Twig core does not want to do, because also for us someone could have used there own frontmatter for their custom project

- Aleksip's point is: We are defining a new file format here.

- Therefore we would need to ensure (for the start at least) that .html.twig files remain unchanged and the front-matter is only used for .help.twig files (for pure BC reasons) [unfortunately]

- I think the Frontmatter extractor component is nice and right for the help system, but it might not be the right choice for the overall theme system

- @Aleksip unfortunately also is right that implementing front-matter in the twig engine itself is problematic as you need to change the core, you cannot just define an extension to twig, which further brings us away from the goal of interoparability.

- I still think frontmatter itself is useful and the YAML syntax is nice for the help system, but as that has it's own discovery it can also use it's own loader (I know you started there originally) and then Twig does not need to know about it

Which brings us to (what you created as follow-up):

- A metadata tag as proposed and which has chances to get into Twig Core at some point (if there is widespread usage):

{% metadata {
  'key': 'value',
}
%}

As said before we can compile that to a public getMetadata() method on the template.

There are no problems with performance as we can just load the (compiled) template and call the method to return the key-value things and even if (method_exists($template, 'getMetadata') is not as clean as I'd like it will work.

- We can also implement the tag completely as an extension, which means others could just use this extension

- We can way easier export the metadata to the template itself

Setting to Needs review to give it some more thought (sorry, but Aleksip's points are too valid to just dismiss).

Not sure this is already said but (my2cnts)

- Why not make it a new file extension .help or .dhelp which gets preprocessed into Frontmatter plus twig file.
- how many *.help.twig are added to the current *.twig set? Is that helping a FE developer or gets in the way?

find d8/core -name *.twig | wc -l
     558
find d9/core -name *.twig | wc -l
     609

No, he's not right, Fabian... read above.

It's NOT A BC BREAK.... the tests PROVE that!

What happens when core decides to add a new twig function/tag/filter?

Any 3rd party code would have to also implement them if it wishes to use templates that were DESIGNED to be consumed in a DRUPAL environment.

The difference being this wouldn't actually cause an exception to be thrown as a missing tag/function would.

It'd just show up as extra "text" until their 3rd party code was adapted to account for it.

Somehow that's considered a "break", but a new core twig function/tag/filter wouldn't be?

It seems extremely convenient and hypocritical.

This is no different.

I think the Frontmatter extractor component is nice and right for the help system, but it might not be the right choice for the overall theme system

#2547363: Use front matter to version templates
#3075432: Inline .layouts.yml definitions with each template
#3050386: Allow loading CSS and JavaScript directly from templates

Why not make it a new file extension .help or .dhelp which gets preprocessed into Frontmatter plus twig file.

Just... no...

This doesn't solve anything and would just confuse matters more. The crux of @aleksip's argument is that they would still have files that are mostly Twig (and would have to be rendered via Twig), but would contain front matter at the top. Changing the extension wouldn't "solve" this, just skirt around the issue of them needing to support front matter in twig templates.

As I stated above though, this isn't any different from having to support a new core Twig function/tag/filter.

Honestly, the bottom line of @aleksip's argument revolves around who's responsibility it is to not interrupt (not "break" BTW) their automation workflow of 3rd party code.

I'm sorry, but it's extremely selfish to hold this issue hostage on something that I've said many times is easily addressable (and "fixable") in that 3rd party code.

Just the fact that https://github.com/pattern-lab/plugin-drupal-twig-components exists is proof that there is some sort of augmentation that has to keep parity with it's upstream Drupal core counterpart. Pattern lab could easily be extended to provide additional support for sub-classing, enhancing... or some other implementation to account for this.

The entire point though is that this is a PATTERN LAB INTEGRATION ISSUE; not core's responsibility.

Okay, let's forget any theoretical BC breaks.

I think the main controversy here is if Twig files in general should have front-matter or not (there is arguments for each position).

One argument against front-matter could be that it is usually (like here as well) used for annotating _content_ (jekyll, gatsby, grey-matter, ) and I agree that it's well suited for that.

An argument for front-matter could be that it's similar to annotations (though those _are_ in a comment) and arguably using {#--- ---#) as delimiter (and e.g. grey-matter supports variable delimiters) is not worse than ---/---, but would have no side effects of displaying the data accidentally.

But all of that is independent of Twig itself supporting it, so I agree with #124:

But I think if we take the TwigEnvironment part of this patch out, we will not lose much with respect to help topics -- we would still need the Twig Loader class in order to specify the directories our help topics can come from. If the TwigEnvironment class can handle stripping the front matter [from this patch], we'd be able to take out a few lines of code from the Twig Loader, but it's not that big of a deal to leave it in our custom class.

So maybe we get the component in first (we have consensus on that I think) and then potentially generalize to the Twig_Environment and further files?

#3069109-18: Replace help_topic meta tags with front matter

Commerce uses Twig for emails. I would love to have front matter. Would allow me to move the email subject and other relevant data directly into the template, and allow it to be customized that way.

TL;DR:

- I propose that we put the matter in a Twig comment

- If you need the metadata of a template delimited in that way, you use a front-matter library (lots are available) with custom delimiters support.

- if you don’t then Twig will just ignore the metadata as it is in a comment.

- This is not different how we parse metadata in PHP files.

- No changes to Twig are needed to support that

Okay, I researched front-matter some more and apparently the only thing that really is defined for front-matter is that it is content that is before anything else.

No one is against front-matter, it's more the how rather the why which was controversial.

What about if we indeed do use our own front-matter syntax:

{#---
YAML like usual
---#}

The disadvantage is that it's a custom front-matter-for-twig thingy and you need to change the component to take a custom delimiter for the start of the matter, but custom delimiters are well supported in the industry and worst case you could always do:

$data = FrontMatter::load(substr(2, file_get_contents($file)))->data;

Better however would be to use:

$data = FrontMatter::load($file, ['delimiters':['{#---', '---#}'] ])->data;

This has several advantages, too:

- No changes to Twig_Environment needed, the Twig environment will just ignore the matter

- Much more similar to how annotations work in PHP and are extracted via doctrine

- It's not a new hybrid file format, where the matter would be displayed, but still a standard Twig file.

- It reaches all the goals originally stated in the issue summary

- I spoke with aleksip and that would alleviate his concerns on interoperability between systems

- It's very simple to add a front-matter component to any other system and even JavaScript in case the front matter needs to be used by another system

In case we find it ugly, we could also do:

{#
---
Matter
---
#)
Twig Template

and just add some more code to our find-delimiter part (implementation detail, but looks a little more like the standard matter).

The code changes should also be pretty minor (mainly adding support for custom delimiters) and not waste much time.

Thoughts?

The reason they do that is, because they include usually content, not generic metadata.

Gray-Matter for example allows specifying the delimiter:

https://github.com/jonschlinkert/gray-matter#optionsdelimiters

The only reason for `---` is that it's the document separator for YAML documents (https://stackoverflow.com/a/28513112), but a Yaml parser that ignores invalid documents would still be able to parse the middle content of this:

{#
---
foo: bar
---
#}
<div class="{{ foo }}">

is invalid YAML, because 2/3 sections are invalid, but:

---
foo: bar
---
<div class="{{ foo }}">

is also invalid YAML, because 1/3 sections are invalid and one section is empty.

So in terms of interoperability setting a syntax highlighter to YAML will work for both cases rudimentary.

But for me the advantages of not having to change Twig (which is hard and we try to go back to more vanilla Twig overall), but instead slightly adjust the front-matter loader (which is very easy) outweight the custom prefix before the actual front-matter document.

It is very unfortunate, that we can't do:

#{#
---
foo: bar
---
#}
<div class="{{ foo }}">

as that would be valid YAML in the prefix, but that would still output an extra "#".

In the same vein we could be asking for front-matter in PHP instead of doxygen:

---
foo: bar
---
<?php
...

but there is a reason we don't do that, but instead use annotations.

That said, I'd love it if we had front-matter instead of annotations in PHP as well ;), but to be still valid to a PHP compiler it would need to look like:

<?php
/**
---
foo: bar
---
  @class Foo
*/

as I find the syntax of front-matter way more intuitive, then the @x = ....

Here we again have three YAML documents:

- An invalid prefix document
- A valid YAML document
- An invalid suffix document

I hope this clarifies the standpoint a little bit.

Here are the two PRs the would add support for this in Pattern Lab:

https://github.com/pattern-lab/patternengine-php-twig/pull/55
https://github.com/pattern-lab/plugin-drupal-twig-components/pull/12

@jhodgdon thanks for the work on reaching out to both sides of the discussion and summarising everything. That is super super helpful. Given there has been disagreement I'm going to post this issue to the other framework managers especially @lauriii as the frontend framework manager and try to get some time to discuss it. Your comment makes have that discussion much easier.

@alexpott Thank you for commenting and posting this issue to the other framework managers.

I am deeply saddened that recent discussion in this issue has not been very constructive. I still don't think my real concerns have been fully understood by all, but I am only blaming myself for not communicating them clearly enough.

I will try to summarise my concerns as well as I possibly can:

- To my knowledge front matter in Twig files is not currently supported by any software. While it will not break code, front matter in a Twig file will cause unexpected and undesired results in software that does not support it. Twig files with front matter could reasonably be considered a different file format.

- Standard Twig files without front matter are supported by a huge number of software projects written in different programming languages, including text editors and IDEs, small tools and large systems. Creating an unsupported variation of Twig files would mean a huge loss of interoperability for Drupal that would never be fully recovered.

- In the current patch front matter support is implemented in a \Twig_Environment subclass. Following this example can cause problems and force undesired architectural decisions for 3rd party PHP software wanting to add support for Drupal Twig files.

- Even if this issue is only about adding support for front matter in Twig files, contrib could already start using it. This issue opens a door that would be difficult to close again.

I would also like to emphasize that my concerns have nothing to do with Pattern Lab specifically. I have only used it as an example, and now deeply regret that I ever mentioned Pattern Lab in this issue. I am also in no way officially representing the Pattern Lab project. I am first and foremost a Drupal developer, wanting what is best for Drupal.

My main concern is losing our current interoperability with the entire Twig ecosystem. Even though Drupal-specific Twig extensions exist, it is currently possible to use plain interoperable Twig files for Drupal theme development. It is this possibility that would be taken away, even though there are alternative, interoperable ways to implement metadata for Twig files.

All I am hoping for is that these concerns would be fully understood and considered before a final decision, whatever it is.

Thanks for the summary!

Before a decision is made I’d like to give @aleksip a chance to post some thoughts he so far shared only privately - it was never about pattern lab.

I also want to clarify two things:

A) in no case this issue is closed, won’t fix.

B) The only capability the front-matter component would be gaining is support for custom delimiters.

IMHO we need to decide which world view we want to pursue:

The two world views are:

- Front-Matter is the standard => We need to adhere to it and Twig does not matter
- Twig is the standard => We need to adhere to it and Front-Matter we define as 2nd YAML document

IMHO both are generally fine and there are arguments for and against both.

- Twig is harder to tweak as an extension is not enough unfortunately.

- Frontmatter generally supports custom delimiters and an effort might be done to allow two or so lines of tolerance till the real document begins

The only controversy here is the Twig_Environment part.

Everything else is committable and useful.

Once Twig Environment gains that capability, everyone in contrib and custom can add front-matter to their Twig templates and they can no longer be reused.

This point is - I think - a summary of aleksips concern and I do share those somewhat.

Do we really want to support hybrid twig files in Core is a question we need to very carefully answer.

Once Twig Environment gains that capability, everyone in contrib and custom can add front-matter to their Twig templates and they can no longer be reused.

Is twig template re-use from a Drupal project to a non Drupal project a real or a theoretical thing?

It’s a very real thing - for example I know of a project that uses the layout builder files with a custom gulp pipeline and twigjs to show a preview of layouts together with some kind of styleguide. (Though indeed in that case the metadata would probably be needed as well)

But yeah - Twig files from Drupal are often liked for building style guides and allowing design teams to theme things without knowing Drupal.

There is also a (proprietary) project for previewing Twig files in an IDE while developing them. Sure they probably could add front-matter support as well, etc.

One of my goals here also is that I want metadata in all Twig templates (we really really need it - no doubt) - however we can also say as Jhogdon pointed out that we only use the standard front-matter for specialized content like things:

- help files
- E-Mail templates

But not for generic templates and reopen the discussion again for layout, plugins, maybe even components.

If we want an @metadata tag, standard front-matter or custom front-matter ...

Meantime it blocks Help topics test coverage
- #3037229: Extract help topics label from meta tag in Twig files
- #3086075: Use Twig to strip Twig syntax from help topics files in the syntax checker

The two world views are:

- Front-Matter is the standard => We need to adhere to it and Twig does not matter
- Twig is the standard => We need to adhere to it and Front-Matter we define as 2nd YAML document

This makes it seem like there is polarization and incompatibility between the two. There's really not.

- Twig is harder to tweak as an extension is not enough unfortunately.

Not true. Took me less than 30 mins to post the PRs in #136.

- Frontmatter generally supports custom delimiters and an effort might be done to allow two or so lines of tolerance till the real document begins

Also not true. Front matter does not support custom delimiters, that is something else entirely.

From https://jekyllrb.com/docs/front-matter/

The front matter must be the first thing in the file and must take the form of valid YAML set between triple-dashed lines.

While it isn't a "spec" per se, it is very explicit about 1) where it should be placed and 2) what form the delimiters take. This is what front matter is. Anything else is a non-standard implementation and isn't as widely supported.

---

FTR, this is exactly why I attempted to bring this issue up with Twig itself. Any, and I mean any, standalone "solution" that we came up with here would inevitably receive some sort of push back (it's so sad that FE issues always seem to get killed by just one person's opinion).

Symfony/Twig are the ones that told us we could (and should) implement this in our [Drupal] environment (specifically as a loader, but that was deemed overkill given it's not actually "loading" anything).

Thus, they're the ones essentially promoting and encouraging "interoperability" as some have put it.

I keep hearing about the "Twig ecosystem", but also completely and utterly baffled as most of the "ecosystem" and solutions pertaining to it are largely governed by whatever larger project is consuming the templates in the first place.

Each have their own data structures, functions, tags, filters, etc. How is this really any different?

Why? Because every project has its own specific needs. That's why you see issues about Twig for Drupal on https://drupal.stackexchange.com/search?q=twig on not elsewhere.

Claiming "interoperability" is a joke when that is literally the byproduct of using software created by people who encourage such customized and decentralized implementations instead of establishing standards for common problems (we weren't even the first ones to ask for this).

Seriously, can anyone say that they can take a Drupal Twig template and plop it into October CMS or Grav... or even Symfony itself? No!

Why is that?

Anyone who has actually worked with a templating system can tell you is that it's because it completely depends on the system that's rendering it in the first place.

Yes, Twig is a templating language. At the end of the day though, it's a customizable templating language. One that Drupal has been customizing literally since day one.

Just the fact that we have the ability to print render arrays directly in a template is a pretty big "interoperability" from conventional Twig files.

---

This issue has been weighing heavily on me. Trying to see and compromise where possible, but also recognizing that anything short of its current implementation will only introduce more DrupalWTFs (looking at you "custom delimiters").

I was reading https://github.com/pattern-lab/plugin-drupal-twig-components/issues/1

This got me to start thinking about how we might need to start approaching this from a more abstract way.

People can argue that using front matter makes it a "different file format" till they're blue in the face. It doesn't change the fact that literally any difference between stock Twig and what core does currently will result in a compilation error when it encounters things like create_attribute() function and the |t filter (the latter being very heavily used).

One could argue that these functions/tags/filters are also variations of said "file format"; meaning that it wouldn't compile if a system didn't support them (which is true).

The reality is this: Drupal has very specific needs in Twig (for a multitude of reasons). There's no getting around this fact and our needs will only likely continue to increase over time.

There are 3rd party implementations that will need the same type of environment/extensions that we're using here in core.

I think we should seriously start to consider decoupling and componentizing our Twig environment and extensions (obviously requiring multiple follow-up issues).

It won't be easy. But we can, at the very least, start to provide some basics that when certain core functionality isn't present (e.g. translation services) it at least falls back more gracefully than a compilation exception halting the entire process.

We should probably do this for twigjs as well.

This will give 3rd party consumers of our Twig files an easier way to support these customizations.

I'm starting to see that the genie is out of the bottle regarding the use of front matter for metadata. I now understand that the decision has in effect already been made in an earlier issue, and that other active issues are now being blocked by this one. I should have brought up my concerns much earlier, but unfortunately I did not know about these plans at that time.

I would not want to "hold this issue hostage" or "kill" it, even if I had the power to do that (which I do not of course have). I only wanted to express related concerns, which might have not been previously considered, and discuss them in a constructive way.

I think my preferred solution for metadata in Twig files would have been a tag ({% metadata %} or similar) because 1) it uses existing Twig syntax, 2) it can be added to existing Twig environments using a standard extension point, and 3) Symfony indicated that it would be the only way they could consider adding metadata support to core Twig. Edit: Since a comment-based approach would cause nothing to break even without support, that would be even better, if support could be added using a standard extension point. As it would not add new syntax, maybe Symfony might consider this approach for Twig core too, I don't know.

However, people will no doubt come up with ways to use pure Twig templates in Drupal even if Drupal's own (core or contrib) templates had front matter. And maybe those efforts are worth it so that front matter as defined by Jekyll can be used for the greater benefit of Drupal.

Yes, it is possible to use pure Twig templates (using no Drupal extensions) in Drupal today. I have built sites in this way for many years now, and there are many others who develop Drupal Twig files in a decoupled way, with other tools providing no or some level of support for Drupal-specific extensions.

And yes, templates like this are indeed used even in other CMSes. There was a case study on using the same templates in Drupal and WordPress just last week at DrupalCon.

@markcarver Thank you for taking time to study and even contribute to Pattern Lab. I am very happy that it made you consider a more abstract approach for the future!

I'm inviting everyone interested in Twig and Drupal's theme system to have a look at what a growing number of people are doing in the area of what is commonly referred to as component-based theming. To quote our BDFL, this approach is "worth exploring" at least! :)

Anyway, I have no wish to halt progress, so let us by all means continue with adding front matter support to Twig files, and hopefully consider effects on the use of pure Twig templates in future development of the theme system.

The decision that framework managers need to make (and I don’t envy them) is if we go with:

A)

—-
foo:bar
—-
<twig-template>

Change Twig to detect and strip front-matter automatically and leave front-matter alone

B)

{#
—-
foo:bar
—-
#}
<twig-template>

Leave Twig unchanged and change the front matter parser to use delimiters of "{#\n—-", "—-".

C) something else and decline both approaches

I think my preferred solution for metadata in Twig files would have been a tag ({% metadata %} or similar)

As I explained to Symfony/Twig, this requires the template to be rendered. Thus making #3075427: Create TemplateDiscovery for plugin managers to use impossible as it would have to somehow
know all the variables to send the templates when it's discovering them. This would also require us to use the twig service rather than just loading the file and parsing the metadata directly like help_topics is currently doing:

$front_matter = FrontMatter::load(file_get_contents($file), Yaml::class)->getData();

Yes, it is possible to use pure Twig templates (using no Drupal extensions) in Drupal today.

Yet, what you fail to mention is that this also requires significant changes to said templates to make them work this way (Drupal isn't "pure Twig"). Thus, work is already being done to remove Drupal specific features. An extra second to also remove the front matter (if it actually encountered it) wouldn't make any difference.

And yes, templates like this are indeed used even in other CMSes. There was a case study on using the same templates in Drupal and WordPress just last week at DrupalCon.

Again, not the point I was making. I didn't say it couldn't be done. What I was inferring is that you couldn't just plop one in another and expect it to just work. Some significant work would be required to make it abstract enough to work in both. See above.

@markcarver Thank you for taking time to study and even contribute to Pattern Lab. I am very happy that it made you consider a more abstract approach for the future!

Of course. Despite how my directness and frustration comes across, I'm not unsympathetic towards the goal here. However, there are certain realities that just are.

Denying the fact that we've already crossed the "interoperability" threshold with Twig only does all of us a disservice.

I'll admit that I have been extremely frustrated. Primarily due to conjecture, hypotheticals and just basic obfuscation of the facts.

I understand how complicated this subsystem/topic is. I'm one of a handful of people who probably truly understand the entire Drupal rendering process from A-Z.

It's a delicate balance for sure, but it doesn't mean I haven't thought a lot about how to implement this (I recommend re-reading how this issue has progressed at the beginning).

---

#145.B isn't front matter. Please stop saying that we can use custom delimiters.

1) Front matter isn't used like that anywhere else. Yes, there are parsers out there that support custom delimiters, but that just means it's not front matter. The entire point of front matter is that it has --- at the very beginning, some YAML in the middle and a closing --- on a separate line.

2) Also, this would require the user to remember to wrap it in comments for that specific language. Markdown doesn't have comments (aside from HTML comments) and it would only introduce inconsistencies between implementations. Front matter can be used with ANY file as long as it's consistent.

3) Placing YAML in a Twig (or HTML) comment means most IDEs would also simply ignore it, thus making the jetbrains issue to support front matter in files completely pointless. There wouldn't be any syntax highlighting or autocompletion using the Symfony plugin.

@aleksip, I feel I also need to clarify that I am not frustrated with you specifically; I really don't even know you.

My frustration mainly stems from the fact that core FE issues receive more scrutiny than any other core issues.

I have literally seen countless core FE issues get completely ignored, closed or end up in an endless bikeshed.

Requiring the entire history of the theme system to be brought forth as "evidence"... it's time-consuming and exhausting.

It's why, aside from Twig (which is literally the last step in the rendering process), the theme system really hasn't changed much since D6.

I do understand the need to think about the larger implications of what this issue entails; I did.

The benefits of what this could bring to our theme system far outweigh any minor (and temporary) inconvenience it may bring to other 3rd party systems consuming templates designed specifically for this system.

Something I tried to convey way back in #92 and #98... I guess I just didn't do that great of a job. Not sure if it was because I was about to head back home from Amsterdam and just didn't give it enough thought or... IDK.

I apologize if what I have said was taken personally. It wasn't intended as such.

I am pretty sure as many IDEs have annotations support that adding front-matter within comments could certainly be supported.

The - - - is historically the separator of two YAML documents. Only because some project defined originally that it needs to start with the delimiter does not mean that that is set in stone. If you define it as the 2nd YAML document in a file, you can still parse all other front-matter documents.

Regardless which decision is taken (A or B), I’d want to say that I am incredibly excited for meta data in Twig templates.

As everyone else I have a trillion use cases for that and can’t wait ;).

I fully agree that we need metadata support for/in Twig files, as soon as possible! I think that is the main point, and one that everyone agrees on.

This issue has been weighing heavily on me too, since I consider it such an important decision. So I will continue to offer some thoughts, but as I have said previously, I don't want to slow down progress, and I will accept the final decision, whatever it is. I only want to state these concerns to get them off my mind.

I think one thing that can easily cause misundertandings is that referring to standard/plain Twig could mean 1) just the Twig syntax or 2) a file rendered with just core Twig, or both.

So for example a {% metadata %} tag would be valid in case 1) but not in 2). And a --- front: matter --- block would be valid (with side effects) in 2) but not in 1).

Having read the related Symfony issues I believe that although being doubtful about the value of metadata for the entire Twig userbase, their primary position is that they do not want to add any new syntax. Problems with adding new syntax are that 1) someone might be already using that syntax in a different way, and 2) the files would not work properly in software not supporting the new syntax. I understand why this is a big thing for Symfony, and a somewhat less important thing for us. How important it is for us, well, let's just say I'm glad I don't have to make that decision!

Couldn't {% metadata %} and {% endmetadata %} be used as delimiters in the new component as well, without having to render the template? Using a tag (or possibly a comment and a NodeVisitor?) could solve #3092387: Convert front matter metadata into a custom variable in Twig templates too? And, by using a standard extension point, it could be easily added by 3rd party software, and could be considered by Symfony for core Twig.

Are there existing plans to use front matter in other types of files in Drupal? PHP? JavaScript? Something else? I don't think we could add front matter to PHP using the Jekyll standard because the execution of those files is out of our control. And JS could be problematic too?

I do agree with @Fabianx that using front matter in a hybrid file format is more or less a choice between fully supporting the two syntaxes. Our userbase is much larger than for example Jekyll's, so I feel we have a bigger responsibility than them for the consequences of our decisions.

I feel I also need to clarify that I am not frustrated with you specifically; I really don't even know you.

I apologize if what I have said was taken personally. It wasn't intended as such.

@markcarver I appreciate you saying this, thank you. I do understand your frustration very well.

Commerce uses Twig for emails. I would love to have front matter. Would allow me to move the email subject and other relevant data directly into the template, and allow it to be customized that way.

We have the first potential contrib user for metadata in Twig, and a major one too!

Again, this was just some further thoughts I wanted to express, that is all.

their [Symfony/Twig's] primary position is that they do not want to add any new syntax

That was for the initial proposal I made, yes (adding support for {@ in Twig, which has nothing to do with this current patch). However, their response was very terse and, from my point of view, mostly boiled down to the fact that they don't want to support a new feature/code. It has very little to do with actually solving the need for it. I don't think it does us any good to try and speculate any further towards their reasons towards front matter specifically as that wasn't the original goal of that issue.

Couldn't {% metadata %} and {% endmetadata %} be used as delimiters in the new component as well

No, because that isn't front matter. We cannot introduce a non-standard front matter component in core. That is why the "decision" in #145 is actually a fallacy.

The real decision to be made here is:

A) Go with the patch in #116, which just flushes out and finalizes a component and minor Twig implementation that was already decided on and committed in #3069109: Replace help_topic meta tags with front matter.

Or

B) Close this issue and revert #3069109: Replace help_topic meta tags with front matter since anything other than the current approach would introduce a non-standard component.

And, by using a standard extension point, it could be easily added by 3rd party software, and could be considered by Symfony for core Twig.

A normal Twig extension is too late in the process. It requires the template to be rendered, which, as I have stated before means that it would also require whatever is rendering the template (to get the metadata) to also know which variables to send to the template so it doesn't cause a compilation error. This is not an option; it never was. That's why I originally created the Twig issue to introduce something that could be used outside of the render pipeline; they declined.

Are there existing plans to use front matter in other types of files in Drupal?

Off the top of my head, yes, I can also see this being the basis for #2191525: [PP-1][policy, no patch] Extend Markdown coding standards to support API module (DOXYGEN). Without getting into too much of that issue, it basically allows markdown files to be used as topic pages in the API module. It's what we're doing in Drupal Bootstrap to generate our API documentation.

Currently, it's using a very non-standard mashup between HTML comments and the API module DOXYGEN @ keywords. It would make more sense to just use front matter here in retrospect.

To further expand that idea, front matter support could then easily become part of the https://www.drupal.org/project/markdown module as well.

I don't think we could add front matter to PHP using the Jekyll standard because the execution of those files is out of our control.

Not using normal include methods, no. PHP is/would be the only real exception due to the fact that it's the runtime language core runs on. Front matter is intended to only be used by code where it can be loaded/parsed/reformatted prior to consumption.

And JS could be problematic too?

Not necessarily. JS and CSS can be manipulated prior to consumption. We've already proven that with how we aggregate them in core. While I don't seen any immediate use case for using front matter there, in theory, it's not impossible.

I do agree with @Fabianx that using front matter in a hybrid file format is more or less a choice between fully supporting the two syntaxes.

It's not choosing between two syntaxes. It's fusing them; that's the beauty of front matter. Front matter allows the original language's syntax to actually remain intact because it's prepended to a document and stripped prior to its consumption. From the words of @stof from Twig: Anyway, what #2440 mentioned is that front matter are generally implemented around a given syntax, not inside it.

We have the first potential contrib user for metadata in Twig, and a major one too!

While, yes this is certainly exciting, it's also just sugar on top of something that module can already do. In cases like Drupal Bootstrap (themes) where we're more restricted in what and how we're allowed to participate with core, something like this would actually allow us to finally do #2547363-16: Use front matter to version templates. IMO, that's vastly more important.

edit: Not to undermine the importance to Commerce either. Just meant to say that this has a larger theme impact I think.

In #3064854-123: Allow Twig templates to use front matter for metadata support I suggest to introduce a new file extension ie .help which

1. makes it clear it is not a .twig file
2. Make DX/developer decide how the IDE must handle it
3. Keeps .twig files ehm .twig files

And in #126, I explained why that was a bad idea as it just skirts around the same issue, just in a different file format. It also just solves one specific use case (help topics), where it has already been proven that there are several use cases for front matter in core.

No, because that isn't front matter. We cannot introduce a non-standard front matter component in core.

I agree, this wouldn't be front matter, but we could just rename the component to YAMLMetadataParser or something similar. So it could be used for front matter and also other things.

A normal Twig extension is too late in the process. It requires the template to be rendered

My idea was a hybrid approach: YAMLMetadataParser would read and treat the file as a text file during plugin discovery (in Drupal), and the tag/comment, providing the metadata to the template, would be used when Twig is compiled/rendered (in Drupal and 3rd party PHP software).

But this was just an idea, I suppose there could be some (performance related?) problems with that too.

B) Close this issue and revert #3069109: Replace help_topic meta tags with front matter since anything other than the current approach would introduce a non-standard component.

Yes, this is what I didn't understand at first, that the decision has kind of been made already! But maybe we could also keep this issue open, tweak the component slightly and add the tag/comment support, then do a quick search-and-replace followup to that issue? I would be ready to help with this immediately and full-time so that it could be done ASAP.

Unless we really don't just want metadata in Twig, but specifically front matter in Twig. And I can understand that, front matter is super nice, and it is a real shame that Symfony won't support it in core Twig. But there is a price we must pay for it. How high it would be in reality, I don't know.

Unless we really don't just want metadata in Twig, but specifically front matter in Twig.

Yes... that is how this issue started out (and mostly still about): supporting metadata in Twig.

However, due to research and feedback (despite how terse it was) the actual format this metadata took morphed into front matter.

This is, primarily, because I realized this wasn't just about Twig.

Front matter's biggest strength is mostly about its portability and usability between any file.

It's well established, consistent and has essentially become the industry standard for fusing static metadata into any file.

But maybe we could also keep this issue open, tweak the component slightly and add the tag/comment support

My primary concern with this approach (and it's a very big one) is that it then "front matter" (or YAMLMetadataParser) becomes something completely arbitrary and open to interpretation. Leaving the delimiters to whatever decides to implement it; thus, creating vast variations amongst implementations through core/contrib.

This has even larger implications because we would then need to create multiple coding standards for all the "supported" file types that allow "metadata", but using their own "flavor" of "front matter".

Opposed to just a single "standard" that explicitly defines what "front matter" is and how it should be used.

Inherently, "tweak the component" really just means "deviate from established implementations outside of core to suit 'our' needs"; a practice that we have adamantly and purposefully distanced ourselves from; we want to avoid more #DrupalWTFs.

This is why it has to be a true front matter component or not at all.

A normal Twig extension is too late in the process. It requires the template to be rendered, which, as I have stated before means that it would also require whatever is rendering the template (to get the metadata) to also know which variables to send to the template so it doesn't cause a compilation error. This is not an option; it never was. That's why I originally created the Twig issue to introduce something that could be used outside of the render pipeline; they declined.

That is not true. The template needs to be compiled (and we cache it anyway) and then you call:

$template = twig_compile_template($file); // or however we implement that

if (method_exists($template, 'getMetadata') {
  return $template->getMetadata();
}

return [];

We technically absolutely can add a {% metadata {'foo':'bar'} %} tag. There is nothing stopping us or upstream Twig from doing so and they even confirmed in the upstream issue that all of that (except adding a custom interface, hence the need for method_exists) can very easily be done by a twig-metadata twig extension library.

So that's not stopping us.

Inherently, "tweak the component" really just means "deviate from established implementations outside of core to suit 'our' needs"; a practice that we have adamantly and purposefully distanced ourselves from; we want to avoid more #DrupalWTFs.

This is why it has to be a true front matter component or not at all.

While we don't want to integrate more DrupalWTF for obvious reasons, we also have over the time set some standards when we worked with the necessary committees in upstream PHP (PSR-FIG for example).

And with front-matter it's really that you keep everything, but just define it as the 2nd YAML document in a file instead of starting with '---' and yes this allows to add front-matter to PHP, JS and CSS files all with the really nice YAML syntax, but just with some prepending things.

e.g.

Valid "middle-matter":

Foo
---
foo: bar
---

Invalid middle-matter:

Foo---
foo: bar
---

Maybe invalid middle-matter:

Foo
Bar
Baz
Tea
---
foo: bar
---

Maximum three lines (?) of pre-amble, and if there happens to be this special --- on the first three lines the chances that valid YAML is in between are still more than zero ...

I can see and understand that pure front-matter would be much nicer, but for practical reasons non-pure front-matter allows for way more flexibility, also for JS:

/*
---
component: Vue
---
*/

and PHP:

<?php

/*
---
plugin: foo
---
*/

Wouldn't it be nice to have a nicer readable alternative to annotations?

CSS:

/*
---
color_scheme: red
---
*/

etc. for way for flexibility without having to process all those files extra to remove the front-matter.

So it's not a fallacy, but if framework managers decide that a hybrid format is okay for Drupal's twig, PHP, etc. files then we can go with that. And yes the non-pure Front-matter component would still be able to parse normal front-matter as well. So it would just be a convention anyway.

Btw. I heard that framework managers will meet and make a decision then.

Maximum three lines (?) of pre-amble

Everything you described requires just one. It'd be nice and easy to describe: file format specific opening line then three dashes for delimiter. it's possible even with PHP https://3v4l.org/5kamC

I thought a summary of the different approaches discussed might be useful. I'm hoping that everyone agrees on the pros (+) and cons (-) listed, and that the difference in opinion is really only in actual significance. I can update this summary if there is something important missing, or something in it is considered factually incorrect.

Regarding the pros/cons in the summary referring to automated processing: if one of the arguments for choosing front matter is that it can be used for any type of file, it would be good have some discussion on how it would affect the use of CSS and JavaScript files, for example. Especially since CSS and JavaScript files are commonly processed by external tools.

One possible solution, which would allow for any kind of metadata implementation in core, would be if core offered an alternative way to provide the metadata, even for overriden core files. That way contrib and custom projects could use regular files, and there could even be a general contrib solution for doing so.

Thank you to @jhodgdon for summarizing the discussion on #134 and @aleksip for summarizing the pros on cons on #157. Also, thank you all for having this valuable discussion. I'm planning to meet with other framework managers later this week to review this issue.

Everything you described requires just one. It'd be nice and easy to describe: file format specific opening line then three dashes for delimiter. it's possible even with PHP https://3v4l.org/5kamC

You are right, that's much simpler to remember and the adjustment to the provided regexp is trivial.

Discussed this with @alexpott. We looked into the different solutions to try to find the most suitable solution. After reviewing the discussion and the summaries, we still believe that the current front matter is on the right path. We believe the current front matter is the most suited approach because it’s simple to read and use and it’s a pre-defined pattern in other software. We don’t believe we should try to optimize the experience for advanced frontend developers building advanced systems with the cost of making it harder for everyone else to use this system.

One of the key arguments in favor of the competing approaches is to make using templates on different platforms. The type of information that front matter is being used could be an integral part of rendering templates. For example, help topics templates need the data from the front matter to be meaningful. If Layout Builder started using front matter, the same would be true there. But if the information presented in the front matter isn’t an integral part of rendering the template, any component library could choose to remove the information from their templates or omit it as part of rendering.

We also didn’t believe that the front matter or annotations in comments would be the right solution given that it would be omitted by default. If the front matter is an integral part of rendering templates, people should explicitly opt-out rather than.

Moving the information to separate files is conceptually a very different solution. Front matter is supposed to be used in use cases where the information is tightly coupled to the template. Given that it’s coupled to the template, it should also follow the template suggestions.

There’s also a JavaScript library for front matter that could be used for adding support in JavaScript projects.

So front matter it is! Good to have a decision that allows us to move forward and hopefully land this issue as soon as possible.

@lauriii Did you discuss the possibility of core allowing alternative ways to provide the metadata? Similar to how Compony, Multiple Definition Files and other projects currently enable Twig template specific .libraries.yml and .layouts.yml files using hook_library_info_alter() and hook_layout_alter(). This way core could use front matter and advanced developers could use an alternative way provided by contrib. Adding this support should not block this or related issues, and could be done later in a separate issue.

@jhodgdon Yes, I guess my question/hope is that it will eventually be implemented in such a way - as the patch in this issue only seems to provide the possibility for front matter, and future issues will address how it is used in core.

Thanks for the decision.

Re-reviewed the patch: RTBC + 1 :)

Yes, thank you for the decision.

---

One tiny question/request: can we commit this to 8.8.x? Seeing how this is a pure addition of code and 8.8.0 hasn't been released, I feel like getting this in ASAP will help the other related issues here move forward more easily as well as give contrib (i.e. Drupal Bootstrap) the chance to flush out some of the real-world use cases that will likely inform core how to move forward with said other related issues.

I also went ahead and created this follow-up issue based on the previous discussion above: #3096483: Decouple and componentize our Twig implementation.

This issue has been sitting for a while now, is there something else that needs to happen?

Looks like the patch does not apply to 9.1.x, so we need an updated patch for D9. Thanks!

Also 9.0.0 needs yo be replaced with 9.1.0

@andypost There are no "9.0.0" occurrences in the patch.

I mean a CR to update and composer part

+  "require": {
+    "php": ">=7.0.8",
+    "drupal/core-serialization": "^8.8"