Problem/Motivation

While working on #3364108: Configuration schema & required keys and many other "config schema validation"-related issues, I've encountered a number of bugs in Drupal core's config schema.

The latest one is

filter_settings.filter_html:
  type: filter
…

which makes no sense because:

# Filter with module and status.
filter:
  type: mapping
  label: 'Filter'
  mapping:
    id:
      type: string
      label: 'ID'
    provider:
      type: string
      label: 'Provider'
    status:
      type: boolean
      label: 'Status'
    weight:
      type: integer
      label: 'Weight'
    settings:
      type: filter_settings.[%parent.id]

See how at the very end of that definition, there's type: filter_settings.[%parent.id]. So this is essentially a circular schema definition 😅

This is a bug that went unnoticed for almost a decade. It's becoming apparent now that we're starting to use more of the pre-existing config schema for richer validation.

👆 This was fixed in #3404431: Filter settings schema types are incorrect.

This issue uncovered one more occurrence of a circular config schema type reference, in type: layout_builder.section. See #21 for details.

Steps to reproduce

N/A

Proposed resolution

Add basic validation to config schema definitions: verify

  1. that the type is valid
  2. that there's no circular reference

Remaining tasks

  • Basic validation: verify at least the type is valid and that there's no circular reference.
  • Write tests

User interface changes

None.

API changes

API addition: Extension::getAbsolutePath()change record

Data model changes

None.

Release notes snippet

None.

CommentFileSizeAuthor
#11 3401837-nr-bot.txt90 bytesneeds-review-queue-bot

Issue fork drupal-3401837

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Wim Leers created an issue. See original summary.

wim leers’s picture

Assigned: wim leers » Unassigned
Status: Active » Needs review
Issue tags: +Needs tests
wim leers’s picture

LOTS of failures like this:

Drupal\Tests\history\Kernel\Views\HistoryTimestampTest::testHandlers
LogicException: Config schema type "filter_settings.filter_html" has a circular type reference, where it uses the type "filter_settings.[%parent.id]".

👍

wim leers’s picture

The code needs refining, but I'd first like feedback from a subsystem maintainer before spending many more hours on this (this already took me 2–3 hours).

I would like an explicit +1 that this change would be welcomed.

wim leers’s picture

borisson_’s picture

I'm not a subsystem maintainer, but I like this - it would make schema definitions more robust. I wonder if this is something that we need to add to TypedConfigManager, or somewhere else, but that is an implementation detail.

wim leers’s picture

I wonder if this is something that we need to add to TypedConfigManager, or somewhere else,

What would the other possible location be? 🤔

Maybe the config.storage.schema service, which is used by TypedConfigManager? That would force it to happen at a lower level/earlier time, which may be better?

phenaproxima’s picture

Status: Needs review » Needs work
wim leers’s picture

Status: Needs work » Needs review

See #5.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

wim leers’s picture

Status: Needs work » Needs review
Issue tags: +no-needs-review-bot

kostyashupenko made their first commit to this issue’s fork.

kostyashupenko’s picture

Issue tags: -no-needs-review-bot

I don't think we need to add this tag. It just meant MR needs rebase and rebase is done

wim leers’s picture

Related issues:

@kostyashupenko I know. But the point is that this MR should NOT chase HEAD. See #10 and #5.

Thanks for the rebase though! :D

borisson_’s picture

#8, it has taken me a while to get back to this, I have thought this trough, I really wanted to suggest to put the validation of the schema in a new service/class, but I think it makes most sense to put it here, where we are adding a lot of the other validation passes as well.

wim leers’s picture

FYI the actual fixes that this MR makes are also in #3404431: Filter settings schema types are incorrect. This issue aims to add infrastructure to prevent similar problems.

smustgrave’s picture

Status: Needs review » Needs work

Not sure if still needs subsystem review.

But MR appears to be unmergable.

wim leers’s picture

wim leers’s picture

All feedback except one piece has been addressed, and tests are now passing!

Moving on to the last piece of feedback… which is also the only @todo I had added 👍

wim leers’s picture

Assigned: wim leers » Unassigned
Issue summary: View changes
Status: Needs work » Needs review

The last piece of feedback/the @todo I had added was to not just run this validation for the filter_settings.filter_html config schema type, but for ALL config schema types.

Doing this seemed to work fine locally, but caused 254 test failures on GitLab CI. They all look like this:

LogicException: Config schema type "core.entity_view_display.*.*.*.third_party.layout_builder" has a circular type reference, where it uses the type "[%parent.%parent.%type].third_party.[%key]".

Turns out that #2942661: Sections should have third-party settings introduced the ability to define third party settings on Layout Builder Sections. So the layout_builder.section type got this addition:

    third_party_settings:
      type: sequence
      label: 'Third party settings'
      sequence:
        type: '[%parent.%parent.%type].third_party.[%key]'

(Copy/pasted from type: config_entity.) But this is wrong; it should have used the same pattern as for example field formatter settings (which are stored in EntityViewDisplay config entities aka core.entity_view_display.*.*.*) getting third party settings; they used type: field.formatter.third_party.[%key]. Similarly, here we needed:

        type: 'layout_builder.section.third_party.[%key]'

Why is this necessary? Because:

  1. Layout Builder stores its settings as third party settings on Entity View Displays, at core.entity_view_display.*.*.*.third_party.layout_builder", which is only one of many possible third party settings on entity view display config entities — all config entities allow arbitrary third party settings using [%parent.%parent.%type].third_party.[%key]
  2. That is a mapping containing a sections key that is a list of Layout Builder Sections (i.e. layout_builder.section)
  3. And because #2942661 added third party settings to type: layout_builder.section using that same [%parent.%parent.%type].third_party.[%key], the grandparent/two levels up (%parent.%parent) does not refer to the root of the config entity and uses its type (core.entity_view_display.*.*.*), but instead it refers to … layout_builder.section itself! 👈 this is the circular reference! ⚠️

Next up: test coverage. This needs test coverage using a pattern similar to the one in \Drupal\KernelTests\Config\Schema\MappingTest::testInvalidMappingKeyDefinition().

wim leers’s picture

Assigned: Unassigned » wim leers
Issue summary: View changes
Related issues: +#2942661: Sections should have third-party settings

Tried to write tests.

The more representative test coverage pattern to copy is actually \Drupal\Tests\ckeditor5\Kernel\CKEditor5PluginManagerTest::testInvalidPluginDefinitions().

Writing test coverage for this is very complicated because of the hardcoded assumptions in ExtensionDiscovery. See \Drupal\Tests\ckeditor5\Kernel\CKEditor5PluginManagerTest::mockModuleInVfs(), where I previously had to deal with this. 😬 To be continued tomorrow.

phenaproxima’s picture

Assigned: wim leers » Unassigned
Status: Needs review » Needs work

This is making more and more sense.

It's very obscure, complicated logic (not your fault...that seems to be the nature of the config schema system). I'm wondering if we can change the comments/method names in some places to make it as clear as possible to future developers (or just "future us") who need to come to grips with this stuff.

But the logic itself, I don't have any real complaints about.

wim leers’s picture

Assigned: Unassigned » phenaproxima
Status: Needs work » Needs review

🏓

It's very obscure, complicated logic

It's not obscure, but it is complicated. To understand the logic, you have to understand how config schema type definitions. Once you understand that, it should no longer be obscure. And in fact, this logic should help make config schema in general less obscure for the next person 😅

phenaproxima’s picture

Hopefully I didn't hit a nerve :) I meant it's obscure because config schema itself is kind of obscure. Hopefully as it becomes more "useful" (and consistent!) thanks to this work, the obscurity will disappear.

wim leers’s picture

Assigned: phenaproxima » wim leers
Status: Needs review » Needs work

Hahaha, not at all — no worries! 😄 What you're saying in #25 is the same as what I said in #24. We agree that knowledge about the config schema system is obscure (hard-won) knowledge, that we're trying to make more accessible in this issue by removing some of it 👍

wim leers’s picture

The tests were hard to write — I was able to copy most of the pattern in \Drupal\Tests\ckeditor5\Kernel\CKEditor5PluginManagerTest::mockModuleInVfs() but not everything. Which also led me to push #2961541 forward again — see #2961541-26: ExtensionDiscovery fails to scan the vfsStream filesystem for modules or profiles.

wim leers’s picture

Title: Validate config schema definitions to prevent nonsensical definitions » Provide guidance to config schema developers: detect broken config schema types
Assigned: wim leers » Unassigned
Priority: Normal » Major
Status: Needs work » Needs review

All feedback addressed 👍

wim leers’s picture

Issue summary: View changes
phenaproxima’s picture

A couple of questions but I don't think I have any problems with this in general.

Although I think I understand the code here, I'm not really comfortable RTBCing it; it needs subsystem maintainer review anyway, so that's probably who should give it the green light.

wim leers’s picture

Assigned: Unassigned » phenaproxima

Remaining feedback addressed!

@phenaproxima I'm hoping you can close 3 of the 4 threads 🙏 (one is a committer FYI and can remain open)

wim leers’s picture

Issue summary: View changes

Addressed @alexpott's MR feedback. That resulted in an API addition (Extension::getAbsolutePath()) and a change record: https://www.drupal.org/node/3420082.

phenaproxima’s picture

Assigned: phenaproxima » Unassigned

I'm very nearly out of complaints here. I think I only found one or two nits, plus a request for clarification that should be trivial.

I think someone else should give the official RTBC since I'm not an expert in the subject matter. But as far as I can tell, this is RTBC after my final comments are addressed.

wim leers’s picture

Assigned: Unassigned » wim leers
Status: Needs review » Needs work
wim leers’s picture

Assigned: wim leers » Unassigned

#3392903: Validate inputs of TypeResolver::resolveExpression(): only allow %parent, %type and %key landed!

I won't have time for this soon though … so unassigning.

bbrala made their first commit to this issue’s fork.

bbrala’s picture

Rebased this issue, fixed some phpstan/phpcs issues. Hopefully fixed all remarks. Lets see how it runs on gitlab :)

eduardo morales alberti’s picture

In our case we use https://www.drupal.org/project/config_inspector to fix errors with schemas related to config

bbrala’s picture

Status: Needs work » Needs review

Made a follow up for the remaining comment. That same comment did say this is ok for now.

https://www.drupal.org/project/drupal/issues/3517417

Guessing subsystem maintiner would be Alex? Since that has been a while and it will probably go through alex when committing im going to set this to NR.

(test failure was unrelated)

borisson_’s picture

There is a note in the MR about not having to deprecate a method because it was not shipped in a release, since then it has though, so do we need to do a proper deprecation now?

bbrala’s picture

Yeah you're right. We might as well, seems quite easy.

bbrala’s picture

Updated the code to deprecate the method instead of just removing it.

borisson_’s picture

Status: Needs review » Reviewed & tested by the community

Looks good, thanks!

quietone’s picture

Title: Provide guidance to config schema developers: detect broken config schema types » Add basic validation to config schema definitions

Trying for a better title.

alexpott’s picture

I guess a year later I feel stronger that we should address this this way.

On the other hand this feels like the tail wagging the dog - we're making this change for testing purposes only - maybe there is a better way.

So looking at core and grepping for root.*>getPath\( reveals a few places the new method can be used...

So perhaps my instinct to add the method in 2024 was a good - let's do it. Especially since \Drupal\Core\Extension\Extension::__wakeup and \Drupal\Core\Extension\Extension::__sleep take care not to serialize the root.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

We need to add the new method to \Drupal\Core\Config\TypedConfigManagerInterface

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.