Problem/Motivation

Timeline:

  1. #3361534: KernelTestBase::$strictConfigSchema = TRUE and BrowserTestBase::$strictConfigSchema = TRUE do not actually strictly validate brought config validation to tests.
  2. #3379899: Follow-up for #3361534: config validation errors in contrib modules should cause deprecation notices, not test failures made it so that for contrib modules, only deprecation notices are shown (thanks to \Drupal\Core\Config\Schema\SchemaCheckTrait::isContribViolation()). @jibran confirmed that this fixed the test failures in https://www.drupal.org/project/dynamic_entity_reference 👍

Problems:

  1. ⚠️ Even deprecation notices are too disruptive: @alexpott, @bircher, @borisson_ and I agreed at DrupalCon Lille that this is still too disruptive […] (extracted out of #3395099: [meta] Allow config types to opt in to config validation, use "FullyValidatable" constraint at root as signal to opt in to "keys/values are required by default")
    👆 Fix.
  2. ⚠️😱 @Berdir discovered that the Webform tests are failing nonetheless in 10.2.x, see #3400868: [PP-2] to_email and similar config keys have schema type email but allow other things. So I automated testing that in #3402134: Test Webform against upcoming 10.2.x on GitLab CI — and use concurrency to make tests 6x faster (on GitLab CI). The test results confirm what @Berdir said. Which makes this critical because it disrupts contrib significantly 😐
    👆 Fix.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#7 Screenshot 2023-11-16 at 4.07.10 PM.png399.48 KBwim leers

Issue fork drupal-3402168

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Wim Leers created an issue. See original summary.

Wim Leers credited Berdir.

Wim Leers credited bircher.

wim leers’s picture

StatusFileSize
new399.48 KB

Determined the root cause:

  private function isContribViolation(): bool {
    $test_file_name = (new \ReflectionClass($this))->getFileName();
…

$this was assumed to be the test class. Unfortunately, this is inaccurate, because it depends on WHAT is calling
\Drupal\Core\Config\Schema\SchemaCheckTrait::checkConfigSchema():

✅ it is accurate when
called by \Drupal\Tests\SchemaCheckTestTrait::assertConfigSchema()
❌ it is wrong when
called by \Drupal\Core\Config\Development\ConfigSchemaChecker::onConfigSave(), which happens to be called when … installing configuration during tests

OOPS 😱

IMHO this is a 10.2.x beta blocker because virtually every contrib maintainer will run into this 😅🙈

xjm’s picture

Title: Follow-up for #3361534: config validation errors can still occur for contrib modules, disrupting contrib » Follow-up for #3361534: Config validation errors can still occur for contrib modules, disrupting contrib
Parent issue: » #3385081: [meta] Release preparation steps for 10.2.0
wim leers’s picture

Status: Active » Needs review
wim leers’s picture

Assigned: wim leers » Unassigned
Issue summary: View changes
  1. ⚠️ Even deprecation notices are too disruptive: @alexpott, @bircher, @borisson_ and I agreed at DrupalCon Lille that this is still too disruptive […] (extracted out of #3395099: [meta] Allow config types to opt in to config validation, use "FullyValidatable" constraint at root as signal to opt in to "keys/values are required by default")

👆 Fixed in https://git.drupalcode.org/project/drupal/-/merge_requests/5430/diffs?co...

alexpott’s picture

Status: Needs review » Needs work

I don't think this fix will work because the schema checker also runs in the site under test. See \Drupal\Core\Test\FunctionalTestSetupTrait::prepareSettings().

We need to pass another argument into the testing.config_schema_checker service to determine whether to do config validation - this argument can be based on the test class.

wim leers’s picture

Assigned: Unassigned » wim leers

I will run the Webform test suite against 10.2.x + this MR to verify.

wim leers’s picture

First made the Webform test suite run much faster on GitLab CI, then used @Berdir's tip in Slack to automatically test with this MR on 10.2.

Quoting the findings from #3402134-12: Test Webform against upcoming 10.2.x on GitLab CI — and use concurrency to make tests 6x faster (on GitLab CI):

Test against 10.2.x HEAD
Results
❌
296) Drupal\Tests\webform\Kernel\Entity\WebformEntityTest::testWebformMethods
Exception: Exception when installing config for module webform, message was: Schema errors for webform.settings with the following errors: 0 [mail.default_to_mail] <em class="placeholder">"[site:mail]"</em> is not a valid email address., 1 [mail.default_from_mail] <em class="placeholder">"[site:mail]"</em> is not a valid email address.

/builds/issue/webform-3402134/web/core/tests/Drupal/KernelTests/KernelTestBase.php:732
/builds/issue/webform-3402134/tests/src/Kernel/Entity/WebformEntityTest.php:32
/builds/issue/webform-3402134/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

Caused by
Drupal\Core\Config\Schema\SchemaIncompleteException: Schema errors for webform.settings with the following errors: 0 [mail.default_to_mail] <em class="placeholder">"[site:mail]"</em> is not a valid email address., 1 [mail.default_from_mail] <em class="placeholder">"[site:mail]"</em> is not a valid email address.

/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/Development/ConfigSchemaChecker.php:94
/builds/issue/webform-3402134/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php:111
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/Config.php:229
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/ConfigInstaller.php:396
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/ConfigInstaller.php:149
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/ProxyClass/Config/ConfigInstaller.php:75
/builds/issue/webform-3402134/web/core/tests/Drupal/KernelTests/KernelTestBase.php:729
/builds/issue/webform-3402134/tests/src/Kernel/Entity/WebformEntityTest.php:32
/builds/issue/webform-3402134/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
❌
222) Drupal\Tests\webform\Functional\States\WebformStatesManagerTest::testStatesToText
Drupal\Core\Config\Schema\SchemaIncompleteException: Schema errors for webform.settings with the following errors: 0 [mail.default_to_mail] <em class="placeholder">"[site:mail]"</em> is not a valid email address., 1 [mail.default_from_mail] <em class="placeholder">"[site:mail]"</em> is not a valid email address.

/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/Development/ConfigSchemaChecker.php:94
/builds/issue/webform-3402134/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php:111
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/Config.php:229
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/ConfigInstaller.php:396
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Config/ConfigInstaller.php:149
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/ProxyClass/Config/ConfigInstaller.php:75
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Extension/ModuleInstaller.php:334
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/ProxyClass/Extension/ModuleInstaller.php:83
/builds/issue/webform-3402134/web/core/lib/Drupal/Core/Test/FunctionalTestSetupTrait.php:466
/builds/issue/webform-3402134/web/core/tests/Drupal/Tests/BrowserTestBase.php:559
/builds/issue/webform-3402134/web/core/tests/Drupal/Tests/BrowserTestBase.php:367
/builds/issue/webform-3402134/tests/src/Functional/WebformBrowserTestBase.php:44
/builds/issue/webform-3402134/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
Test against 10.2.x + #3402168: Follow-up for #3361534: Config validation errors can still occur for contrib modules, disrupting contrib
Results
✅
[32mDrupal\Tests\webform\Kernel\Entity\WebformEntityTest           3 passes                                      
❌
[0mFATAL Drupal\Tests\webform\Functional\States\WebformStatesManagerTest: test runner returned a non-zero error code (2).
[31mDrupal\Tests\webform\Functional\States\WebformStatesManagerT   0 passes   1 fails                            

(Difference in output is due to switching from sequential to parallel test running per @Berdir in #6.

Conclusion

@alexpott was right in #3402168-12: Follow-up for #3361534: Config validation errors can still occur for contrib modules, disrupting contrib: the core MR does fix it for kernel tests, but not yet for functional tests.

On it …

wim leers’s picture

2926c398 is a hardening that fixes the incomplete detection logic, resulting in more functional tests passing, but not all:

Drupal\Tests\webform\Functional\States\WebformStatesManagerT   1 passes                                      
Drupal\Tests\webform\Functional\Variant\WebformVariantElemen   1 passes                                      
Drupal\Tests\webform\Functional\States\WebformStatesHiddenTe   0 passes             1 exceptions             
FATAL Drupal\Tests\webform\Functional\States\WebformStatesHiddenTest: test runner returned a non-zero error code (2).

Now working on the actual bug that @alexpott pointed out, which only occurs if the the functional test is using the UI to make changes to config. This is true for \Drupal\Tests\webform\Functional\Settings\WebformSettingsSerialTest for example, as this HTML response it triggers proves:

The website encountered an unexpected error. Try again later.<br><br><em class="placeholder">Drupal\Core\Config\Schema\SchemaIncompleteException</em>: Schema errors for webform.webform.contact with the following errors: 0 [handlers.email_confirmation.settings.to_mail] &amp;lt;em class=&amp;quot;placeholder&amp;quot;&amp;gt;&amp;amp;quot;[current-user:mail]&amp;amp;quot;&amp;lt;/em&amp;gt; is not a valid email address., 1 [handlers.email_confirmation.settings.from_mail] &amp;lt;em class=&amp;quot;placeholder&amp;quot;&amp;gt;&amp;amp;quot;_default&amp;amp;quot;&amp;lt;/em&amp;gt; is not a valid email address., 2 [handlers.email_notification.settings.to_mail] &amp;lt;em class=&amp;quot;placeholder&amp;quot;&amp;gt;&amp;amp;quot;_default&amp;amp;quot;&amp;lt;/em&amp;gt; is not a valid email address., 3 [handlers.email_notification.settings.from_mail] &amp;lt;em class=&amp;quot;placeholder&amp;quot;&amp;gt;&amp;amp;quot;[webform_submission:values:email:raw]&amp;amp;quot;&amp;lt;/em&amp;gt; is not a valid email address. in <em class="placeholder">Drupal\Core\Config\Development\ConfigSchemaChecker-&gt;onConfigSave()</em> (line <em class="placeholder">94</em> of <em class="placeholder">core/lib/Drupal/Core/Config/Development/ConfigSchemaChecker.php</em>). <pre class="backtrace">call_user_func(Array, Object, &#039;config.save&#039;, Object) (Line: 111)
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher-&gt;dispatch(Object, &#039;config.save&#039;) (Line: 229)
Drupal\Core\Config\Config-&gt;save() (Line: 278)
Drupal\Core\Config\Entity\ConfigEntityStorage-&gt;doSave(&#039;contact&#039;, Object) (Line: 486)
Drupal\Core\Entity\EntityStorageBase-&gt;save(Object) (Line: 257)
Drupal\Core\Config\Entity\ConfigEntityStorage-&gt;save(Object) (Line: 108)
Drupal\webform\WebformEntityStorage-&gt;save(Object) (Line: 352)
Drupal\Core\Entity\EntityBase-&gt;save() (Line: 609)
Drupal\Core\Config\Entity\ConfigEntityBase-&gt;save() (Line: 293)
Drupal\Core\Entity\EntityForm-&gt;save(Array, Object) (Line: 50)
Drupal\webform\EntitySettings\WebformEntitySettingsBaseForm-&gt;save(Array, Object) (Line: 847)
Drupal\webform\EntitySettings\WebformEntitySettingsSubmissionsForm-&gt;save(Array, Object)
…

wim leers’s picture

Assigned: wim leers » Unassigned
Status: Needs work » Needs review

c0ae1f84 made the Webform test suite pass — with the exception of unrelated failures (for example, WebformStatesHiddenTest and WebformSettingsPreviewTest fail because it's asserting markup literally instead of using CSS selectors, XPath or DOMDocument — probably these output changes are due to #2441811: Upgrade filter system to HTML5). See the results.

wim leers’s picture

@alexpott's alternative MR looks much simpler and achieves the same. 👍 I was just trying to minimize change, but I'm of course fine with his proposal.

However, his proposal only fixes problem #2 in the issue summary, not problem #1 (i.e no deprecation error for contrib modules … at least not yet, because even that is considered too disruptive). The new name of the new argument he proposed is so strongly connected to deprecation errors that we're forced to. So my only concern with the MR is the $triggerDeprecationOnValidationError argument.

Unless … @alexpott now thinks that the deprecation error is actually fine to keep? 🤔

wim leers’s picture

Testing Alex' MR (commit: https://git.drupalcode.org/project/webform/-/merge_requests/380/diffs?co...).

Hopefully these will match:

(Which is possible despite his MR still triggering a deprecation error because Webform is suppressing deprecations.)

Update: they do match! 👍 — when grapping the results page for the string 0 passes (55 occurrences), exceptions (27 occurrences), fails (28 occurrences)

wim leers’s picture

Status: Needs review » Reviewed & tested by the community

Cannot find a single remaining thing to complain about in @alexpott's alternate MR.

The test results for webform on top of his MR still (see #17) match ✅

It's a very different approach from mine — much simpler, more explicit, less magic. 👍 My only contribution to this MR is adding a data provider to the test.

wim leers’s picture

Status: Reviewed & tested by the community » Needs work

… except that our YAML parsing of services.yml files does not quite like enums yet:

1) Drupal\Tests\image\Functional\ImageDimensionsTest::testImageDimensions
Symfony\Component\DependencyInjection\Exception\InvalidArgumentException: The file "/builds/project/drupal/sites/simpletest/75153141/services.yml" does not contain valid YAML: The string "!php/const Drupal\Core\Config\Schema\SchemaCheckConstraintValidation::Error" could not be parsed as a constant. Did you forget to pass the "Yaml::PARSE_CONSTANT" flag to the parser? at line 12 (near "testing.config_schema_checker: { class: Drupal\Core\Config\Development\ConfigSchemaChecker, arguments: ['@config.typed', [config_schema_test.no_schema, config_schema_test.some_schema, config_schema_test.schema_data_types, config_schema_test.no_schema_data_types, config_test.dynamic.system], !php/const Drupal\Core\Config\Schema\SchemaCheckConstraintValidation::Error], tags: [{ name: event_subscriber }] }")

😬

wim leers’s picture

wim leers’s picture

Status: Needs work » Reviewed & tested by the community

Ready now :)

  • catch committed 19acc1d5 on 10.2.x
    Issue #3402168 by Wim Leers, alexpott, Berdir, bircher, borisson_,...

  • catch committed c8b39f01 on 11.x
    Issue #3402168 by Wim Leers, alexpott, Berdir, bircher, borisson_,...
catch’s picture

Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Fixed

This looks good, easy to follow the new logic and mostly moving things around otherwise. Committed/pushed to 11.x and cherry-picked to 10.2.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.