Problem/Motivation

As per #2183983: Find hidden configuration schema issues there are fails with several migration tests when using strict schema checking.

Proposed resolution

1. Add strict schema checking in migrations.
2. Resolve failures.
3. Review.
4. Commit.

Remaining tasks

1. Resolve fails.
2. Review.
3. Commit.

User interface changes

None.

API changes

Any schema changes as needed.

Files: 
CommentFileSizeAuthor
#23 2358269-migrate-schema-fix-23.patch14.34 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,044 pass(es). View
#23 interdiff.txt8.57 KBGábor Hojtsy
#21 interdiff.txt1.97 KBGábor Hojtsy
#21 2358269-migrate-schema-fix-21.patch10.44 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,141 pass(es), 20 fail(s), and 0 exception(s). View
#19 2358269-migrate-schema-fix-19.patch8.47 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,058 pass(es), 30 fail(s), and 0 exception(s). View
#19 interdiff.txt3.91 KBGábor Hojtsy
#15 interdiff.txt2.08 KBGábor Hojtsy
#15 2358269-migrate-schema-fix-14.patch5.42 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,207 pass(es), 35 fail(s), and 2 exception(s). View
#13 2358269-migrate-schema-fix-13.patch5.45 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,169 pass(es), 27 fail(s), and 7 exception(s). View
#13 interdiff.txt722 bytesGábor Hojtsy
#10 interdiff.txt4.1 KBGábor Hojtsy
#10 2358269-migrate-schema-fix-10.patch4.75 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,153 pass(es), 26 fail(s), and 7 exception(s). View
#9 2358269-migrate-schema-fail.patch656 bytesGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,056 pass(es), 87 fail(s), and 0 exception(s). View
#4 2358269-menu-generate-schema-fails.patch2.31 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,364 pass(es). View

Comments

vijaycs85’s picture

vijaycs85’s picture

Gábor Hojtsy’s picture

Issue tags: -

@vijaycs85: are you working on this? You put it on the sprint :)

Gábor Hojtsy’s picture

Status: Active » Needs review
FileSize
2.31 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,364 pass(es). View

Seems like this issue is way too wide in scope. The menu_link_config and page_manager fails listed are not even from core. And then you added some system.action and system.theme stuff which is not mentioned in the summary... Anyway, let's look at some menu stuff first.

Gábor Hojtsy’s picture

Issue summary: View changes

Removed the menu_link_config and page_manager related bits from the summary.

vijaycs85’s picture

    [menu_ui.settings:main_links] => Missing schema.
    [menu_ui.settings:secondary_links] => Missing schema.
system.theme.global
Array
(
    [system.theme.global:features.main_menu] => Missing schema.
    [system.theme.global:features.secondary_menu] => Missing schema.
)

Looks like D8 totally rewrote this functionality more flexible instead of just a variable. Here is how the flow of this variable from D6 to D8

1. D7 converted menu_primary_links_source to menu_main_links_source
2. Initially D8 converted menu_main_links_source to menu.settings:source.main
3. Then moved to theme.settings (as per change notice) as part of converting menu as block

However any block that we assign to primary_menu region is a primary menu. So for the D6 to D7 conversion, we have to create a block and assign to this region?

vijaycs85’s picture

Issue summary: View changes
  1. FIXED: As mentioned in #6, menu already covered/solved differently.
  2. FIXED: system.action.*:configuration - looks like the configuration element added recently. So fixed and won't be an issue anymore
  3. NEEDS FIX: migration - Need to fix them individually
Gábor Hojtsy’s picture

Title: Adding missing translation schema for menu, migration & system configuration » Migrations fail configuration schema checking
Issue summary: View changes
Issue tags: -D8MI, -language-config, -sprint
Parent issue: » #2183983: Find hidden configuration schema issues
Gábor Hojtsy’s picture

FileSize
656 bytes
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,056 pass(es), 87 fail(s), and 0 exception(s). View

Schema fail patch.

Gábor Hojtsy’s picture

FileSize
4.75 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,153 pass(es), 26 fail(s), and 7 exception(s). View
4.1 KB

Reasons for test fails that I could not resolved but dabbled in resolving:

- MigrateBlockTest: all the visibility settings are migrated to the wrong place (settings.visibility instead of visibility); then all the tests confirm that the path visibility was empty which of course is, the settings are migrated to the wrong place; I tried to set this up properly (see attached) but then it fails on not finding the condition plugins; can someone help?
- MigrateNodeBundleSettingsTest: migrates boolean base field default values as a simple boolean instead of an array of "value: $bool" pairs; again tried to resolve this but made it worse; someone with better migrate experience can help maybe?

Test fails that I did resolve:

- MigrateUploadEntityFormDisplayTest: migrates entity form modes and attempts to set a nonexistent label setting. This was easy to adjust, I managed to actually fix this.
- MigrateUserPictureEntityFormDisplayTest: same thing as with the MigrateUploadEntityFormDisplayTest, same plugin
- MigrateUserProfileEntityFormDisplayTest same problem but different plugin, resolved also.

Need help for the first two. Should not be big for those who have migrate experience hopefully :)

Gábor Hojtsy’s picture

Title: Migrations fail configuration schema checking » Migration bugs in block visibility, field overrides and form modes found by configuration schema checking

The last submitted patch, 9: 2358269-migrate-schema-fail.patch, failed testing.

Gábor Hojtsy’s picture

Title: Migration bugs in block visibility, field overrides and form modes found by configuration schema checking » Migration bugs in block visibility, field overrides, cron settings and form modes found by configuration schema checking
FileSize
722 bytes
5.45 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,169 pass(es), 27 fail(s), and 7 exception(s). View

Finally the system cron migration also fails. This does not seem to have its own tests but only fails as part of the full D6 test suite. The target config names are incorrect.

Gábor Hojtsy’s picture

Issue tags: +Ghent DA sprint
Gábor Hojtsy’s picture

FileSize
5.42 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,207 pass(es), 35 fail(s), and 2 exception(s). View
2.08 KB

Actually figured out the right fix for the booleans. Now only need to fix the block visibility one with the right constants syntax use. I tried to add constants but it says would not be supported by this plugin?!

The last submitted patch, 10: 2358269-migrate-schema-fix-10.patch, failed testing.

The last submitted patch, 13: 2358269-migrate-schema-fix-13.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 15: 2358269-migrate-schema-fix-14.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
3.91 KB
8.47 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,058 pass(es), 30 fail(s), and 0 exception(s). View

The cron stuff had its own test. Fixing that. I made an attempt to move the block one forward, but its still dabbling...

- Added some constants to be able to use the static values I needed.
- Found another incorrect settings visibility code in the block settings plugin, that needs to be moved to somewhere but not sure how?
- Also now the user roles may be null but should be an array.

The whole visibility thing may require its own plugin due to the logic required?

I *think* this should only fail on migrate block tests. At least locally it did not fail on the labels anymore. Not sure why it does on the above results?!

Status: Needs review » Needs work

The last submitted patch, 19: 2358269-migrate-schema-fix-19.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
10.44 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,141 pass(es), 20 fail(s), and 0 exception(s). View
1.97 KB

Looked at the fails again:

1. Missed a form mode label extra in user pictures.

2. System maintenance enablement is stored in state not in config. Not sure if there is a state target for migrate. But its clearly not correct to migrate the enabled setting to config, it is not stored there in D8.

Status: Needs review » Needs work

The last submitted patch, 21: 2358269-migrate-schema-fix-21.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
8.57 KB
14.34 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 82,044 pass(es). View

So we needed that custom plugin and I figured it out. Now the old visibility setting and the paths, roles are passed to the new visibility migration plugin which sets up the target structure as needed. That exposed problems with the block tests, because actually two source blocks have custom paths (yay), so fixed the test to test for that and now that actually proves the block visibility *is* migrated. Prior to that all tests were ensuring the block visibility was NOT migrated, which is wrong because again some blocks do have visibility constraints in this test (thankfully).

This should resolve all the problems with the remaining block fails. I think this fixes all the bugs found. While it is certainly possible to improve the block visibility stuff further, this is required to satisfactorily close #2183983: Find hidden configuration schema issues so I would love to get this in as soon as possible. It fixes clear bugs found :)

Gábor Hojtsy’s picture

Title: Migration bugs in block visibility, field overrides, cron settings and form modes found by configuration schema checking » Migration bugs in block visibility, field overrides, cron, maintenance settings and form modes found by configuration schema checking
benjy’s picture

Status: Needs review » Reviewed & tested by the community

I reviewed the patch, everything looks good to me. Great work!

+++ b/core/modules/migrate_drupal/src/Tests/MigrateDrupalTestBase.php
@@ -15,6 +15,15 @@
+   * @see \Drupal\Core\Config\Testing\ConfigSchemaChecker
+   *
+   * @var bool
+   */
+  protected $strictConfigSchema = TRUE;

This is awesome! I'm almost surprised there were so few wrong config keys :)

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Awesome work. Committed 194539a and pushed to 8.0.x. Thanks!

  • alexpott committed 194539a on 8.0.x
    Issue #2358269 by Gábor Hojtsy: Migration bugs in block visibility,...
Gábor Hojtsy’s picture

Fantastic, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.