Suggested commit message

Issue #2361615 by Berdir, Gábor Hojtsy: Field type config schemas are not in the base schema.

Problem/Motivation

In #2183983-74: Find hidden configuration schema issues @berdir found that several base field schemas are with modules even though they should be provided by the system because they are usable and in fact used without enabling the modules.

Proposed resolution

Move those field schemas to the system level.

Remaining tasks

Tests.

User interface changes

None.

API changes

None.

Files: 
CommentFileSizeAuthor
#38 interdiff.txt2.22 KBGábor Hojtsy
#38 2361615-field-base-schemas-38.patch12.13 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,714 pass(es). View
#36 interdiff.txt807 bytesGábor Hojtsy
#36 2361615-field-base-schemas-36.patch11.33 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,836 pass(es). View
#32 interdiff.txt785 bytesGábor Hojtsy
#32 2361615-field-base-schemas-32.patch11.3 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,826 pass(es), 3 fail(s), and 0 exception(s). View
#30 interdiff.txt2.37 KBGábor Hojtsy
#30 2361615-field-base-schemas-30.patch10.54 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,222 pass(es), 2 fail(s), and 0 exception(s). View
#28 interdiff.txt1.12 KBGábor Hojtsy
#28 2361615-field-base-schemas-28.patch11.07 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Failed to run tests: PHP Fatal error encountered during run_tests.sh. See review log for details.. View
#23 interdiff-array.txt3.02 KBGábor Hojtsy
#23 2361615-field-base-schemas-23-array.patch11.35 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,855 pass(es), 2 fail(s), and 1,460 exception(s). View
#20 interdiff-9-to-19.txt1.46 KBGábor Hojtsy
#20 2361615-field-base-schemas-19.patch8.86 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 75,977 pass(es), 794 fail(s), and 61 exception(s). View
#13 interdiff.txt6.87 KBGábor Hojtsy
#13 2361615-field-base-schemas-13.patch11.1 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Drupal installation failed. View
#9 2361615-field-base-schemas-9.patch7.64 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,981 pass(es), 15 fail(s), and 0 exception(s). View
#9 2361615-field-base-schemas-9-test-only.patch1.82 KBGábor Hojtsy
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,970 pass(es), 39 fail(s), and 0 exception(s). View
#8 2361615-field-base-schemas-8.patch5.82 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,683 pass(es). View
#8 interdiff.txt3.07 KBGábor Hojtsy
#1 2361615-field-base-schemas.patch5.88 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,547 pass(es). View

Comments

Gábor Hojtsy’s picture

Issue summary: View changes
Status: Active » Needs review
FileSize
5.88 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,547 pass(es). View

This is very much aligned with other field types in the same file. Several were missing, the string one was missing a storage settings item. This is based on @berdir's patch in #2183983-74: Find hidden configuration schema issues with updates to key names. The tests there do not apply this needs its own tests. @Berdir should be credited when this is committed, adding suggested commit message.

Gábor Hojtsy’s picture

Priority: Normal » Critical
Issue tags: +Configuration system, +D8MI, +language-config, +sprint, +D8 upgrade path

Carrying on tags from that issues, since we are resolving part of it here

Gábor Hojtsy’s picture

The storage settings schemas are probably not correct yet, I took them verbatim from @berdir, they need one by one checking. It is highly unlikely they are all sequences of strings(?).

Berdir’s picture

I did that based on other examples, it looks very weird to me as well but something else didn't work ?

I also included the string schema in #2068655: Entity fields do not support case sensitive queries as I had to change text there. Will conflict with that but I don't really care what gets in first.

Berdir’s picture

Also, it should be possible to verify this by adding the schema trait stuff to the ContentTranslationSettingsTest. That exploded badly without this and should pass with these fixes.

alexpott’s picture

+++ b/core/config/schema/core.data_types.schema.yml
@@ -402,7 +402,13 @@ core.date_format.*:
+field.string.storage_settings:
+  type: sequence
+  label: 'String settings'
+  sequence:
+    - type: string

I think this should be:

field.string.storage_settings:
  type: mapping
  label: 'String settings'
  mapping:
    max_length:
      type: integer
      label: 'Maximum length'
alexpott’s picture

Status: Needs review » Needs work
  1. +++ b/core/config/schema/core.data_types.schema.yml
    @@ -422,7 +428,143 @@ field.string.value:
    +field.string_long.storage_settings:
    +  type: sequence
    +  label: 'String (long) settings'
    +  sequence:
    +    - type: string
    

    This should just be of the type: field.string.storage_settings

  2. +++ b/core/config/schema/core.data_types.schema.yml
    @@ -422,7 +428,143 @@ field.string.value:
    +field.string_long.field_settings:
    +  type: sequence
    +  label: 'String (long) settings'
    +  sequence:
    +    - type: string
    +      label: 'Setting'
    

    Use type: field.string.field_settings - actually string has no field settings afaics

  3. +++ b/core/config/schema/core.data_types.schema.yml
    @@ -422,7 +428,143 @@ field.string.value:
    +field.uri.storage_settings:
    +  type: sequence
    +  label: 'URI settings'
    +  sequence:
    +    - type: string
    

    Similar to string this just has a max_length integer value

  4. +++ b/core/config/schema/core.data_types.schema.yml
    @@ -422,7 +428,143 @@ field.string.value:
    +field.uri.field_settings:
    +  type: sequence
    +  label: 'URI settings'
    +  sequence:
    +    - type: string
    +      label: 'Setting'
    

    There are no uri field settings.

  5. +++ b/core/config/schema/core.data_types.schema.yml
    @@ -422,7 +428,143 @@ field.string.value:
    +field.created.storage_settings:
    +  type: sequence
    +  label: 'Created timestamp settings'
    +  sequence:
    +    - type: string
    ...
    +field.created.field_settings:
    +  type: sequence
    +  label: 'Created timestamp settings'
    +  sequence:
    +    - type: string
    +      label: 'Setting'
    

    There are no created field settings or storage settings. Tempting to just set these to type: ignore - or somehow fix the field config entities to not export the empty array.

  6. +++ b/core/config/schema/core.data_types.schema.yml
    @@ -422,7 +428,143 @@ field.string.value:
    +field.changed.storage_settings:
    +  type: sequence
    +  label: 'Changed timestamp settings'
    +  sequence:
    +    - type: string
    ...
    +field.changed.field_settings:
    +  type: sequence
    +  label: 'Changed timestamp settings'
    +  sequence:
    +    - type: string
    +      label: 'Setting'
    

    Exactly the same as created. There are none.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
3.07 KB
5.82 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,683 pass(es). View

Implemented suggestions based on @alexpott. By defining the sequence without an element type, we define it as having elements of type 'undefined' which fails validation with amy strict understanding of the schema such as with config_inspector or SchemaCheckTrait that we are going to use for the tests.

I also noticed that the string long type's default value should be text type, not string and likewise the created and changed field default value types need to be ints not string.

Fixed one more minor typo in entity reference comment.

Gábor Hojtsy’s picture

FileSize
1.82 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,970 pass(es), 39 fail(s), and 0 exception(s). View
7.64 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,981 pass(es), 15 fail(s), and 0 exception(s). View

Let's see a test for the config files in the content translation settings test as @Berdir suggested. Did not run this locally yet.

The last submitted patch, 9: 2361615-field-base-schemas-9-test-only.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 9: 2361615-field-base-schemas-9.patch, failed testing.

Gábor Hojtsy’s picture

Most fails on the fix+test patch are about base field overrides schemas missing. @vijaycs85 also identified those at #2358263: Adding missing configuration schema for views blocks but was deemed out of scope there. It should not be a huge deal to fix it here.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
11.1 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Drupal installation failed. View
6.87 KB

A first try at content translation settings and base field overrides (which seem to be singular and not multivalued in defaults).

Status: Needs review » Needs work

The last submitted patch, 13: 2361615-field-base-schemas-13.patch, failed testing.

Berdir’s picture

base field overrides (which seem to be singular and not multivalued in defaults).

That assumption is not correct, base fields can be multi-valued (they do need custom storage for that now, though) and could have multiple default values. But I've noticed this as well.

It's not just the multi vs. single value. The problem is also that the schema defines properties, like 'value' and default values are most of the time specified as scalar, as the entity field api takes care of expanding that to the main property of the first field item.

We probably need to convert it internally at some point, similar to how we expand it during setValue(). So 'whatever' needs to become array(array('value' => 'whatever')). Also note that 'value' is not hardcoded but should be derived from the main property name method of the specified field type.

Gábor Hojtsy’s picture

@Berdir:

That assumption is not correct, base fields can be multi-valued (they do need custom storage for that now, though) and could have multiple default values. But I've noticed this as well.

All right, is this field type based? Where do I find this in code?

It's not just the multi vs. single value. The problem is also that the schema defines properties, like 'value' and default values are most of the time specified as scalar, as the entity field api takes care of expanding that to the main property of the first field item.

That does not seem to be related to this issue? If this is also field type specific, then fine because the schema defines those keys per field type now not on system level.

Berdir’s picture

multi or single values is not field type specific. All field types can have multiple values theoretically and that is why the schema/default value.

The problem is that you need to go from this:

$entity_reference_field_defininition->setDefaultValue(1);
$integer_field_definition->setDefaultValue(1);

To the correct structure, which would be:

entity reference: array(0 => array('target_id' => 1)
integer: array(0 => array('value' => 1)

The reason setDefaultValue(1) works is that $entity->$field = 1, works as well, and it basically has the following logic:

1. In FieldItemList::setValue(), we have this:

// Support passing in only the value of the first item.
      if (!is_array($values) || !is_numeric(current(array_keys($values)))) {
        $values = array(0 => $values);
      }

2. In FieldItemBase::setValue():

    // Treat the values as property value of the first property, if no array is
    // given.
    if (isset($values) && !is_array($values)) {
      $keys = array_keys($this->definition->getPropertyDefinitions());
      $values = array($keys[0] => $values);
    }

This is actually not correct, instead of just using the first property, it should use $definition->getFieldStorageDefinition()->getMainProperty(), which would return value/target_id.

What I would try is add similar logic to BaseFieldDefinition::setDefaultValue(), but it might be easier to reverse it: First if you do not have an array, convert it to array($main_property => $value), and second if the first array key is not numeric, convert it to array(0 => $value).

Gábor Hojtsy’s picture

In short from a schema perspective we should assume multiple default values, same as for other places and just fix the base field override config to express it like so?

Berdir’s picture

Yes. And it might be easier to already convert it to multiple in the base field definition and not just the override, so you don't have the additional complexity there when comparing and so on.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
8.86 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 75,977 pass(es), 794 fail(s), and 61 exception(s). View
1.46 KB

Reworking the schemas as a first step for multi-default values, which basically means mostly back to #9, so I rolled an interdiff comparing to that directly.

Status: Needs review » Needs work

The last submitted patch, 20: 2361615-field-base-schemas-19.patch, failed testing.

vijaycs85’s picture

Status: Needs work » Needs review
FileSize
8.88 KB
564 bytes
+++ b/core/config/schema/core.data_types.schema.yml
@@ -449,8 +574,7 @@ field.boolean.storage_settings:
+  type: sequence

I guess, this will fail. we need to provide element under sequence.

Gábor Hojtsy’s picture

FileSize
11.35 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 79,855 pass(es), 2 fail(s), and 1,460 exception(s). View
3.02 KB

Fixing the schema issue first in the content translation settings. So we have separate test results, I am uploading the fix for the base field definition default values as a separate patch (which includes the schema fix). I extended the unit tests and they pass locally. Not sure what else will explode and how.

Edit: looks like @vijaycs85 already submitted the schema fix. So my interdiff includes that too but ignore that. :) Removed the schema only fix patch since @vijaycs85's will result in test feedback properly.

Status: Needs review » Needs work

The last submitted patch, 23: 2361615-field-base-schemas-23-array.patch, failed testing.

Berdir’s picture

+++ b/core/lib/Drupal/Core/Field/BaseFieldDefinition.php
@@ -443,6 +443,12 @@ public function setDefaultValueCallback($callback) {
+    if (!is_array($value)) {
+      $value = array(array($this->getMainPropertyName() => $value));
+    }
+    elseif (!is_numeric(array_keys($value)[0])) {
+      $value = array(0 => $value);
+    }

Not else if, just two separate ifs.. Both checks need to be applied if you pass in "1".

The last submitted patch, 22: 2361615-field-base-schema-22.patch, failed testing.

Gábor Hojtsy’s picture

@Berdir: uhm, no? The first one already does array(array()), no need for the second one? Why inspect the array if we know what we find?

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
11.07 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Failed to run tests: PHP Fatal error encountered during run_tests.sh. See review log for details.. View
1.12 KB

So the impressive 1.4k exceptions are due to #2248709: The root type of a configuration object can not be a sequence, the get() method on sequences will not resolve to subkeys. We definitely should not fix it here and neither postpone this on fixing that because the content translation settings are to be refactored to third party setting on language settings anyway in #2363155: content_translation.settings config is not scalable, so the best we can do here is to type it with ignore. Oh well.

Status: Needs review » Needs work

The last submitted patch, 28: 2361615-field-base-schemas-28.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
10.54 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,222 pass(es), 2 fail(s), and 0 exception(s). View
2.37 KB

Well, so ignore does not support get() either. Since that would be a crude workaround, instead of starting to fix things to make a crude workaround work (unrelated to this issue), I opted for a different workaround, not asserting the schema of the settings for now. Added appropriate todo comments to existing issues.

Status: Needs review » Needs work

The last submitted patch, 30: 2361615-field-base-schemas-30.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
11.3 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,826 pass(es), 3 fail(s), and 0 exception(s). View
785 bytes

So the remaining 2 fails are funnily due to the EntityTest type specifying a uid reference value in a totally wrong way. It either needs to be using the substructure also with the main property name or just pass in a literal integer like in my fix. That then works flawlessly.

Based on prior results and this one locally tested, it should be green.

Status: Needs review » Needs work

The last submitted patch, 32: 2361615-field-base-schemas-32.patch, failed testing.

The last submitted patch, 32: 2361615-field-base-schemas-32.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
11.33 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,836 pass(es). View
807 bytes

So this simplification of EntityTest's uid default value would require that the EntityViewsDataTest unit test mocks more of the services so to be able to look that up. We can avoid that extra complexity by providing the full array structure properly, since we know it anyway. This should be green.

Berdir’s picture

+++ b/core/lib/Drupal/Core/Field/BaseFieldDefinition.php
@@ -443,6 +443,12 @@ public function setDefaultValueCallback($callback) {
   public function setDefaultValue($value) {
+    if (!is_array($value)) {
+      $value = array(array($this->getMainPropertyName() => $value));
+    }
+    elseif (!is_numeric(array_keys($value)[0])) {
+      $value = array(0 => $value);
+    }

We should probably document somewhere what formats are supported here?

Gábor Hojtsy’s picture

FileSize
12.13 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 80,714 pass(es). View
2.22 KB

@Berdir: well the existing docs on the method actually document it quite well:

   * @param mixed $value
   *   The default value for the field. This can be either:
   *   - a literal, in which case it will be assigned to the first property of
   *     the first item.
   *   - a numerically indexed array of items, each item being a property/value
   *     array.
   *   - NULL or array() for no default value.
   *

This is not in the diff scope but is already there prior to patch. It is true the patch does not accept an empty array or null gracefully anymore. Null would become a nested array with a value/target_id of null. Not what is intended :D

However there was no test coverage for this, so I added test coverage to the unit tests and fixed the code. Also documented the non-indexed array option that the above snippet did not document before.

Looks good?

vijaycs85’s picture

Status: Needs review » Reviewed & tested by the community

Looks good.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/config/schema/core.data_types.schema.yml
@@ -434,7 +439,127 @@ field.string.value:
+      mapping:
+        target_id:
+          type: integer
+          label: 'Value'

Hmmm... I don't think this can be an integer - should be a string since entity IDs do not have to be an integer. We are also missing the 'target_uuid' key. Other than that looks awesome.

alexpott’s picture

Status: Needs work » Reviewed & tested by the community

Actually let's handle #40 in a critical follow since this is just moving the definition into core and this patch unblocks discovering other schema issues.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

This issue addresses a critical bug and is allowed per #2350615: [policy, no patch] What changes can be accepted during the Drupal 8 beta phase?. Committed 8f89d7b and pushed to 8.0.x. Thanks!

  • alexpott committed 8f89d7b on 8.0.x
    Issue #2361615 by Gábor Hojtsy: Fixed Field type config schemas are not...
Gábor Hojtsy’s picture

@alexpott opened that followup at #2366877: Entity Reference field schema incorrect for reference.

Gábor Hojtsy’s picture

Issue tags: -sprint

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.