From #2320253-10: FIC/FSC::preSave() do not call the parent implementation :

not sure why we initialize default settings in preSave(), this leaves freshly created entities broken for runtime use (incomplete settings) until they are actually saved. We should do this as early as postCreate().

Similarly, FieldStorageConfig::$module is currently only populated in preSave(), meaning the entity is not reliable until that point. This should be done also in postCreate().

How did we end up with the current approach after several rounds of back and forth?

Entity reference fields -- that is, any field type based on \Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem -- need special handling. They have a setting called handler, which normally has a value of default that needs to be changed to something like default:taxonomy_term, depending on what kind of entities are being referenced, just before the field is saved. In HEAD, this is done on pre-save by the field module (field_field_config_presave()). This is also necessary for certain Entity Reference-based contrib modules, like Dynamic Entity Reference, to work properly.

BUT! If we want to make sure the settings are always complete and coherent, then we need to be able to do this normalization continuously, whenever field settings change, regardless of whether the field's been saved or not.

So, how to do that? A few approaches were tried, and it was a twisty, turny road:

  • Initially, we thought it should be a special method of \Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem, which the field config entity calls whenever settings are changed.
  • But that seemed icky and poorly encapsulated, so we moved it to a new method of \Drupal\Core\Field\FieldItemInterface, thinking that the API addition was justified.
  • But then we thought we could just repurpose that interface's existing fieldSettingsToConfigData() method for this purpose, but it turned out that was mismatched to our needs.
  • So we decided we'd add a new method to FieldItemInterface after all.
  • But then it was pointed out that we don't really need new API -- we could just do the normalization in field_field_config_create(), since field entities are now created with complete, coherent settings.
  • After all of this, it was pointed out that none of the runtime config entities are actually valid because you would need to run \Drupal\Core\Config\Entity\ConfigEntityStorage::mapToStorageRecord before they become valid. This is currently only run on save.

So that's where we ended up: when an entity reference field is created (with complete settings, now), those settings get normalized. Then they get normalized again when the field is saved. No new API surface! At least, not now.

We still may need new API to ensure that the settings are always normalized even if they undergo multiple changes (before saving). But, if that's necessary, let's do it in #3347291: Combine field storage and field instance forms and/or #3358049: Save FieldStorageConfig at the same time as FieldConfig, in which we'll be figuring out just how, where, and when a field's settings can be changed as it gets built in the UI by a site builder.

What about base field overrides, I hear you asking? Well...those already don't get normalized in HEAD, because field_field_config_presave() (which, remember, normalizes the field's settings) only acts on...field_config entities. Not base_field_override entities. So sure, it should be fixed, but it's not in scope here.

But...but...doing special normalization only for particular field type plugin classes sucks and is restrictive, you say. Yup, that's true. It should probably be applied to particular kinds of field types (for example, perhaps it could be done for anything in the reference category). But that's also not in scope here.

Basically: this thing we're doing, where we're implementing entity hooks for field_config entities, is a temporary measure. It gets us past the current, specific problem of having incomplete settings, and puts us in a position to add a well-designed, properly scoped API in place for continuous settings normalization in other issues, where it will become more relevant. Hook implementations aren't API, so we can easily remove them.

Issue fork drupal-2327883

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

yched’s picture

Issue summary: View changes
StatusFileSize
new7.95 KB

Attached patch removes the "defaults settings merging" from preSave() and adds it to postCreate() and postLoad(), in both:
- FieldConfigBase (= BaseFieldOverride & FieldInstanceConfig)
- FieldStorageConfig

yched’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, 1: 2327883-settings_init-1.patch, failed testing.

yched’s picture

Status: Needs work » Needs review
StatusFileSize
new1.24 KB

Turns out, FSC::preSave() needs to run before its parent because we need to populate $this->module before we can calculate dependencies.

Quick fix to see what the bot says for now, but waiting for preSave() to have the $module property populated looks wrong as well. the entity cannot be considered functionnal until that point. This should move to postCreate() too...

Status: Needs review » Needs work

The last submitted patch, 4: 2320253-FIC_preSave_parent-4.patch, failed testing.

yched’s picture

Status: Needs work » Needs review
StatusFileSize
new10.38 KB
new8.27 KB

Doh, #4 was actually the interdiff...
Reuploading

yched’s picture

StatusFileSize
new9.29 KB
new1.92 KB

- Moving the initialization of FSC::$module from preSave() to postCreate()
- Removing the manual initialization of FSC::$id in preSave(), since no other config entity seems to bother with this.

Interdiff is with patch #1.

yched’s picture

Issue summary: View changes

Status: Needs review » Needs work

The last submitted patch, 7: 2327883-settings_init-7.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
larowlan’s picture

andregp’s picture

Status: Needs work » Needs review
StatusFileSize
new5.99 KB
new10.8 KB

Basic reroll of patch #7 for D9.3

ranjith_kumar_k_u’s picture

StatusFileSize
new5.97 KB
new762 bytes
ranjith_kumar_k_u’s picture

StatusFileSize
new6.32 KB
new537 bytes
andregp’s picture

Status: Needs review » Needs work

Reroll is returning too much fails, still needs work

yogeshmpawar’s picture

Status: Needs work » Needs review
StatusFileSize
new5.98 KB
new863 bytes

Tested patch locally for CS issues & didn't find one. problem with #22 & #23 is that they are tested against 9.4.x branch & issue version 9.3.x

So let's see this updated patch will work or not & hiding #22 & #23

yogeshmpawar’s picture

Issue tags: -Needs reroll
yogeshmpawar’s picture

StatusFileSize
new5.99 KB
new675 bytes

Updated patch will fix test build issue.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

andregp’s picture

Status: Needs review » Needs work

Too much fails on the last test (almost 3k)

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

wim leers’s picture

Version: 9.5.x-dev » 11.x-dev
Issue tags: +Configuration schema, +validation, +blocker
wim leers’s picture

Priority: Normal » Major
Related issues: +#3358049: Save FieldStorageConfig at the same time as FieldConfig

And TIL I was pinged to review/provide feedback on https://git.drupalcode.org/project/drupal/-/merge_requests/3926#note_189783 2 days ago, and that happens to run into exactly this problem space, but not \Drupal\field\Entity\FieldConfig::preSave() specifically: field_field_config_presave() is doing things that are critical for certain FieldConfig entities (those that are for Entity Reference fields) to be correct/saveable.

Increasing priority.

lauriii made their first commit to this issue’s fork.

lauriii’s picture

Committed a version of the patch that doesn't lead into WSOD to see how many test fails it causes. I expect there will be a lot of test fails because normalizing the config in preSave could be the expected behavior.

phenaproxima made their first commit to this issue’s fork.

phenaproxima’s picture

Hiding patches in favor of the merge request.

phenaproxima’s picture

I watched with increasing bafflement as @lauriii and @catch got into the weeds discussing this issue's backwards compatibility concerns on Slack. After that, I finally Zoomed with @lauriii to get the executive summary and understand it for myself.

The plan is this: we want the calling code to be responsible for merging the default settings into FieldConfigStorage::$settings (and its counterparts in BaseFieldOverride and FieldConfig). postCreate() will do this for you when it creates a new entity (by calling the $this->initSettings() method). In other situations, you need to do it yourself, and preSave() will check your homework -- $this->settings must contain every key in the default settings, or you'll get a deprecation notice (and maybe, in Drupal 11, an exception). postLoad()'s behavior will be unchanged.

borisson_’s picture

Issue tags: +ddd2023

Left a couple of comments on the MR, the remaining fails are all because of the new deprecation. Do we solve those in this issue as well?

phenaproxima’s picture

Assigned: Unassigned » phenaproxima

Assigning to myself to fix the tests and find out which code paths are triggering these deprecations.

phenaproxima’s picture

Assigned: phenaproxima » Unassigned
Status: Needs work » Needs review
berdir’s picture

Did a review. issue summary definitely needs an update.

What I don't quit understand is why we plan to enforce that settings are passed in, we have the default settings, we can complete them, why not keeping doing that? That's how plugins work as well (19 instances of $this->configuration = $configuration + $this->defaultConfiguration(); in core, I think a ConfigurablePluginBase would be neat ;))

IMHO, this is going to hurt us down the line. This would make adding a new setting to a field type plugin a BC break as all existing code who creates fields of that type is going to break.

Less sure about invalid settings, I can see how that's more sensible to enforce, but we would have the same problem then if we remove a setting from a field type, then that's suddenly invalid too.

Edit: Looking at this again, I think we are merging defaults on create, so this is "only" a problem if you manually change settings afterwards? For example calling setSettings() with an incomplete list of settings? I guess that is less of an issue then, still having 50+ failing tests in core shows that this seems fairly common.

smustgrave’s picture

Status: Needs review » Needs work

#43 mentioned an issue summary update.

And number of test failures in MR.

phenaproxima’s picture

Status: Needs work » Needs review
phenaproxima’s picture

Issue summary: View changes
Issue tags: +Needs change record

Updating the issue summary with the proposed resolution. I'm also tagging this for a change record, which I'm pretty sure will be needed.

phenaproxima’s picture

Issue summary: View changes
lauriii’s picture

There's one more soft blocker to #3358049: Save FieldStorageConfig at the same time as FieldConfig related to invalid config which should be resolved either by this issue or a follow-up. The field_field_config_presave normalizes the selection handle plugin IDs and we need to manually duplicate that so that we can run it earlier. Thoughts on whether we should address that here or in a follow-up issue?

Edit: note that this was mentioned by @Wim Leers in #33.

phenaproxima’s picture

Issue tags: -Needs change record

Okay, filled in the CR!

phenaproxima’s picture

Issue tags: +Needs followup

Re #48, I think we should do this in a follow-up. As you can see, I did a little experimentation around this but it seems rather tricky, and I'm not sure that experimental approach was the right one anyway.

wim leers’s picture

Issue summary: View changes
Status: Needs review » Needs work

First: AWESOME progress here! 🤩

Thanks for #39, that's really crucial context wrt BC concerns and release manager guidance! Linking it from the issue summary 👍 Makes sense that we don't want to modify postLoad() at all in particular, because that would have the biggest BC (runtime) impact.

I have to agree with @Berdir in #43: that simple approach seems preferable and less disruptive. But AFAICT since then, that was largely adopted. In reviewing this, I did spot a few more BC concerns though. 😅 Or at least, potential ones — hopefully you can show why none of them are actual BC concerns! 😄

Regarding @lauriii's remark in #48: I think that if we're making the behavior of Field(Storage)Config's creation/saving logic sane & centralized, that it makes sense to consider field_field_config_presave() in scope too — it's touching the same problem space.

phenaproxima’s picture

Assigned: Unassigned » wim leers
Status: Needs work » Needs review

Reassigning to Wim for another round of review.

wim leers’s picture

Assigned: wim leers » phenaproxima
Status: Needs review » Needs work

This is much closer to RTBC'ability now thanks to your work on FieldConfigBase::set('settings', …)! It allows us to revert a LOT of changes.

I see that you commented many times that that was impossible, but … I think you wrote those comments before you did that work? 😅 In my spot testing, it sure does work every single time!

wim leers’s picture

Also figured out why there was a weird deprecation-testing-related test failure. The root cause is simple: thanks to the excellent work you did on FieldConfigBase::set('settings', …), the deprecation error

Saving a field_config with incomplete settings is deprecated in drupal:10.2.0 and will throw an exception in drupal:11.0.0. Adjust your code to ensure that all supported settings are provided. See https://drupal.org/node/3375334.

simply cannot ever occur anymore! 😅

And I'm 99% certain that this also explains why FieldSettingsTest is now failing: the expectations just need to be updated thanks to set('settings', …) no longer being able to add invalid keys 👍

wim leers’s picture

Before this can be RTBC'd, this definitely needs an issue summary update.

Also, thanks to FieldConfigBase::set('settings', …), I think the CR needs to be updated? 🤓

phenaproxima’s picture

Assigned: phenaproxima » wim leers
Status: Needs work » Needs review
phenaproxima’s picture

Issue summary: View changes
phenaproxima’s picture

Do we still need a follow-up here?

I ask because...as far as I can tell, it's no longer possible to put incomplete settings on a field or field storage config entity, unless you use reflection. So rather than deprecate, we can just throw an exception.

wim leers’s picture

Assigned: wim leers » phenaproxima
Status: Needs review » Needs work

Looking great!

But in doing a thorough final review, I made one very startling discovery:
\Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigData() has existed since 2014 and AFAICT does exactly the same thing as the new \Drupal\Core\Field\FieldItemInterface::normalizeSettings() we're adding?! 😳

phenaproxima’s picture

Assigned: phenaproxima » wim leers
Status: Needs work » Needs review
wim leers’s picture

Assigned: wim leers » phenaproxima
Status: Needs review » Needs work

I see a way forward thanks to https://www.drupal.org/node/3376455! 🥳

Also still needs a change record for the migration change.

Those are the only 2 remaining threads 👍

wim leers’s picture

Issue tags: +Needs change record

This is ready, but still needs change record updates, as well as 2 additional change records (the new interface argument and the modified migration destination). It also still needs a follow-up for the @todo that was just added for Drupal 11.

Once those things are done … this will be RTBC! 😄🥳

phenaproxima’s picture

Okay, the change records have been added/updated.

phenaproxima’s picture

Assigned: phenaproxima » wim leers
Status: Needs work » Needs review
Issue tags: -Needs followup

Opened #3379022: [Drupal 11] Require $field_definition to be passed to FieldItemInterface::fieldSettingsToConfigData() to change the interface method in Drupal 11, and linked to it in the todos.

borisson_’s picture

Status: Needs review » Reviewed & tested by the community

I have read trough the PR several times, in the hope to find something that was not clear, could use better documentation or can otherwise be improved. I can't seem to find anything.
I also read through the Change Records, those are very clear as well.
I think that means this can go to rtbc, but I would feel more comfortable with one of the field or entity api maintainers +1 here as well. Do we need to tag this as needs subsystem maintainer review?

wim leers’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: +Needs subsystem maintainer review

Agreed with @borisson_; tagged Needs subsystem maintainer review 😊


Refined the migration destination CR, added link to #3379022: [Drupal 11] Require $field_definition to be passed to FieldItemInterface::fieldSettingsToConfigData() to the fieldSettingsToConfigData() CR.

The third CR (for field config/storage settings) sounds … a bit like it's breaking quite a lot of things? 😅🙈

So let's see if all those *Test.php changes are necessary, by reverting all modified tests. It's easy to get those by comparing to the last upstream commit that was merged in:

git checkout 2327883-field-storage-config
git pull
git diff e72fa7b3759972013c3e943372baaae85722fbfe HEAD -- **/**Test.php

Because if we could show that none of those test changes are necessary, then we simultaneously prove that the risk of disruption/BC break is extremely low. And … unless I made a mistake, every single one of the tests still passes if I revert all test changes! 🤩 The changes were definitely necessary at an earlier iteration of this merge request, but now they're no longer necessary: thanks to core/lib/Drupal/Core/Field/Entity/CompleteFieldSettingsTrait.php, and the updates to core/lib/Drupal/Core/Field/FieldConfigBase.php and subclasses, everything is guaranteed to be complete, always! 😊

The sole exception: \Drupal\KernelTests\Core\Field\FieldSettingsTest. But that makes sense: it's literally testing the updated behavior in a unit-like kernel test that checks how Field(Storage)Config entities are created and how their settings are saved 👍

wim leers’s picture

Assigned: wim leers » Unassigned

**Test.php didn't catch everything. Reverting the changes in FileFieldTestBase.php and ImageFieldCreationTrait appears to work fine too 🤓

Now, to be fair, the code that I reverted in the last 2 commits was:

  1. actually more accurate, even though unnecessary
  2. going to be reused in its entirety in #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints or one of its child issues, because then those invalid settings should start triggering validation errors!

But AFAICT this means that the third CR (https://www.drupal.org/node/3375334) can simply be deleted — or at least associated with #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints instead of this issue! 😊 Because as FieldEntitySettingsTest already says:

    // If the field storage is saved with incomplete settings (which, as we've
    // just proven, is impossible to do by calling public methods), we should
    // get an exception on save.

IOW: none of the work was in vain, but let's keep this issue's scope as tight as possible and hence make it easier to get this issue committed 🤞

This also should make subsystem maintainer review a lot simpler.

wim leers’s picture

Still green. This is IMHO RTBC now.

I would like explicit confirmation for the 2 commits I pushed from @phenaproxima and @borisson_. After that, we need subsystem maintainer review.

phenaproxima’s picture

I have no objections to Wim's changes. Fewer files touched is good, and indeed, we can delete the third change record -- even nicer!

borisson_’s picture

Fewer touched files is indeed more better.

wim leers’s picture

Associated the 3rd CR with #3324140 instead 👍 2 CRs left here.

amateescu’s picture

Status: Needs review » Needs work

Posted a couple of comments in the MR.

I think a possible solution to the first problem (calling fieldSettingsToConfigData() in FieldConfigBase::postCreate()) is to move the code which sets the handler setting for ER fields from field_field_config_presave to a hook_entity_create() implementation.

wim leers’s picture

Assigned: Unassigned » phenaproxima
Issue tags: -Needs subsystem maintainer review

Thanks, @amateescu! 😊🙏

Assigning to @phenaproxima to address your remarks, but … I did ask 2 clarifying follow-up questions on the MR, and I have one other question for you here: 🤓

[…] move the code which sets the handler setting for ER fields from field_field_config_presave to a hook_entity_create() implementation.

It sounds like this means that my discovery/"aha" moment in #59 was wrong? 😅 @phenaproxima was actually first adding a new FieldItemInterface::normalizeSettings() method, and then in doing a final review, I discovered ::fieldSettingsToConfigData(), which I'd never heard of before, and it seemed to be doing the exact same thing.

amateescu’s picture

@Wim Leers, unfortunately.. yes :) I learned about fieldSettingsToConfigData() from this issue, but then I read the issue where it was introduced and realized that it should only be used prior to saving a field to config.

As for a test case about not calling fieldSettingsToConfigData() in postCreate(), a very good example is \Drupal\options\Plugin\Field\FieldType\ListItemBase::storageSettingsToConfigData() and \Drupal\options\Plugin\Field\FieldType\ListItemBase::storageSettingsFromConfigData(), which show how/why a field type might need to have a certain structure for its settings in the runtime object, and a different settings structure when stored in config.

wim leers’s picture

Okay, great, thank you SO MUCH for confirming that!

Sounds like we should restore the ::normalizeSettings() approach! 😇

phenaproxima’s picture

Assigned: phenaproxima » wim leers
Status: Needs work » Needs review
wim leers’s picture

Assigned: wim leers » Unassigned
Status: Needs review » Reviewed & tested by the community

This is ready now, although an explicit confirmation from @amateescu sure would be nice! 😇

amateescu’s picture

Status: Reviewed & tested by the community » Needs review

Sounds like we should restore the ::normalizeSettings() approach!

Is there a problem with the hook_entity_create() suggestion for setting the handler setting of ER fields? A hook implementation has the advantage that it can be removed at any time if we refactor the code around selection plugins and their special handling of plugin IDs, while a new method on FieldItemInterface would take a major release cycles to deprecate and remove.

wim leers’s picture

Assigned: Unassigned » amateescu

Is there a problem with the hook_entity_create() suggestion for setting the handler setting of ER fields? A hook implementation has the advantage […]

Yes, there is a significant problem with that: it gets us the same exact problem that field_field_config_presave() causes today: modules like https://github.com/jibran/dynamic_entity_reference then have to hack their way around it with hook_module_implements_alter() because these hooks cause the field's (entity reference selection handler) settings processing to happen at the wrong level!

See #33, which points to @larowlan's comments, and most critically, to https://git.drupalcode.org/project/drupal/-/merge_requests/3926#note_194955. See https://git.drupalcode.org/project/dynamic_entity_reference/-/blob/3.1.0....

Basically: while we're touching this code, we might as well make it so that core won't be breaking contrib modules in this way anymore.

amateescu’s picture

it gets us the same exact problem that field_field_config_presave() causes today: modules like https://github.com/jibran/dynamic_entity_reference then have to hack their way around it

That's very easy to fix though: we can just check if the field type provides a target_type storage setting and return early if it doesn't.

wim leers’s picture

Assigned: amateescu » phenaproxima
Status: Needs review » Needs work

#80: but that will work only for this particular subclass of Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem. i.e. it'll work for Drupal\dynamic_entity_reference\Plugin\Field\FieldType\DynamicEntityReferenceItem because that is the one that happens to not set target_type. But what if a different subclass does something different? 🤔 Aha, no, because target_type is the only additional key that \Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem::defaultStorageSettings() specifies! So yes, then this could totally work, and we avoid increasing the API surface.

I think the solution that we had arrived at here is nice & fine. It seems that @larowlan, @jibran, @phenaproxima and I were all fine with it (maybe @lauriii too, not sure). But it does mean expanding the Field API more, and if we can avoid that, that'd be better. It'll also mean a simpler upgrade path from Drupal 10 to 11, which in its own is worth a lot (i.e. no field type implementation changes necessary!).

So: +1 to restoring field_field_config_presave() and returning early if target_type is absent from the default storage settings. Thank you for helping keep scope & impact as minimal as possible, @amateescu! 😊🙏🙏🙏

wim leers’s picture

After these twists and turns, the IS became outdated again, and we're missing a change record for the solution that @amateescu helped us arrive at. Tagging for both.

amateescu’s picture

Thanks for the updates, this looks much cleaner now. Left two more comments on the MR then this can be set to RTBC.

@Wim Leers, if you mean that we need a change record for the field_field_config_presave() update, we can just tell @larowlan and @jibran about it and save a few hundred people from reading stuff they'll never have to care about :)

This CR needs to be deleted though.

phenaproxima’s picture

Assigned: phenaproxima » wim leers
Status: Needs work » Needs review

OK, I've made the requested changes here. Re-assigning to Wim for a final review, now that we have subsystem maintainer approval, and for change record/issue summary updates.

phenaproxima’s picture

Issue summary: View changes
lauriii’s picture

Assigned: wim leers » phenaproxima
Status: Needs review » Needs work
Issue tags: +Needs tests

If I read #72 and #78 correctly, it's proposing moving the entity reference normalization from hook_field_config_presave to hook_field_config_create which should be sufficient to ensure that entity reference fields are valid after creation.

However, the MR currently keeps the normalization in hook_field_config_presave which is insufficient because it means that the handler is normalized only at the saving stage. It also looks like tests are passing which means that we have insufficient test coverage.

I confirmed that the MR as it is is not sufficient by confirming that \Drupal\Tests\field\Functional\EntityReference\EntityReferenceAdminTestfails with the current MR and a version of #3358049: Save FieldStorageConfig at the same time as FieldConfig that doesn't have the hard coded normalization as part of field creation.

lauriii’s picture

Discussed with @phenaproxima on Slack because there were some confusion between comments #72 and #81. As I mentioned in #86, I understood #72 and #78 so that @amateescu is proposing to move the entity reference normalization from hook_field_config_presave to hook_field_config_create. I said in #86 that it should be sufficient to ensure that entity reference fields are valid after creation.

However, @phenaproxima pointed out that it won't be sufficient because the handler settings may change during the life cycle of the field config. This is why we would need at least hook_field_config_presave and hook_field_config_create to normalize the config. However, this defeats the whole purpose of this issue because the intent is also to be able to validate the config before saving it. For this reason, we would need the normalization to happen on a separate layer, that could be triggered before saving.

I'm wondering if this provides sufficient justification for the new API that we were proposing to add until #72 😊

lauriii’s picture

Assigned: phenaproxima » Unassigned
Status: Needs work » Needs review

Discussed with @amateescu on Slack and he said that even with the justification in #87, he would prefer to introduce the new API as part of #3347291: Combine field storage and field instance forms. I can see how that makes sense because then we can make sure that the API is flexible enough to support the needs of that issue.

Doing the entity reference normalization both, on hook_field_config_create and hook_field_config_presave addresses the bug where field config entities are invalid after creation which blocks both #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints and #3358049: Save FieldStorageConfig at the same time as FieldConfig. We can't remove hook_field_config_presave without introducing the new normalization API because existing code may be relying on it, e.g. field_field_storage_config_update in core.

I think it would be great if we could add test coverage that I asked in #86 to make sure that this doesn't regress until we have the additional test coverage from #3358049: Save FieldStorageConfig at the same time as FieldConfig.

jibran’s picture

Does that mean DER can remove the hook implementation alter and implement it's own hook_field_config_create and update existing hook_field_config_presave?

lauriii’s picture

@jibran yes, the current implementation would allow you to remove hook_module_implements_alter from DER once 10.1.x is EOL 😊

wim leers’s picture

Status: Needs review » Reviewed & tested by the community

This is ready, just needs a change record. Since this is blocking high-impact usability issues, and the CR would affect so few people, I think it'd be fine to land this ahead of the CR existing 😇

tim.plunkett’s picture

Issue tags: -Needs tests

I'm not sure if we'll get away with not having a CR but it would be nice to have an updated IS 😇

wim leers’s picture

Assigned: Unassigned » amateescu

@phenaproxima will tackle the issue summary update + change record tomorrow.

But more importantly, @Berdir has surfaced concerns with the current approach, which was proposed by his fellow subsystem maintainer @amateescu 😄 So, I think we really need @amateescu's input now.
(I did point @Berdir to the relevant comments though, so maybe that is already enough? 😇)

berdir’s picture

We discussed my feedback in slack and I'm fine with addressing that in a follow-up.

In short, the situation for DER is indeed improved as I realized in my second comment due to the target_type check, I still think this should target specific field types only or at least give subclasses a way to customize it, but it is in line with the existing logic and behavior and that's exactly what the follow-up is about.

lauriii’s picture

Assigned: amateescu » Unassigned

Unassigning since @amateescu was part of the discussion mentioned in #94 and it sounded like everyone was onboard with the current solution.

Still needs an issue summary update and change record 😇

phenaproxima’s picture

Issue summary: View changes

Partial IS update.

phenaproxima’s picture

Issue summary: View changes
wim leers’s picture

IS looks great!

Thanks for deleting that one CR, and keeping the other CR (https://www.drupal.org/node/3379013), that one is still relevant.

phenaproxima’s picture

Issue tags: -Needs change record

Change record written and the issue summary is rounded out. https://www.drupal.org/node/3381705

wim leers’s picture

Thanks for https://www.drupal.org/node/3381705!

Let's do this :D 🚢

wim leers’s picture

🧐 Let's see if the deprecation @lauriii and I thought would make sense triggers failures in core's test suite. Note that for now, I did not yet do the conditional part. If there are failures specifically for FieldStorageConfig settings being set on FieldConfig, the commit I just pushed should fail tests.

EDIT: we expect this to fail,and that’s a good thing, we’ll then know where @lauriii's excellent observation would’ve caused disruption in contrib! 👍

lauriii’s picture

Status: Reviewed & tested by the community » Needs review

Moving to needs review because there has been changes to the MR that would benefit from further review.

lauriii’s picture

Status: Needs review » Needs work

Discussed with @Wim Leers, @phenaproxima, @tedbow, and @tim.plunkett.

First of all, there are 2 hard requirements:

  1. We absolutely cannot break BC (meaning ANY change in behavior) at any point in the life cycle of Field(Storage)Config
  2. We absolutely need the ability to have complete Field(Storage)Config before saving

We cannot normalize in ::setSettings() because it's possible for any contrib module to call ::set('settings', …) instead. (Plus, \Drupal\Core\Entity\EntityForm::copyFormValuesToEntity() also calls ::set(…)!)

👆 Hence the current approach in the MR: to normalize during both ::set() and ::setSettings().

⚠️ However, we discovered that we cannot do it in ::set(), ⚠️ because it could introduce very complex issues to debug. The key challenge is that ::set() is being called

  • At SAVE time by \Drupal\Core\Config\Entity\ConfigEntityStorage::doSave after it has called \Drupal\Core\Config\Entity\ConfigEntityStorage::mapToStorageRecord and \Drupal\Core\Config\Entity\ConfigEntityStorage::mapFromStorageRecords
  • But it's not called at LOAD time by \Drupal\Core\Config\Entity\ConfigEntityStorage::doLoadMultiple

… meaning that normalizing in ::set() does lead to differences in settings depending on where in the life cycle the config entity is, i.e. after saving the entity, it would be in the normalized state, but after loading it from the database, it would NOT be in the normalized state. This is highly confusing, and breaking BC for \Drupal\Core\Config\Entity\ConfigEntityStorage::mapFromStorageRecords during save.

This leaves only one option that we can see: restore FieldItemInterface::normalizeSettings(), make it a public method, so that it can be called before saving the Field(Storage)Config from Field UI, and when validating config.

A tangentially related problem is that the sole thing in Drupal core testing this is \Drupal\field_test\Plugin\Field\FieldType\TestItem::storageSettingsToConfigData() + \Drupal\KernelTests\Core\Field\FieldSettingsTest::testConfigurableFieldSettings(), but TestItem methods are adding NEW keys instead of transforming the values of EXISTING keys. The latter was always the intent — see #2293773: Field allowed values use dots in key names - not allowed in config. So we should fix the test coverage around TestItem::class because it's currently asserting things that do not make sense.
In the absurd case that something in contrib is relying on this: we can and should detect incorrect implementations of toConfigData() and fromConfigData(): any key-value pairs that it returns MUST also exist in \Drupal\field_test\Plugin\Field\FieldType\TestItem::defaultStorageSettings(). If that's not the case, that should trigger a \LogicException or at least a deprecation. See https://drupal.org/node/2307365 for the change record when the API was introduced.

lauriii’s picture

Discussed with @amateescu on Slack: https://drupal.slack.com/archives/C1BMUQ9U6/p1692622430919759.

The key insight is that @amateescu pointed out that we provide an API that allows runtime field [storage] config entities to NOT be valid config. The API that allows this is \Drupal\Core\Field\FieldItemInterface::storageSettingsToConfigData and \Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigData, because these are required to have run for some field types before they can be valid config entities to store. Given that there is a difference between the stored config entity, and the runtime entity, he also pointed out that he doesn't see why the solution in #2293773: Field allowed values use dots in key names - not allowed in config should be limited to massaging existing values.

Given this, to be able to validate field [storage] config, we'll need to trigger \Drupal\Core\Field\FieldItemInterface::storageSettingsToConfigData and \Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigData before triggering config validation. This could be done in an ephemeral object used for validating the config.

The proposed changes to the current MR:

  • Merge default settings in postCreate and preSave
  • Stop doing it on set, setSetting, setSettings to avoid introducing changes with the runtime field config object
  • Move the current trigger_error code from \Drupal\Core\Field\FieldConfigBase::mergeDefaultSettings to \Drupal\Core\Field\FieldConfigStorageBase::mapToStorageRecord. This should be run after \Drupal\Core\Field\FieldItemInterface::storageSettingsToConfigData and \Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigData to make the callbacks responsible for normalizing the settings before saving.
lauriii’s picture

Status: Needs work » Needs review
Issue tags: +Needs followup, +Needs change record updates, +Needs issue summary update

Implemented #104 with the exception that I left the preSave changes to a follow-up. There's around 50 tests failing if we introduce the deprecation. I think we may want implement the changes to preSave, but that's not needed for either of the blocked issues. I think we could open a follow-up for that.

I'm wondering if we could now revert the migration related changes too and move them to a follow-up? Maybe @phenaproxima could confirm that.

On top of that, we need to update CRs and the issue summary.

lauriii’s picture

Updated the issue summary and the change records.

Not going to file an explicit follow-up: we can define what we need in #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints.

amateescu’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for the updates, @lauriii, let's move forward! :)

wim leers’s picture

I dug through this in detail with @laurii and I'm confident that we can make this work for config validation support in Field(Storage)Config + BaseFieldOverride.

One thing I definitely hadn't thought of yet so far is how config entity objects may have different values in their runtime state vs their storage state, thanks to \Drupal\Core\Config\Entity\ConfigEntityStorage::mapToStorageRecord()! 😳 Fortunately, only 3 classes in core override it:

  1. \Drupal\Core\Field\FieldConfigStorageBase::mapToStorageRecord()
  2. \Drupal\field\FieldStorageConfigStorage::mapToStorageRecord()
  3. \Drupal\layout_builder\Entity\LayoutBuilderEntityViewDisplayStorage::mapToStorageRecord()

… plus 6 in contrib, 5 of which are virtually identical to core's Layout Builder one.

This means that for config entity validation to work, the infrastructure that provides validation for config entities must also be able to call mapToStorageRecord() not just prior to save, but prior to validation. But that method is currently protected 😅 So we need some way to get config entities into the same state as just before saving. We may want to do that as part of #2907163: Split validation-related methods from FieldableEntityInterface into a separate ValidatableInterface or in another issue.

In any case, it's clear that it's possible, that this change won't block it. That's the thing we wanted to make sure.

This has been very thoroughly reviewed, and the end result after >100 comments is a far smaller patch with a tighter scope than before subsystem maintainers started reviewing it. Special thanks to @amateescu & @Berdir! 😊

EDIT: cross-post RTBC 😅🥳

effulgentsia’s picture

Status: Reviewed & tested by the community » Needs work

Overall, I think this patch makes a lot of sense. I'm glad it ended up with a fairly simple implementation. Since there might be some subtle BC breaks here, before committing it I want to read through the comment history to see what's been discussed as far as that goes. In the meantime, these are the only two things that jumped out at me:

  1. From the IS:

    If settings are changed any point (either by setSettings(), setSetting(), or set('settings', ...)), the default settings will be merged in automatically

    Where is this done?

  2. +    // Filter out any unknown (unsupported) settings.
    +    $supported_settings = array_intersect_key($this->getSettings(), $default_settings);
    +    $this->set('settings', $supported_settings + $default_settings);
    

    We need a change record for this (either added to https://www.drupal.org/node/3381705, or in my opinion it would be better as a separate CR).

lauriii’s picture

Issue summary: View changes
Status: Needs work » Reviewed & tested by the community

Thanks for the review @effulgentsia! Looks like we had an outdated paragraph in the issue summary. We're not doing that anymore due to the DX concerns raised in #103.

Updated the CR. I think it's fine to have one CR since copying the default values should matter less.

lauriii’s picture

Something that's worth noting is that the deleting of unsupported keys is currently done in \Drupal\field\Entity\FieldConfig::preSave so adding that step to the postCreate() might not be as big of change as it seems.

  • effulgentsia committed e8787136 on 11.x
    Issue #2327883 by phenaproxima, lauriii, Wim Leers, yched, amateescu,...
effulgentsia’s picture

Status: Reviewed & tested by the community » Fixed

I read through this issue's comments, and I didn't see anything in them that concerned me about committing this as an interim improvement. The issue summary mentions that additional changes/APIs are likely needed, but those can happen in their respective issues.

Therefore, I pushed this to 11.x and published the CR. Great work, everyone!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

mxr576’s picture

It is more and more likely that the commited change breaks site install from existing config when Content Moderation is in use...

More details in #3441962-15: Call to a member function getSettings() on null in Drupal\Core\Field\FieldConfigBase->getSettings()