From #2320253-10: FIC/FSC::preSave() do not call the parent implementation :
not sure why we initialize default settings in preSave(), this leaves freshly created entities broken for runtime use (incomplete settings) until they are actually saved. We should do this as early as postCreate().
Similarly, FieldStorageConfig::$module is currently only populated in preSave(), meaning the entity is not reliable until that point. This should be done also in postCreate().
How did we end up with the current approach after several rounds of back and forth?
Entity reference fields -- that is, any field type based on \Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem -- need special handling. They have a setting called handler, which normally has a value of default that needs to be changed to something like default:taxonomy_term, depending on what kind of entities are being referenced, just before the field is saved. In HEAD, this is done on pre-save by the field module (field_field_config_presave()). This is also necessary for certain Entity Reference-based contrib modules, like Dynamic Entity Reference, to work properly.
BUT! If we want to make sure the settings are always complete and coherent, then we need to be able to do this normalization continuously, whenever field settings change, regardless of whether the field's been saved or not.
So, how to do that? A few approaches were tried, and it was a twisty, turny road:
- Initially, we thought it should be a special method of
\Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem, which the field config entity calls whenever settings are changed. - But that seemed icky and poorly encapsulated, so we moved it to a new method of
\Drupal\Core\Field\FieldItemInterface, thinking that the API addition was justified. - But then we thought we could just repurpose that interface's existing
fieldSettingsToConfigData()method for this purpose, but it turned out that was mismatched to our needs. - So we decided we'd add a new method to
FieldItemInterfaceafter all. - But then it was pointed out that we don't really need new API -- we could just do the normalization in
field_field_config_create(), since field entities are now created with complete, coherent settings. - After all of this, it was pointed out that none of the runtime config entities are actually valid because you would need to run
\Drupal\Core\Config\Entity\ConfigEntityStorage::mapToStorageRecordbefore they become valid. This is currently only run on save.
So that's where we ended up: when an entity reference field is created (with complete settings, now), those settings get normalized. Then they get normalized again when the field is saved. No new API surface! At least, not now.
We still may need new API to ensure that the settings are always normalized even if they undergo multiple changes (before saving). But, if that's necessary, let's do it in #3347291: Combine field storage and field instance forms and/or #3358049: Save FieldStorageConfig at the same time as FieldConfig, in which we'll be figuring out just how, where, and when a field's settings can be changed as it gets built in the UI by a site builder.
What about base field overrides, I hear you asking? Well...those already don't get normalized in HEAD, because field_field_config_presave() (which, remember, normalizes the field's settings) only acts on...field_config entities. Not base_field_override entities. So sure, it should be fixed, but it's not in scope here.
But...but...doing special normalization only for particular field type plugin classes sucks and is restrictive, you say. Yup, that's true. It should probably be applied to particular kinds of field types (for example, perhaps it could be done for anything in the reference category). But that's also not in scope here.
Basically: this thing we're doing, where we're implementing entity hooks for field_config entities, is a temporary measure. It gets us past the current, specific problem of having incomplete settings, and puts us in a position to add a well-designed, properly scoped API in place for continuous settings normalization in other issues, where it will become more relevant. Hook implementations aren't API, so we can easily remove them.
| Comment | File | Size | Author |
|---|
Issue fork drupal-2327883
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 2327883-field-storage-config
changes, plain diff MR !4371
Comments
Comment #1
yched commentedAttached patch removes the "defaults settings merging" from preSave() and adds it to postCreate() and postLoad(), in both:
- FieldConfigBase (= BaseFieldOverride & FieldInstanceConfig)
- FieldStorageConfig
Comment #2
yched commentedComment #4
yched commentedTurns out, FSC::preSave() needs to run before its parent because we need to populate $this->module before we can calculate dependencies.
Quick fix to see what the bot says for now, but waiting for preSave() to have the $module property populated looks wrong as well. the entity cannot be considered functionnal until that point. This should move to postCreate() too...
Comment #6
yched commentedDoh, #4 was actually the interdiff...
Reuploading
Comment #7
yched commented- Moving the initialization of FSC::$module from preSave() to postCreate()
- Removing the manual initialization of FSC::$id in preSave(), since no other config entity seems to bother with this.
Interdiff is with patch #1.
Comment #8
yched commentedComment #20
larowlanComment #21
andregp commentedBasic reroll of patch #7 for D9.3
Comment #22
ranjith_kumar_k_u commentedComment #23
ranjith_kumar_k_u commentedComment #24
andregp commentedReroll is returning too much fails, still needs work
Comment #25
yogeshmpawarTested patch locally for CS issues & didn't find one. problem with #22 & #23 is that they are tested against 9.4.x branch & issue version 9.3.x
So let's see this updated patch will work or not & hiding #22 & #23
Comment #26
yogeshmpawarComment #27
yogeshmpawarUpdated patch will fix test build issue.
Comment #29
andregp commentedToo much fails on the last test (almost 3k)
Comment #31
wim leersRunning into this over at #3324140-22: Convert field_storage_config and field_config's form validation logic to validation constraints.
Comment #32
wim leersComment #33
wim leersAnd TIL I was pinged to review/provide feedback on https://git.drupalcode.org/project/drupal/-/merge_requests/3926#note_189783 2 days ago, and that happens to run into exactly this problem space, but not
\Drupal\field\Entity\FieldConfig::preSave()specifically:field_field_config_presave()is doing things that are critical for certainFieldConfigentities (those that are for Entity Reference fields) to be correct/saveable.Increasing priority.
Comment #36
lauriiiCommitted a version of the patch that doesn't lead into WSOD to see how many test fails it causes. I expect there will be a lot of test fails because normalizing the config in preSave could be the expected behavior.
Comment #38
phenaproximaHiding patches in favor of the merge request.
Comment #39
phenaproximaI watched with increasing bafflement as @lauriii and @catch got into the weeds discussing this issue's backwards compatibility concerns on Slack. After that, I finally Zoomed with @lauriii to get the executive summary and understand it for myself.
The plan is this: we want the calling code to be responsible for merging the default settings into FieldConfigStorage::$settings (and its counterparts in BaseFieldOverride and FieldConfig). postCreate() will do this for you when it creates a new entity (by calling the $this->initSettings() method). In other situations, you need to do it yourself, and preSave() will check your homework -- $this->settings must contain every key in the default settings, or you'll get a deprecation notice (and maybe, in Drupal 11, an exception). postLoad()'s behavior will be unchanged.
Comment #40
borisson_Left a couple of comments on the MR, the remaining fails are all because of the new deprecation. Do we solve those in this issue as well?
Comment #41
phenaproximaAssigning to myself to fix the tests and find out which code paths are triggering these deprecations.
Comment #42
phenaproximaComment #43
berdirDid a review. issue summary definitely needs an update.
What I don't quit understand is why we plan to enforce that settings are passed in, we have the default settings, we can complete them, why not keeping doing that? That's how plugins work as well (19 instances of
$this->configuration = $configuration + $this->defaultConfiguration();in core, I think a ConfigurablePluginBase would be neat ;))IMHO, this is going to hurt us down the line. This would make adding a new setting to a field type plugin a BC break as all existing code who creates fields of that type is going to break.
Less sure about invalid settings, I can see how that's more sensible to enforce, but we would have the same problem then if we remove a setting from a field type, then that's suddenly invalid too.
Edit: Looking at this again, I think we are merging defaults on create, so this is "only" a problem if you manually change settings afterwards? For example calling setSettings() with an incomplete list of settings? I guess that is less of an issue then, still having 50+ failing tests in core shows that this seems fairly common.
Comment #44
smustgrave commented#43 mentioned an issue summary update.
And number of test failures in MR.
Comment #45
phenaproximaComment #46
phenaproximaUpdating the issue summary with the proposed resolution. I'm also tagging this for a change record, which I'm pretty sure will be needed.
Comment #47
phenaproximaComment #48
lauriiiThere's one more soft blocker to #3358049: Save FieldStorageConfig at the same time as FieldConfig related to invalid config which should be resolved either by this issue or a follow-up. The
field_field_config_presavenormalizes the selection handle plugin IDs and we need to manually duplicate that so that we can run it earlier. Thoughts on whether we should address that here or in a follow-up issue?Edit: note that this was mentioned by @Wim Leers in #33.
Comment #49
phenaproximaOkay, filled in the CR!
Comment #50
phenaproximaRe #48, I think we should do this in a follow-up. As you can see, I did a little experimentation around this but it seems rather tricky, and I'm not sure that experimental approach was the right one anyway.
Comment #51
wim leersFirst: AWESOME progress here! 🤩
Thanks for #39, that's really crucial context wrt BC concerns and release manager guidance! Linking it from the issue summary 👍 Makes sense that we don't want to modify
postLoad()at all in particular, because that would have the biggest BC (runtime) impact.I have to agree with @Berdir in #43: that simple approach seems preferable and less disruptive. But AFAICT since then, that was largely adopted. In reviewing this, I did spot a few more BC concerns though. 😅 Or at least, potential ones — hopefully you can show why none of them are actual BC concerns! 😄
Regarding @lauriii's remark in #48: I think that if we're making the behavior of
Field(Storage)Config's creation/saving logic sane & centralized, that it makes sense to considerfield_field_config_presave()in scope too — it's touching the same problem space.Comment #52
phenaproximaReassigning to Wim for another round of review.
Comment #53
wim leersThis is much closer to RTBC'ability now thanks to your work on
FieldConfigBase::set('settings', …)! It allows us to revert a LOT of changes.I see that you commented many times that that was impossible, but … I think you wrote those comments before you did that work? 😅 In my spot testing, it sure does work every single time!
Comment #54
wim leersAlso figured out why there was a weird deprecation-testing-related test failure. The root cause is simple: thanks to the excellent work you did on
FieldConfigBase::set('settings', …), the deprecation errorsimply cannot ever occur anymore! 😅
And I'm 99% certain that this also explains why
FieldSettingsTestis now failing: the expectations just need to be updated thanks toset('settings', …)no longer being able to add invalid keys 👍Comment #55
wim leersBefore this can be RTBC'd, this definitely needs an issue summary update.
Also, thanks to
FieldConfigBase::set('settings', …), I think the CR needs to be updated? 🤓Comment #56
phenaproximaComment #57
phenaproximaComment #58
phenaproximaDo we still need a follow-up here?
I ask because...as far as I can tell, it's no longer possible to put incomplete settings on a field or field storage config entity, unless you use reflection. So rather than deprecate, we can just throw an exception.
Comment #59
wim leersLooking great!
But in doing a thorough final review, I made one very startling discovery:
\Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigData()has existed since 2014 and AFAICT does exactly the same thing as the new\Drupal\Core\Field\FieldItemInterface::normalizeSettings()we're adding?! 😳Comment #60
phenaproximaComment #61
wim leersI see a way forward thanks to https://www.drupal.org/node/3376455! 🥳
Also still needs a change record for the migration change.
Those are the only 2 remaining threads 👍
Comment #62
wim leersThis is ready, but still needs change record updates, as well as 2 additional change records (the new interface argument and the modified migration destination). It also still needs a follow-up for the
@todothat was just added for Drupal 11.Once those things are done … this will be RTBC! 😄🥳
Comment #63
phenaproximaOkay, the change records have been added/updated.
Comment #64
phenaproximaOpened #3379022: [Drupal 11] Require $field_definition to be passed to FieldItemInterface::fieldSettingsToConfigData() to change the interface method in Drupal 11, and linked to it in the todos.
Comment #65
borisson_I have read trough the PR several times, in the hope to find something that was not clear, could use better documentation or can otherwise be improved. I can't seem to find anything.
I also read through the Change Records, those are very clear as well.
I think that means this can go to rtbc, but I would feel more comfortable with one of the field or entity api maintainers +1 here as well. Do we need to tag this as needs subsystem maintainer review?
Comment #66
wim leersAgreed with @borisson_; tagged 😊
Refined the migration destination CR, added link to #3379022: [Drupal 11] Require $field_definition to be passed to FieldItemInterface::fieldSettingsToConfigData() to the
fieldSettingsToConfigData()CR.The third CR (for field config/storage settings) sounds … a bit like it's breaking quite a lot of things? 😅🙈
So let's see if all those
*Test.phpchanges are necessary, by reverting all modified tests. It's easy to get those by comparing to the last upstream commit that was merged in:Because if we could show that none of those test changes are necessary, then we simultaneously prove that the risk of disruption/BC break is extremely low. And … unless I made a mistake, every single one of the tests still passes if I revert all test changes! 🤩 The changes were definitely necessary at an earlier iteration of this merge request, but now they're no longer necessary: thanks to
core/lib/Drupal/Core/Field/Entity/CompleteFieldSettingsTrait.php, and the updates tocore/lib/Drupal/Core/Field/FieldConfigBase.phpand subclasses, everything is guaranteed to be complete, always! 😊The sole exception:
\Drupal\KernelTests\Core\Field\FieldSettingsTest. But that makes sense: it's literally testing the updated behavior in a unit-like kernel test that checks howField(Storage)Configentities are created and how their settings are saved 👍Comment #67
wim leers**Test.phpdidn't catch everything. Reverting the changes inFileFieldTestBase.phpandImageFieldCreationTraitappears to work fine too 🤓Now, to be fair, the code that I reverted in the last 2 commits was:
But AFAICT this means that the third CR (https://www.drupal.org/node/3375334) can simply be deleted — or at least associated with #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints instead of this issue! 😊 Because as
FieldEntitySettingsTestalready says:IOW: none of the work was in vain, but let's keep this issue's scope as tight as possible and hence make it easier to get this issue committed 🤞
This also should make subsystem maintainer review a lot simpler.
Comment #68
wim leersStill green. This is IMHO RTBC now.
I would like explicit confirmation for the 2 commits I pushed from @phenaproxima and @borisson_. After that, we need subsystem maintainer review.
Comment #69
phenaproximaI have no objections to Wim's changes. Fewer files touched is good, and indeed, we can delete the third change record -- even nicer!
Comment #70
borisson_Fewer touched files is indeed more better.
Comment #71
wim leersAssociated the 3rd CR with #3324140 instead 👍 2 CRs left here.
Comment #72
amateescu commentedPosted a couple of comments in the MR.
I think a possible solution to the first problem (calling
fieldSettingsToConfigData()inFieldConfigBase::postCreate()) is to move the code which sets the handler setting for ER fields fromfield_field_config_presaveto ahook_entity_create()implementation.Comment #73
wim leersThanks, @amateescu! 😊🙏
Assigning to @phenaproxima to address your remarks, but … I did ask 2 clarifying follow-up questions on the MR, and I have one other question for you here: 🤓
It sounds like this means that my discovery/"aha" moment in #59 was wrong? 😅 @phenaproxima was actually first adding a new
FieldItemInterface::normalizeSettings()method, and then in doing a final review, I discovered::fieldSettingsToConfigData(), which I'd never heard of before, and it seemed to be doing the exact same thing.Comment #74
amateescu commented@Wim Leers, unfortunately.. yes :) I learned about
fieldSettingsToConfigData()from this issue, but then I read the issue where it was introduced and realized that it should only be used prior to saving a field to config.As for a test case about not calling
fieldSettingsToConfigData()inpostCreate(), a very good example is\Drupal\options\Plugin\Field\FieldType\ListItemBase::storageSettingsToConfigData()and\Drupal\options\Plugin\Field\FieldType\ListItemBase::storageSettingsFromConfigData(), which show how/why a field type might need to have a certain structure for its settings in the runtime object, and a different settings structure when stored in config.Comment #75
wim leersOkay, great, thank you SO MUCH for confirming that!
Sounds like we should restore the
::normalizeSettings()approach! 😇Comment #76
phenaproximaComment #77
wim leersThis is ready now, although an explicit confirmation from @amateescu sure would be nice! 😇
Comment #78
amateescu commentedIs there a problem with the
hook_entity_create()suggestion for setting the handler setting of ER fields? A hook implementation has the advantage that it can be removed at any time if we refactor the code around selection plugins and their special handling of plugin IDs, while a new method onFieldItemInterfacewould take a major release cycles to deprecate and remove.Comment #79
wim leersYes, there is a significant problem with that: it gets us the same exact problem that
field_field_config_presave()causes today: modules like https://github.com/jibran/dynamic_entity_reference then have to hack their way around it withhook_module_implements_alter()because these hooks cause the field's (entity reference selection handler) settings processing to happen at the wrong level!See #33, which points to @larowlan's comments, and most critically, to https://git.drupalcode.org/project/drupal/-/merge_requests/3926#note_194955. See https://git.drupalcode.org/project/dynamic_entity_reference/-/blob/3.1.0....
Basically: while we're touching this code, we might as well make it so that core won't be breaking contrib modules in this way anymore.
Comment #80
amateescu commentedThat's very easy to fix though: we can just check if the field type provides a
target_typestorage setting and return early if it doesn't.Comment #81
wim leers#80:
but that will work only for this particular subclass ofAha, no, becauseDrupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem. i.e. it'll work forDrupal\dynamic_entity_reference\Plugin\Field\FieldType\DynamicEntityReferenceItembecause that is the one that happens to not settarget_type. But what if a different subclass does something different? 🤔target_typeis the only additional key that\Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem::defaultStorageSettings()specifies! So yes, then this could totally work, and we avoid increasing the API surface.I think the solution that we had arrived at here is nice & fine. It seems that @larowlan, @jibran, @phenaproxima and I were all fine with it (maybe @lauriii too, not sure). But it does mean expanding the Field API more, and if we can avoid that, that'd be better. It'll also mean a simpler upgrade path from Drupal 10 to 11, which in its own is worth a lot (i.e. no field type implementation changes necessary!).
So: +1 to restoring
field_field_config_presave()and returning early iftarget_typeis absent from the default storage settings. Thank you for helping keep scope & impact as minimal as possible, @amateescu! 😊🙏🙏🙏Comment #82
wim leersAfter these twists and turns, the IS became outdated again, and we're missing a change record for the solution that @amateescu helped us arrive at. Tagging for both.
Comment #83
amateescu commentedThanks for the updates, this looks much cleaner now. Left two more comments on the MR then this can be set to RTBC.
@Wim Leers, if you mean that we need a change record for the
field_field_config_presave()update, we can just tell @larowlan and @jibran about it and save a few hundred people from reading stuff they'll never have to care about :)This CR needs to be deleted though.
Comment #84
phenaproximaOK, I've made the requested changes here. Re-assigning to Wim for a final review, now that we have subsystem maintainer approval, and for change record/issue summary updates.
Comment #85
phenaproximaComment #86
lauriiiIf I read #72 and #78 correctly, it's proposing moving the entity reference normalization from
hook_field_config_presavetohook_field_config_createwhich should be sufficient to ensure that entity reference fields are valid after creation.However, the MR currently keeps the normalization in
hook_field_config_presavewhich is insufficient because it means that the handler is normalized only at the saving stage. It also looks like tests are passing which means that we have insufficient test coverage.I confirmed that the MR as it is is not sufficient by confirming that
\Drupal\Tests\field\Functional\EntityReference\EntityReferenceAdminTestfails with the current MR and a version of #3358049: Save FieldStorageConfig at the same time as FieldConfig that doesn't have the hard coded normalization as part of field creation.Comment #87
lauriiiDiscussed with @phenaproxima on Slack because there were some confusion between comments #72 and #81. As I mentioned in #86, I understood #72 and #78 so that @amateescu is proposing to move the entity reference normalization from
hook_field_config_presavetohook_field_config_create. I said in #86 that it should be sufficient to ensure that entity reference fields are valid after creation.However, @phenaproxima pointed out that it won't be sufficient because the handler settings may change during the life cycle of the field config. This is why we would need at least
hook_field_config_presaveandhook_field_config_createto normalize the config. However, this defeats the whole purpose of this issue because the intent is also to be able to validate the config before saving it. For this reason, we would need the normalization to happen on a separate layer, that could be triggered before saving.I'm wondering if this provides sufficient justification for the new API that we were proposing to add until #72 😊
Comment #88
lauriiiDiscussed with @amateescu on Slack and he said that even with the justification in #87, he would prefer to introduce the new API as part of #3347291: Combine field storage and field instance forms. I can see how that makes sense because then we can make sure that the API is flexible enough to support the needs of that issue.
Doing the entity reference normalization both, on
hook_field_config_createandhook_field_config_presaveaddresses the bug where field config entities are invalid after creation which blocks both #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints and #3358049: Save FieldStorageConfig at the same time as FieldConfig. We can't removehook_field_config_presavewithout introducing the new normalization API because existing code may be relying on it, e.g.field_field_storage_config_updatein core.I think it would be great if we could add test coverage that I asked in #86 to make sure that this doesn't regress until we have the additional test coverage from #3358049: Save FieldStorageConfig at the same time as FieldConfig.
Comment #89
jibranDoes that mean DER can remove the hook implementation alter and implement it's own
hook_field_config_createand update existinghook_field_config_presave?Comment #90
lauriii@jibran yes, the current implementation would allow you to remove
hook_module_implements_alterfrom DER once 10.1.x is EOL 😊Comment #91
wim leersThis is ready, just needs a change record. Since this is blocking high-impact usability issues, and the CR would affect so few people, I think it'd be fine to land this ahead of the CR existing 😇
Comment #92
tim.plunkettI'm not sure if we'll get away with not having a CR but it would be nice to have an updated IS 😇
Comment #93
wim leers@phenaproxima will tackle the issue summary update + change record tomorrow.
But more importantly, @Berdir has surfaced concerns with the current approach, which was proposed by his fellow subsystem maintainer @amateescu 😄 So, I think we really need @amateescu's input now.
(I did point @Berdir to the relevant comments though, so maybe that is already enough? 😇)
Comment #94
berdirWe discussed my feedback in slack and I'm fine with addressing that in a follow-up.
In short, the situation for DER is indeed improved as I realized in my second comment due to the target_type check, I still think this should target specific field types only or at least give subclasses a way to customize it, but it is in line with the existing logic and behavior and that's exactly what the follow-up is about.
Comment #95
lauriiiUnassigning since @amateescu was part of the discussion mentioned in #94 and it sounded like everyone was onboard with the current solution.
Still needs an issue summary update and change record 😇
Comment #96
phenaproximaPartial IS update.
Comment #97
phenaproximaComment #98
wim leersIS looks great!
Thanks for deleting that one CR, and keeping the other CR (https://www.drupal.org/node/3379013), that one is still relevant.
Comment #99
phenaproximaChange record written and the issue summary is rounded out. https://www.drupal.org/node/3381705
Comment #100
wim leersThanks for https://www.drupal.org/node/3381705!
Let's do this :D 🚢
Comment #101
wim leers🧐 Let's see if the deprecation @lauriii and I thought would make sense triggers failures in core's test suite. Note that for now, I did not yet do the conditional part. If there are failures specifically for
FieldStorageConfigsettings being set onFieldConfig, the commit I just pushed should fail tests.EDIT: we expect this to fail,and that’s a good thing, we’ll then know where @lauriii's excellent observation would’ve caused disruption in contrib! 👍
Comment #102
lauriiiMoving to needs review because there has been changes to the MR that would benefit from further review.
Comment #103
lauriiiDiscussed with @Wim Leers, @phenaproxima, @tedbow, and @tim.plunkett.
First of all, there are 2 hard requirements:
We cannot normalize in
::setSettings()because it's possible for any contrib module to call::set('settings', …)instead. (Plus,\Drupal\Core\Entity\EntityForm::copyFormValuesToEntity()also calls::set(…)!)👆 Hence the current approach in the MR: to normalize during both
::set()and::setSettings().⚠️ However, we discovered that we cannot do it in
::set(), ⚠️ because it could introduce very complex issues to debug. The key challenge is that::set()is being called\Drupal\Core\Config\Entity\ConfigEntityStorage::doSaveafter it has called\Drupal\Core\Config\Entity\ConfigEntityStorage::mapToStorageRecordand\Drupal\Core\Config\Entity\ConfigEntityStorage::mapFromStorageRecords\Drupal\Core\Config\Entity\ConfigEntityStorage::doLoadMultiple… meaning that normalizing in
::set()does lead to differences in settings depending on where in the life cycle the config entity is, i.e. after saving the entity, it would be in the normalized state, but after loading it from the database, it would NOT be in the normalized state. This is highly confusing, and breaking BC for\Drupal\Core\Config\Entity\ConfigEntityStorage::mapFromStorageRecordsduring save.This leaves only one option that we can see: restore
FieldItemInterface::normalizeSettings(), make it a public method, so that it can be called before saving the Field(Storage)Config from Field UI, and when validating config.A tangentially related problem is that the sole thing in Drupal core testing this is
\Drupal\field_test\Plugin\Field\FieldType\TestItem::storageSettingsToConfigData()+\Drupal\KernelTests\Core\Field\FieldSettingsTest::testConfigurableFieldSettings(), butTestItemmethods are adding NEW keys instead of transforming the values of EXISTING keys. The latter was always the intent — see #2293773: Field allowed values use dots in key names - not allowed in config. So we should fix the test coverage aroundTestItem::classbecause it's currently asserting things that do not make sense.In the absurd case that something in contrib is relying on this: we can and should detect incorrect implementations of
toConfigData()andfromConfigData(): any key-value pairs that it returns MUST also exist in\Drupal\field_test\Plugin\Field\FieldType\TestItem::defaultStorageSettings(). If that's not the case, that should trigger a\LogicExceptionor at least a deprecation. See https://drupal.org/node/2307365 for the change record when the API was introduced.Comment #104
lauriiiDiscussed with @amateescu on Slack: https://drupal.slack.com/archives/C1BMUQ9U6/p1692622430919759.
The key insight is that @amateescu pointed out that we provide an API that allows runtime field [storage] config entities to NOT be valid config. The API that allows this is
\Drupal\Core\Field\FieldItemInterface::storageSettingsToConfigDataand\Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigData, because these are required to have run for some field types before they can be valid config entities to store. Given that there is a difference between the stored config entity, and the runtime entity, he also pointed out that he doesn't see why the solution in #2293773: Field allowed values use dots in key names - not allowed in config should be limited to massaging existing values.Given this, to be able to validate field [storage] config, we'll need to trigger
\Drupal\Core\Field\FieldItemInterface::storageSettingsToConfigDataand\Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigDatabefore triggering config validation. This could be done in an ephemeral object used for validating the config.The proposed changes to the current MR:
postCreateandpreSaveset,setSetting,setSettingsto avoid introducing changes with the runtime field config objecttrigger_errorcode from\Drupal\Core\Field\FieldConfigBase::mergeDefaultSettingsto\Drupal\Core\Field\FieldConfigStorageBase::mapToStorageRecord. This should be run after\Drupal\Core\Field\FieldItemInterface::storageSettingsToConfigDataand\Drupal\Core\Field\FieldItemInterface::fieldSettingsToConfigDatato make the callbacks responsible for normalizing the settings before saving.Comment #105
lauriiiImplemented #104 with the exception that I left the
preSavechanges to a follow-up. There's around 50 tests failing if we introduce the deprecation. I think we may want implement the changes topreSave, but that's not needed for either of the blocked issues. I think we could open a follow-up for that.I'm wondering if we could now revert the migration related changes too and move them to a follow-up? Maybe @phenaproxima could confirm that.
On top of that, we need to update CRs and the issue summary.
Comment #106
lauriiiUpdated the issue summary and the change records.
Not going to file an explicit follow-up: we can define what we need in #3324140: Convert field_storage_config and field_config's form validation logic to validation constraints.
Comment #107
amateescu commentedThanks for the updates, @lauriii, let's move forward! :)
Comment #108
wim leersI dug through this in detail with @laurii and I'm confident that we can make this work for config validation support in
Field(Storage)Config+BaseFieldOverride.One thing I definitely hadn't thought of yet so far is how config entity objects may have different values in their runtime state vs their storage state, thanks to
\Drupal\Core\Config\Entity\ConfigEntityStorage::mapToStorageRecord()! 😳 Fortunately, only 3 classes in core override it:\Drupal\Core\Field\FieldConfigStorageBase::mapToStorageRecord()\Drupal\field\FieldStorageConfigStorage::mapToStorageRecord()\Drupal\layout_builder\Entity\LayoutBuilderEntityViewDisplayStorage::mapToStorageRecord()… plus 6 in contrib, 5 of which are virtually identical to core's Layout Builder one.
This means that for config entity validation to work, the infrastructure that provides validation for config entities must also be able to call
mapToStorageRecord()not just prior to save, but prior to validation. But that method is currentlyprotected😅 So we need some way to get config entities into the same state as just before saving. We may want to do that as part of #2907163: Split validation-related methods from FieldableEntityInterface into a separate ValidatableInterface or in another issue.In any case, it's clear that it's possible, that this change won't block it. That's the thing we wanted to make sure.
This has been very thoroughly reviewed, and the end result after >100 comments is a far smaller patch with a tighter scope than before subsystem maintainers started reviewing it. Special thanks to @amateescu & @Berdir! 😊
EDIT: cross-post RTBC 😅🥳
Comment #109
effulgentsia commentedOverall, I think this patch makes a lot of sense. I'm glad it ended up with a fairly simple implementation. Since there might be some subtle BC breaks here, before committing it I want to read through the comment history to see what's been discussed as far as that goes. In the meantime, these are the only two things that jumped out at me:
Where is this done?
We need a change record for this (either added to https://www.drupal.org/node/3381705, or in my opinion it would be better as a separate CR).
Comment #110
lauriiiThanks for the review @effulgentsia! Looks like we had an outdated paragraph in the issue summary. We're not doing that anymore due to the DX concerns raised in #103.
Updated the CR. I think it's fine to have one CR since copying the default values should matter less.
Comment #111
lauriiiSomething that's worth noting is that the deleting of unsupported keys is currently done in
\Drupal\field\Entity\FieldConfig::preSaveso adding that step to thepostCreate()might not be as big of change as it seems.Comment #113
effulgentsia commentedI read through this issue's comments, and I didn't see anything in them that concerned me about committing this as an interim improvement. The issue summary mentions that additional changes/APIs are likely needed, but those can happen in their respective issues.
Therefore, I pushed this to 11.x and published the CR. Great work, everyone!
Comment #116
ashetkar commentedComment #117
mxr576It is more and more likely that the commited change breaks site install from existing config when Content Moderation is in use...
More details in #3441962-15: Call to a member function getSettings() on null in Drupal\Core\Field\FieldConfigBase->getSettings()