Versioned Component config entities (SDC, JS: prop_field_definitions, block: default_setting, all: slots for fallback) + component instances refer to versions ⇒ less data to store per XB field row

This is very similar to the explicit input schema change strategies outline I wrote on #3520484: [META] Production-ready ComponentSource plugins.

The key difference: I tried to do it generically, not restricted to SDCs & code components. So, for example, Block plugins may change (evolve) their settings and provide an update path, and so XB should be able to apply such an update path. Layout Builder hasn't solved this yet either — see the #3521221: Block plugins need to be able to update their settings in multiple different storage implementations core issue

Specifically doing this for only the sdc and js component sources does not make that much sense to me yet. What problem does that solve? 🤔

To avoid the need to look up in some central place what field type + storage settings + instance settings + widget should be used for a given SDC prop is … exactly why StaticPropSources contain all that information, and why that is stored in the inputs field property of the XB field type.

And … I think that is what you're aiming to solve here: storing less data in inputs. Even though you omitted it from the issue summary: you only showed the before vs after for the tree field property, not for the inputs field property 😄 Updated issue summary based on this interpretation. But I'd like you to confirm it. 🙏

Very closely related, and one of the oldest XB issues: #3463996: [META] [PP-1] When the field type, storage/instance settings, widget, expression or requiredness for an SDC/code component prop changes, the Content Creator must be able to upgrade.

FYI over at #3515629-16: FieldWidget's XB transforms must be bubbled by the Field Widget rendering to inform the client.2, I sketched how I now think this could actually happen automatically.

From @larowlan's issue summary:

1. Adding a new required prop - this is relevant if we go down the non JSON storage approach
2. Removing a value from an enum
3. Changing the mapped field type
4. Removing a prop - although the net outcome here is redundant data in the database for old components - so we might not care here

I don't think I agree with stating these are things we have to support:

1. This is sort of a BC break, except we could automatically fix this: detect the stored version not matching the current version, and add the missing default — both at render time and at edit time, which means after saving it'll have been automatically fixed.
   EDIT: see #3509115-6: [META] Plan to allow editing props and slots for in-use code components for a more detailed description of this.
2. This is a BC break we cannot recover from: either we drop the user's choice (data loss) or the rendering crashes — which is exactly why #3485878: Server-rendered component instances should NEVER result in a user-facing error, should fall back to a meaningful error instead (+ log) has explicit test coverage for this.
3. This is what hook_storage_prop_shape_alter() allows and what #5 is about. And it's what allows the storage efficiency improvement described in #4.
4. Equivalent to point 1, except that here too there would be data loss — especially if a future revision of the SDC restores this. Note that redundant data (aka prop sources for non-existent props) will not be accepted by \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\GeneratedFieldExplicitInputUxComponentSourceBase::validateComponentInput().

So: what is your purpose with this issue, @larowlan? Storage efficiency, input schema evolvability, both, or something else still?

I agree that slimming down the inputs to remove a lot of duplicated data is worthwhile, although this feels like it adds a huge amount of complexity - and would this be better off as some kind of new core API?

We also still have to handle error cases: because config is mutable we might still end up with references to versions that no longer exist in the config object.

Could this be different config entities for different 'versions' of the component, instead of trying to store all version information in a single config entity?

That would potentially make it easier to determine whether any content is using the old version, allow old versions no longer in use to be deleted etc.

@larowlan in #7:

both

Hah! I should've known 😅

@longwave in #8:

I agree that slimming down the inputs to remove a lot of duplicated data is worthwhile, although this feels like it adds a huge amount of complexity

+1 to both those statements.

would this be better off as some kind of new core API?

🤔 I'm not sure I follow. Are you thinking of #3469082: Add way to "intern" large field item values to reduce database size by 10x to 100x for sites with many entity revisions and/or languages? If so: I don't see how that applies here, because this is not about "interning", but about compacting? If not that: can you elaborate? 🙏

@catch in #9: I don't think there will be a big number of "versions" of a particular component. So storing all versions in a single Component config entity seems reasonable. Especially because we aim to allow instances using the older version to be able to transition/update to the newer versions (until none of the old version are left, and the old version can be deleted).

That being said … making the version part of the Component config entity ID would make what I summarily described in #3520449: [META] Production-ready data storage trivial:

Expand on what #3457504: XB field type: calculate all dependencies, store them, surface in new Component "Audit" operation did: omit Component config entity dependencies from deps_config, and introduce a separate component_versions field prop, to store <component config entity ID>:<version> <component config entity ID>:<version> … → allows finding which revisions are using not the latest version of a component

So that is definitely worth considering, too.

We'd just forcefully disable older versions of a Component — only one version is allowed to have status: true, and hence only that one will be served to the XB UI by /xb/api/v0/config/component, and that way everything will continue to work exactly as it does today 🤓 (The client side is completely oblivious to these details; all it cares about is that it's given some opaque string to identify a component to instantiate.)

I think @larowlan has a good (implied) point: if it's okay for us to rely on "external data" for DynamicPropSources (aka the FieldConfig entity, the host entity an instance lives in, etc.), then why is it not for StaticPropSources?

So: let's get a working PoC and then re-assess 👍

@larowlan Given that this will touch the Component config entity implementation, its config schema, and (Static|Dynamic)PropSource, I think it probably makes sense for me to take this one on. Then you can tackle #3468272: Store the ComponentTreeStructure field property one row per component instance at the same time? 🤓

If so: I don't see how that applies here, because this is not about "interning", but about compacting? If not that: can you elaborate?

If we have some concept of versioned config entities then maybe that can help with #3521221: Block plugins need to be able to update their settings in multiple different storage implementations as well? We can store references/dependencies to specific versioned config, plus have some mechanism that can (somehow) migrate to the newer versions as required.

@longwave: yep, see my response to @catch above :)

Getting this going 👍

#15 sounds very encouraging.

I like it! Wish I thought of it! 😄

Consider it done. I'll be spending my AM getting this going! 🤓🕺

Got the foundations ready:

1. type: versioned_config_entity, a subtype for type: config_entity
2. an accompanying \Drupal\experience_builder\Entity\VersionedConfigEntityBase
3. moved the settings.local_source_id to the top level at source_local_id (sibling to HEAD's source) — which makes
4. moved settings.* (aka everything besides the intra-source plugin ID) to versioned_properties.settings.* — which makes them versionable 👍
5. moved fallback_metadata (currently only slot definitions) to versioned_properties.fallback_metadata
6. thanks to this, there's no more need for changing source — we can use the version itself to determine when to use the Fallback component source plugin

The 4 crucial tests largely (but not completely) pass ((Block|SingleDirectory|Js)ComponentTest + ComponentValidationTest). Initially ~58% was passing , then ~84% passing and at end of day 98.57 % passing.

Key next step (tomorrow): idempotently computing the version hash.

1. I said idempotently computing the version hash was the next step. I forgot about one other one: validation of the new structure, to ensure >=1 version exists, that the active version points to an existing version, that versions have a consistent format, and … making source_local_id immutable.
2. Now then, the computing of that version hash: implemented (including validation, again), but the validation is giving me headaches. 😅

   I begrudgingly added recursive key sorting (copy/pasted from AutoSaveManager::recursiveKsort()), which allowed me to get tests to pass more for SDCs, but still not completely. We shouldn't do that sorting here though, because the order could be meaningful.
   Worse, that wasn't enough. Root cause AFAICT: the configuration system upon saving sometimes converts true to 1 and false to ''. This then unsurprisingly results in the hash not being the one that was expected.

   I have one bit of hope: applying the same trick as \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\BlockComponent::fixBooleansUsingConfigSchema() — i.e. use config schema to perform the appropriate normalization. We'll see if that works out. If it doesn't, I think deferring validating the generated versions to a follow-up issue is reasonable. All of the validation at the very start is what truly matters for data integriy — this was
   going the extra mile. 🤞

🤯
Core's got this:

field.value.boolean:
  type: mapping
  mapping:
    value:
      type: integer
      label: 'Value'

… but that should obviously be type: boolean.

Fixing that alone makes a big difference: before (83.29% passed) vs after (86.89% passed).

I'm trying to understand what the implications of this are if someone changes things then changes them back again.

Will it be:

version 1
version 2
version 3 (identical to version 1)

Or... will it be:
version A
version B
version A

?

For an upgrade path, version 1 would need to be updated to version 2, then to version 3, but Version A would be able to remain version A if it hasn't already been updated to version B.

Apologies if this is hard to parse, that's the third rewrite..

Now using config schema as described in #20.2. That brings us down to 86.18% passing.

To avoid a single piece of (very detailed) validation to cause an already big MR to grow a lot in size (currently: 29 files, +862, −422), I commented out a single config schema line which gets us back to ~98% passing 👍

We could live without this validation although it IMHO would be preferable not to.

I have to switch to other work now. To be continued, probably later this week. Let's hear from @larowlan if this is what he had in mind. 😊

Next steps here (beyond getting all tests to pass) are to update the tree and inputs data structures — although preferably that'd happen after @larowlan's #3468272: Store the ComponentTreeStructure field property one row per component instance is merged, to avoid huge conflicts and significant rework. Marking [PP-1] for that.

@catch in #22: for the actual upgrade path logic, see:
#3524751: [later phase] Component Source plugins: generalized support for schema changes of explicit inputs, with within that:

• #3501708: Prove that it *will* be possible to apply block settings update paths (assuming #3521221 in core) to stored XB component trees in config/content → your core issue: #3521221: Block plugins need to be able to update their settings in multiple different storage implementations
• changing decisions about which field types/storage+instance settings/widget to use: #3463996: [META] [PP-1] When the field type, storage/instance settings, widget, expression or requiredness for an SDC/code component prop changes, the Content Creator must be able to upgrade
• … more to follow

This issue should IMHO be focused solely on:

1. Making the Component config entity know about/remember different versions
2. Updating the stored component instances to also store the version that was active at the time of their creation
3. … and creating that version hash solely based on the data stored in type: experience_builder.component_source_settings.(block|js|sdc), which itself actually is the result of a computation in the respective ComponentSource plugins

Observations for those 3:

1. It's not about updating existing component instances. This is what you're referring to AFAICT.
2. Perhaps this second bit ought to be split out — even if it's just another MR on this same issue. Because as just explained in #23: this is a pretty big MR already.
3. It's also not about making ComponentMetadataRequirementsChecker and friends detect that the explicit input schema has changed, judge whether that's feasible or not, and then for example disable the affected Component config entity. This is why my 3rd "solely about" point is so narrow.

@WimLeers, well this has implications for the upgrade path.

There's a huge difference between:

version 1
version 2
version 3 (identical to version 1)

and:

hash 1
hash 2
hash 1 (identical to version 1).

in that there'd be no need to update hash 1 to hash 1, which would be good. I think that's what this issue results in, but it would be good to confirm.

in that there'd be no need to update hash 1 to hash 1, which would be good.

That's indeed what I'm aiming for, but per #20.2 and #23, I'm running into challenges that AFAICT are caused by the config system modifying the contained values 😅

Deterministic hashing also requires a deterministic input, and that's where the config system is currently getting in the way. That's the bit I'd want to push to a follow-up issue, because that likely requires complex tweaks and shenanigans. This MR in and of itself already moves us to a far better place.

Plus, we can completely transparently recompute the hashes at a future time, given they DO only rely on the stored settings. 👍 The problem is that those settings themselves are neither ordered nor casted deterministically today. AFAICT \Drupal\Core\Config\StorableConfigBase::castValue() should help us get there, but that is AFAICT not available to me for use on arbitrary config subtrees: \Drupal\Core\Config\Config::save() calls it and nothing else does. I'd have to do some shenanigans for XB to be able to use that. That could easily end up being a day-long and multi-100-LoC endeavour, which is why I'm advocating for focusing on getting the mechanics right in this MR.

@larowlan in chat:

In comment #17 you mentioned that you liked this approach proposed by alex and I (#15) - and catch was also in favour of it (#16) but that doesn’t seem to match what’s in the current MR.

Hah, I think it’s actually very simple: I wrote that on May 16 (Friday). Then I didn’t get to spend my AM that day on it like I said/hoped. 😑

I actually started on Monday, May 19. And I literally started from the POV: “how can we do this generically?” — because we likely want more versioned config at a later point. For example, product requirement 37. Revisionable templates — but I know that @lauriii has expressed the intent to do it for more config than that.

Then I looked at the issue summary, and what I'd done was perfectly on track to achieve that.

tl;dr: I simply forgot about #15 🙈🙈🙈

@larowlan also asked this in chat:

I still think the current approach you've taken (versioned config entity) means we can do a dynamic field union thing, so its not a deal breaker. But keen to understand the direction taken.

I kinda answered this above (basically: "oops!"), but digging into it a bit to help ensure the direction you end up taking takes some additional considerations into account 🤞

Having worked on both this and #3468272: Store the ComponentTreeStructure field property one row per component instance, I now have some questions about #15 (which looks super familiar having spent the entire day reviewing #3468272 🤠):

1. the "to" example lists the inputs key twice. That's accidental, right? I think It'd essentially be:
   

   …
       component_id:
         type: string
         label: 'Component config entity ID'
         constraints:
           ConfigExists:
             prefix: experience_builder.component.
       shape_matched_field_union_id:
         type: string
         label: 'Shape-matched auto-generated field union, if any'
         constraints:
           ConfigExists:
             prefix: experience_builder.shape_matched_field_union.
         # NULL for block components, used only for ComponentSource\GeneratedFieldExplicitInputUxComponentSourceBase-powered Component Sources
         nullable: true
   …
      

2. What would sensible identifiers for the new config entity type (as you can see, I suggest ShapeMatchedFieldUnion) be?

   Is it the component ID that it was generated for? That'd make it behave much like the FilterFormat–Editor twin/sibling pairing. In that case, I'd imagine you'd want to use type: versioned_config_entity and VersionedConfigEntityBase .

   Or would the ID be something that's based on the contents of the field union? The appealing thing about that is that it'd allow multiple Components to use the same ShapeMatchedFieldUnion. But: how often would that even happen?

3. But … actually that shape_matched_field_union_id key-value pair that references another config entity … must actually itself be versioned, because we need to be able to know for a given Component config entity what all of its previously used field types/instance+storage settings were (aka all of the ways their explicit inputs have ever been stored).
   Without that, we won't be able to do #3524751: [later phase] Component Source plugins: generalized support for schema changes of explicit inputs (adding as related) nor #3463996: [META] [PP-1] When the field type, storage/instance settings, widget, expression or requiredness for an SDC/code component prop changes, the Content Creator must be able to upgrade (already linked).

I'm behind on the past handful of comments and only replying to #29 and #30 right now, hopefully doesn't make things more confusing.

The two things I think we benefit from something field union-ish:

1. A coherent representation of what field types are in use for the values that now get stored per row with component_id | inputs in #3468272: Store the ComponentTreeStructure field property one row per component instance e.g. component A includes field types foo and bar. This ought to help with dependency calculation and various other things.

I think that's what this issue is explicitly about.

2. The potential to use field unions as part of an overall data model in general

e.g. field unions being available as structured fields which can be laid out by XB, without having to be in exposed slots (the album track listing and other examples). It feels very likely that if this isn't available, people will hack together similar functionality via exposed slots, or continue to use paragraphs for that. It may not be necessary for XB iin tself, because for XB a field is a field, but it might be necessary for XB to displace paragraphs and/or circumvent people coming up with horrible workarounds for this use case.

I had hoped in #3440578-80: [PP-2] JSON-based data storage proposal for component-based page building that XB could 'build upwards' from a dynamic field union so that the XB field would be something like a compound field that included the dynamic field union + the extra slot and parent data. This is also a bit like what some of the approaches discussed in #3477428: [PP-1] Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree, and to reference the field_union type of the prop values were like.

Now that #3468272: Store the ComponentTreeStructure field property one row per component instance is in, it's much closer to this, but not quite the same, but it feels like this issue could bring it a step closer again, and #15 seems like doing that.

e.g. I could imagine a dynamic_field_union module having a field where the two columns are 'field_union' (config entity reference) and 'values' (JSON).

And I think that #15 is saying that if XB splits the config entity, then there would be no difference between those config entities, so that they could potentially even be shared between the XB field and a field union module, or one could extend from the other, or something like that. And then XB would then have a second config entity for any extra bits that aren't explicitly about the field types.

But … actually that shape_matched_field_union_id key-value pair that references another config entity … must actually itself be versioned, because we need to be able to know for a given Component config entity what all of its previously used field types/instance+storage settings were (aka all of the ways their explicit inputs have ever been stored).

Couldn't this information be determined by querying the field table itself - e.g. SELECT DISTINCT(field_union) FROM xb_field WHERE component = %component (but a nice entity aggregate query).

But also:

i.e. I'm thinking $entity->get('component_tree')->get(2)->inputs->some_prop_that_is_modeled_by_a_link_field->uri might be possible.

It seems useful to have this available.

Now that #3468272: Store the ComponentTreeStructure field property one row per component instance is in, it would be helpful if the before and after examples in this issue summary were updated to be based on the new schema that landed in that issue.

AFAICT, the "after" in this issue will result in the inputs column being reduced to containing only the value key for each input, is that correct? For example, for an instance of a heading component with 3 props, the inputs column would look like:

{
    "text": {
        "value": "Example heading",
    },
    "style": {
        "value": "primary",
    },
    "element": {
        "value": "h1",
    }
}

If I'm understanding that correctly, then would it be reasonable for the scope of this issue to also include removing the value key since it would become superfluous? In other words, could we simplify the above to:

{
    "text": "Example heading",
    "style": "primary",
    "element": "h1"
}

Comment #32 is based on the assumption that we're versioning the union of the inputs, not each input individually, and therefore the "version" (or "field union ish" config entity reference) would be pulled out into its own column, but please correct me if that's an incorrect assumption.

#32 shows an example that only includes scalar values for each input, but it would work for objects/arrays as well. For example, if you have an input that's an array of links, you could still remove the superfluous value key. I.e., instead of:

{
  "links": {
    "value": [
      {
        "uri": "https://example.com/page1",
        "options": []
      },
      {
        "uri": "https://example.com/page2",
        "options": []
      }
    ]
  }
}

you could have:

{
  "links": [
    {
      "uri": "https://example.com/page1",
      "options": []
    },
    {
      "uri": "https://example.com/page2",
      "options": []
    }
  ]
}

So do we think there is merit in doing a spike on #15 / #35 ?

Yes, please! 😊🙏

The more concise structures in #32/#34 look great, it's not a massive reduction in storage requirements but it's definitely worth having, and also more readable for me. Also 100% agreed with #35.

Did an early peek at @larowlan's in-progress branch — observations:

1. That proxy functionality+architecture is super interesting — looking forward to reviewing that in detail when there's an MR up :)
2. experience_builder.composite_field.*:fields.<key>.field_type won't itself allow referencing a composite field (type), right? Related: #3467890: [later phase] Support `{type: object, …}` prop shapes with single level that require *multiple* field types: use `field_union`? — OUT OF SCOPE: nested components/component reuse.
3. I have doubts about type: experience_builder.composite_field:.*.third_party.experience_builder. Why not keep that information on type: experience_builder.generated_field_explicit_input_ux? 🤔 The new CompositeField config entity type must by definition not have a reference back to the Component config entity that will use it, which means I don't see any way for being able to validate (aka ensure data integrity). Whereas it'd be trivial to validate if these key-value pairs remained on type: experience_builder.generated_field_explicit_input_ux!
4. CompositeFieldType::isInUse() will only work for content entity types, so it won't work for e.g. an XB ContentTemplate config entity — won't this be problematic if this component (and its associated FieldComposite config entity) happen to only be instantiated in config entities? I see that ::preSave() explicitly relies on ::inUse(), so that seems like a date integrity problem? Or … I bet you're going to address this using a pre_save hook implementation that makes it also respect XB's (atypical!) use of fields in config entities? 🤓

Reviewed this 34 files, +2899, -166 diff at a high level.

1. My go-to test for making changes to the Component config entity is unsurprisingly ComponentValidationTest. But unfortunately that currently fails hard 😅
2. ComponentTreeItemListTest nicely illustrates the simplification for the stored data 👍
3. I have concerns about ::applyComponentTreeItemDefaults(), and I'm especially concerned about how it's used by ClientServerConversionTrait::clientModelToInput(). Left a lengthy comment there.
4. 👏 ComponentInputsEvolutionTest is fantastic! But … see the aforementioned comment for why that actually doesn't cover the full gamut of possible input evolutions.

AFAICT there are more remaining tasks:

1. multiple-cardinality support
2. explain the rationale behind the (auto-generated) naming scheme of the new config entities
3. provide an overview of which files are >=90% copy-pasted-and-renamed from field_union 🙏 I bet that >2K of the new lines are coming from field_union?

good point to stop and decide if we should keep going with this approach

I still am optimistic! So, I'd say: start with my main concerns (explained in detail on the MR), because if you can overcome/explain away those, then I think we're on track here 🤓

Kinda relieved to read my concerns about third-party settings are (seemingly) confirmed by you.

The static prop source defaults work can be combined with Wim's versioned config entity and we can add a version column to the field item that will apply to all components not just specific sources. That will give us the consolidation of the inputs column we're chasing here

I read that as "versions for all Component config entities, not just some, and stored as a new component_version field property for content entities and a same-name key-value pair for config-defined component instances" — which is exactly where I thought this was going last week 😄👍

Can't wait to see what you cook up for me by your EOD/my start! 🤓

To enable @larowlan to focus on the actual tricky bits, and to avoid him having to spend time fixing the remaining test failures I left in #23 (8 days ago), I got that out of the way. I also addressed the @todo for adding validation.

All PHPUnit tests are passing, except for one. That one actually reveals a severe flaw in what I did 😇😬

The last failure:

Drupal\Tests\experience_builder\Kernel\Config\JavascriptComponentStorageTest::testComponentEntityCreation
Drupal\Core\Config\Schema\SchemaIncompleteException: Schema errors for experience_builder.component.js.bxnge13p with the following errors: 0 [versioned_properties.46d5d8c16712f873.settings.prop_field_definitions] 'noodles' is a required key.

… this is because the way Component config entities are currently validated per the current config schema, it is required for all past versions of a Component to have valid source-specific settings.

However … in the case of code components, it's possible that new props/slots are added! 😅 (More to come on that front in #3509115: [META] Plan to allow editing props and slots for in-use code components.) Which then fails due to

…
    prop_field_definitions:
      constraints:
        # There must be a key for every `prop` in the corresponding \Drupal\experience_builder\Entity\JavaScriptComponent::$props.
        SequenceKeysMustMatch:
          configPrefix: experience_builder.js_component.
          configName: '%parent.%parent.%parent.%parent.source_local_id'
          propertyPathToSequence: props

And that is the big flaw: that actually only the active version MUST be valid, older versions MAY be valid. It's impossible to perfectly validate them, given the old implementation is no longer around. Will try to push a solution for that, too.

Will try to push a solution for that, too.

Done:

(I'm aware of the clean-up potential. I'll let @larowlan refactor from this — AFAICT — working state to his liking 😊)

1) Drupal\Tests\experience_builder\Kernel\Plugin\Field\FieldType\ComponentTreeItemTest::testInvalidVersion
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 [
-    '0.version' => ''lol' is not a version that exists on component config entity 'sdc.xb_test_sdc.props-slots'. Available versions: 'c8a016671696090c'.',
-    '1.version' => ''hah' is not a version that exists on component config entity 'sdc.xb_test_sdc.props-no-slots'. Available versions: 'c8a016671696090c'.',
+    '0.version' => ''lol' is not a version that exists on component config entity 'sdc.xb_test_sdc.props-slots'. Available versions: 'c8a016671696090c', 'c8a016671696090c'.',
+    '1.version' => ''hah' is not a version that exists on component config entity 'sdc.xb_test_sdc.props-no-slots'. Available versions: 'c8a016671696090c', 'c8a016671696090c'.',
 ]

🤣

@larowlan I don't understand why this commit brought back the ComponentIdParts constraint, the ::setSource() method etc. — could you explain? 🙏

Reduced PHPUnit CI job from 5 failures, 8 errors to 2 failures, 1 error.

But … that's misleadingly low: FallbackInputTest passes, but triggers some worrying PHP warnings, that I believe is related to #52. Which also leads to an informed guess about the answer to the Q I asked in #52: because of how A) VersionedConfigEntityBase::createVersion() always uses active as the identifier for the active version, B) type: experience_builder.component.versioned.active. The combination of those two means that only the active version has its source-specific settings validated (for IMHO good reasons, to solve #49), but that then of course fails when the component source is changed to fallback! 😅

That means the comment I added is wrong:

# Special case: the `active` version: it uses a non-Fallback source plugin.
# All other cases: key-value pairs under `versioned_properties` MUST contain both:
# 1. the Component Source-specific settings (which MUST be *valid* against the current implementation of the component)
# 2. the fallback metadata
# (because both may vary from one version to the next)

… because as of that commit, the active version might point to the Fallback source 🙈 IOW, that commit made this dead config schema:

# Special case: the `fallback` version: it uses the Fallback source plugin.
# @see \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\Fallback
experience_builder.component.versioned.fallback:
  constraints:
    FullyValidatable: ~
  type: mapping
  mapping:
    settings:
      type: experience_builder.component_source_settings.fallback
    # Note: NO `fallback_metadata`!

So my solution for #49 was incorrect or at best incomplete. I'm hopeful I got it right this time 🤞

D'oh, I missed @larowlan's comment that would've explained #52 🙈

Doing a bit more work on this before passing the baton to @larowlan.

Linking related core issues. PHPUnit tests are green.

Back to @larowlan for:

• getting end-to-end tests to green, too
• reviewing my changes
• addressing questions/thoughts I posted — including a suspected race condition, a suspected bug in the collapsing logic, whether the hashing should be actually made deterministic in a beta-blocking follow-up, and a bunch of smaller things

… so we can hopefully land this MR soonish 🤞

'fallback' is a version, not a plugin is genius! 👏

The one significant change in the validation logic I really didn't understand led me down a pretty deep rabbit hole, but I managed to crawl out 🐰

At the very end, I also scrutinized ComponentInputsEvolutionTest, because that's what proves:

• multiple versions of Component config entities can exist
• instances of those multiple versions can exist and continue to work
• the appropriate versions are created

Clarifying that test coverage happened in the last batch of commits. (And the expanded test coverage revealed a few small bugs 👍)

Follow-ups:

• Extracted critical beta-blocking follow-up issue for deterministic hashing: #3528159: Ensure deterministic version hashes for ComponentSource-specific settings, thanks to config schema-powered normalization. This MR is already 79 files, +2578, -716 — that follow-up won't change the data storage, but will ensure integrity. (This is also what I tagged this issue Needs followup for in #23.)
• Expand ComponentInputsEvolutionTest with "the other side of the coin", already happening in #3501708: Prove that it *will* be possible to apply block settings update paths (assuming #3521221 in core) to stored XB component trees in config/content.
• I contemplated renaming the field property from version to component_version, which would make it match component_id nicely. But perhaps the better way forward is to rename component_id to just component. Extracted a bikeshed issue: #3528167: Rename `version` field property to `component_version`.
• (Low priority) Take advantage of this new infra to reduce data-on-the-wire: #3528043: Reduce the verbosity of the client-side model for static prop sources
• Hopefully bring "versioned config entities" to core: #3528134: Consider adding a VersionedConfigEntityBase.

I thought the issue summary actually was fairly up-to-date, so I removed that tag. But the issue title does not nearly adequately capture the scope. Trying to improve that prior to merging 🤓

Thanks for that follow-up MR — very nice to see ComponentTreeStructureConstraintValidator become simpler 👍

I opened #3528362: Deterministic Component version hash should depend not only on source-specific settings, but also slots + explicit input schema yesterday, that was an oversight here (and kinda in #3528159: Ensure deterministic version hashes for ComponentSource-specific settings, thanks to config schema-powered normalization).

@larowlan identified another oversight: #3528528: `ComponentInputs::getPropSources()` needs to take `GeneratedFieldExplicitInputUxComponentSourceBase::rawInputValueToPropSourceArray()` into account.

I think I spotted one more gotcha: the \Drupal\experience_builder\Audit\ComponentTreeDependencyRepository logic + DB table looks at the dependencies of the Component config entity and stores those calculated dependencies in the xb_component_tree_dependencies table.
As of this issue, it should be calculating and storing the dependencies of the referenced version of the Component config entity used in the component instance — which could be an older version! Fortunately, @larowlan got that right here:
\Drupal\experience_builder\Plugin\Field\FieldType\ComponentTreeItem::calculateFieldItemValueDependencies() does:

public function calculateFieldItemValueDependencies(bool $with_plugin_dependencies, ?FieldableEntityInterface $host_entity = NULL): array {
…
    $component = $this->getComponent();
  

(unchanged here!)

which combined with

  public function getComponent(): ?ComponentInterface {
    return $this->get('component')->getTarget()?->getValue();
  }

and

    $properties['component'] = DataReferenceDefinition::create(ConfigEntityVersionAdapter::PLUGIN_ID)

means that the correct version is indeed used 👍

The oversight: we didn't expand test coverage to explicitly test this.

The oversight: we didn't expand test coverage to explicitly test this.

Being fixed at #3528499-20: Revisit storage of dependencies in separate table now we have separate deltas per component + remove plugin dependencies + make component version usages auditable.

Missing test coverage landed: #3528499-22: Revisit storage of dependencies in separate table now we have separate deltas per component + remove plugin dependencies + make component version usages auditable.