XB field type: calculate all dependencies, store them, surface in new Component "Audit" operation

@catch

1. Would you agree that this could be a fallback solution if #3452756: Ensure querying JSON nested values when parent keys are unknown is possible in all supported databases ends up not working, i.e. an escape hatch?
2. Would you agree that #3452756: Ensure querying JSON nested values when parent keys are unknown is possible in all supported databases is preferable?

Good point. 🤔

Your thinking is: it's gonna be easier to drop these denormalizations in the future, if we end up not needing them. Is that fair?

@catch, I'm experiencing some permission issues with git access. However, based on my findings, I suggest the following approach:

In the widget, we can define the class as:

class ExpressionFieldWidget extends WidgetBase {

  /**
   * {@inheritdoc}
   */
  public function formElement(FieldItemListInterface $items, $delta, array $element, array &$form, FormStateInterface $form_state) {
    $element['value'] = [
      '#type' => 'textfield',
      '#title' => $this->t('Expression'),
      '#default_value' => isset($items[$delta]->value) ? $items[$delta]->value : NULL,
    ];

    return $element;
  }
}

Then, we can define the field storage:

function experience_builder_install() {
  // Define the field storage for the expression field.
  \Drupal::service('entity_type.manager')->getStorage('field_storage_config')->create([
    'field_name' => 'field_expressions',
    'entity_type' => 'node',
    'type' => 'expression_field',
    'cardinality' => FieldStorageDefinitionInterface::CARDINALITY_UNLIMITED,
  ])->save();
}

For querying the new field, we can use:

$query = \Drupal::database()->select('node__field_expressions', 'e')
  ->fields('e', ['entity_id'])
  ->condition('e.value', '%' . $field_expression . '%', 'LIKE');
$result = $query->execute();

By transitioning to a standard Field API field that accepts multiple values, we can avoid database-specific JSON manipulation issues and ensure the code is compatible across different database systems.

Please let me know your thoughts. Once I get the access issue resolved, I will apply these changes if everything looks good to you.

Re: #8, see #3452756-18: Ensure querying JSON nested values when parent keys are unknown is possible in all supported databases for some notes on the JSON querying side of this.

So, @catch, are you essentially proposing this?

   /**
    * {@inheritdoc}
    */
   public static function propertyDefinitions(FieldStorageDefinitionInterface $field_definition) {
     $properties['tree'] = DataDefinition::create('component_tree_structure')
       ->setLabel(new TranslatableMarkup('A component tree without props values.'))
       ->setRequired(TRUE);
 
     $properties['props'] = DataDefinition::create('component_props_values')
       ->setLabel(new TranslatableMarkup('Prop values for each component in the component tree'))
       ->setRequired(TRUE);
 
     $properties['hydrated'] = DataDefinition::create('component_tree_hydrated')
       ->setLabel(new TranslatableMarkup('The hydrated component tree: structure + props values combined — provides render tree for client side or render array for server side.'))
       ->setComputed(TRUE)
       ->setInternal(FALSE)
       ->setReadOnly(TRUE)
       ->setRequired(TRUE);
 
+    $properties['dependencies'] = DataDefinition::create('component_tree_props_dependencies')
+      ->setLabel(new TranslatableMarkup('All config and modules that the stored values of this XB field depend upon..'))
+      ->setComputed(TRUE)
+      ->setInternal(TRUE)
+      ->setReadOnly(TRUE);
+
     return $properties;
   }

(in \Drupal\experience_builder\Plugin\Field\FieldType\ComponentTreeItem::propertyDefinitions())

My initial reaction is whether the denormalization proposed by @catch would get out of sync and lead to all the problems we've had over the years with file_usage. But presumably we could add code somewhere that can rebuild it fairly easily, which I think would alleviate that?

Let's do this as a stepping stone towards #3477428: [PP-1] Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree, and to reference the field_union type of the prop values.

I'm still collecting my thoughts, but there's at least 3 use cases already identified for adding more field properties (i.e. expanding ComponentTreeItem::::propertyDefinitions()) and hence DB columns:

1. the dependencies implied by a component tree and its inputs
   👉 (this issue)
2. multi-valued XB field with a unique exposed_slot property per delta, to target a specific exposed slot in a ContentTemplate config entity, i.e. for the per-content entity unique creative freedom while the overall component tree is guaranteed to be consistent by the ContentTemplate
   👉 being worked on at #3519352: Content templates, part 3b: store exposed slot subtrees on individual entities, as part of #3455629: [PP-1] [META] 7. Content Templates — aka "default layouts" — affects the tree+props data model
3. (surfaced yesterday by @lauriii at DDD and discussed at length with @pdureau and others at DDD) the fact that some SDC props may need to be considered either translatable or not (see #3454257: Allow Experience Builder fields to support Asymmetric and Symmetric translations) — for example an image in some SDC might be considered translatable or not — details TBD
   👉 issue to still follow

(Not linking any of those issues nor crediting those people, because that's what that follow-up will be for. I'm just writing it down as context, and as a reminder for myself when I do write that next week.)

I will implement basically what I wrote in #15, but @catch was concerned in #16 that would still require a flat JSON structure.
AFAICT he thought that because it'd store both config and module dependencies:

  config:
    - a
    - b
  module:
   - x

But that's easily transformed to:

config:a config:b module:x

… at which point it's a flat list that is trivially queryable :)

Given what @catch wrote in #18, I think that my thinking has arrived at the same point: we basically want for each row for an XB field revision a flat list of dependencies, that is functionally equivalent to what a config entity's dependencies stores. IOW: we want the equivalent of \Drupal\Core\Config\Entity\ConfigEntityInterface::calculateDependencies(), but for the XB field, to allow efficiently (and with 100% confidence — unlike file_usage) determining which XB fields (and even which revisions of XB fields) are impacted by a potential change in available modules/themes/config.

IOW:

so I think this issue would be a small step forwards if those are not being prioritised yet.

💯

P.S.: I'm no DB nor SQL expert, so perhaps it seems super naïve. That's fine! I just want to restart this conversation, @catch! 😄

Working PoC live, with plenty of @todos.

Thoughts, @catch? 😊

I just pushed a sequence of commits:

1. 100% of prop expressions now implement calculateDependencies()
2. Then I started using that to implement DynamicPropSource::calculateDependencies() with essentially zero effort, with test coverage to prove it works correctly.
3. … at which point it was trivial to impose ::calculateDependencies() for all 4 of XB's "prop sources", by adding it to abstract class PropSourceBase, implementing it took only a few dozen LoC
4. The above enabled me to drastically simplify ComponentInputs::calculateDependencies(): it now just forwards the responsibility of calculating dependencies to each prop source, and the (Static|Dynamic)PropSources largely forward that responsibility again to the low-level "prop expressions

The result (assuming the test coverage is expanded): simple code, and the responsibility for ensuring correct dependencies is pushed down to the individual code that is actually interacting with those dependencies!

Next steps:

1. review by @larowlan
2. update Pattern::calculateDependencies() etc to leverage this new infrastructure
3. update FieldTypeUninstallValidator(Test) to leverage this new infrastructure
4. add missing low-level test coverage
5. …

P.S.: IMHO core's config dependency infrastructure is missing a type in addition to config, content, module and theme: plugin. A module still being present is a very weak guarantee for a plugin being present. I think XB should lead the way here, but I'm curious what @larowlan and others think.

I think the next steps here are to perform some EXPLAIN queries against this data in JSON and string format and see whether we can query it efficiently. If there isn't an efficient way to query the multi-value nature of the field then I think a separate table makes more sense. SqlContentEntityStorage wraps hook_entity_insert and hook_entity_update in a transaction so we can ensure writes to that table are atomic. We can keep the logic @wim leers has written in the current MR but mark the fields as computed and use those hooks to update the dependencies table. We can make use of the new OO hooks to put the update/insert hook logic in a service and write tests for it. The service can make use of things like just-in-time table creation like we have for other services in core (i.e. no need to have a hook_schema)

This blocks #3518336 — see #3518336-14: When a field instance used by a `ContenTemplate` is deleted, or a field-type providing module is uninstalled, replace affected `inputs` with default `StaticPropSource`.

@catch in #23:

And this does not use an index.

🤯 WTAF?! That is absurd!

Thanks for super explicitly confirming my P.S. from #19:

P.S.: I'm no DB nor SQL expert, so perhaps it seems super naïve. That's fine! I just want to restart this conversation, @catch! 😄

… which you illustrated very well now 😅

Which brings me to @larowlan's #26:

I think the next steps here are to perform some EXPLAIN queries against this data in JSON and string format and see whether we can query it efficiently.

I think @catch is saying the answer to that is "no". 😭

I think a separate table makes more sense.

… really? That's making things harder again. I'd really like to the data that belongs together, stored together. That'll make debugging much simpler.

@catch, any ideas/suggestions? 🙏 In the original issue summary you wrote: store them in a more predictable structure (like a flat JSON array or just an unlimited cardinality field API field) outside the tree structure — so is a flat JSON array what we want to do instead?

Overhauled issue summary to outline the full plan in a fair amount of detail.

Solved a long-standing @todo in multiple config entity types by executing part of the plan! :D

This now also explicitly surfaces what @larowlan predicted in his reviews: ContentTemplateValidationTest::testEntityIsValid() now fails with MissingHostEntityException.

And … now ContentTemplateValidationTest::testEntityIsValid() is passing thanks to fixing parts of @larowlan's concerns by updating \Drupal\experience_builder\PropExpressions\StructuredData\FieldPropExpression::calculateDependencies() to do a superset of what @phenaproxima was doing in #3518336: When a field instance used by a `ContenTemplate` is deleted, or a field-type providing module is uninstalled, replace affected `inputs` with default `StaticPropSource`'s https://git.drupalcode.org/project/experience_builder/-/merge_requests/908.

This would also unblock #3500074: Usage info for code components with unpublished changes!

Thanks, that's very helpful! I'm just baffled that this is the reality in 2025 — where to get the simple thing you first kinda need to make it more complicated 😅

So, to make absolutely sure we're on the same page, you think that/propose:

• this issue should lead by example and NOT store additional props, rather than waiting for #3477428: [PP-1] Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree, and to reference the field_union type of the prop values to refactor All The Thing
• IOW: do what @larowlan described in #26 and introduce a new xb_dependencies table with
  

  entity_type_id,bundle,entity_id,revision_id,delta,dependency_type,dependency_name
  node,article,42,456,0,config,experience_builder.component.block.system_branding_block
  node,article,42,456,0,config,experience_builder.component.sdc.xb_test_sdc.props-slots
  …
  

#37:

#1

I can't answer that, because I don't know why this issue is being worked on before

Because I view this is a step that unblocks other things, whereas #3477428 is a HUGE change.

So, I thought that this could be a nice step towards #3477428 — quoting yourself from #18: Note that if #3477428 happens, then the component would already be stored as a string in a multi-valued field. […] Then this issue would have been done implicitly,.

Is this issue blocking that one? Is that issue blocked on other ones? Is this issue blocking other urgent issues? There is really not much visibility for me into why various issues get worked on or not at different times with Experience Builder.

I can promise you it's super hectic inside our team too. 😬 It's constantly changing based on shifting priorities not decided by me; if it were up to me we'd be finishing one layer before building the next one on top of it. Alas.

I'm working on a plan/meta issue for stabilizing the data model: #3520449: [META] Production-ready data storage. I see I didn't make that the parent of this issue yet — fixed. That meta issue culminates in #3477428: [PP-1] Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree, and to reference the field_union type of the prop values. Why not start with it? Because the current data storage doesn't yet do all the things we need to do, and nor does #3477428. It's safer to first evolve what we have to do all the things we need to do rather than starting with the enormous change that is #3477428.

Both of these issues have been open for some time now.

Because the goal until recently has been "add features, show potential", not "stabilize". We just started stabilizing, towards #3515932: Milestone 1.0.0-beta1: Enable creation of non-throwaway sites.

I'm also not entirely sure how this issue ties into configuration for entity templates/displays (as opposed to xb field data)

Because XB's key data structure, "the component tree" is shared between content and config. They're stored differently (2 JSON blobs in a config YAML or 2 JSON blobs in a DB row), but they're internally, in memory, while executing logic, exactly the same. Search for the string ComponentTreeMeetRequirements and you'll find the same validation constraint used in both content and config! Look at the validation logic for that constraint, and you'll see that both end up working with a ComponentTreeItem object — yes, a field item instance. While certainly unusual, this ensures that we don't need to write equivalent (validation) logic twice.

That's why the code changes in this issue seem to be about the XB field type solely, but they also impact config entities: because e.g. Pattern::getComponentTree() also just returns … a ComponentTree, conjured/instantiated from the two JSON blobs stored in config, and then we can simply use the same logic to determine the full set of dependencies. Because regardless of how component trees are stored, they ALL end up having dependencies on config, modules, etc.

#2

Can I summarize that as: flat JSON structure more so for ease of access, not for querying itself?

#3

I think this means I can summarize it as "let's try and find out".

f we researched it now, it would give us information about how queryable the flat JSON structure is for prop data too, which would be useful information. So I don't think effort working on it would be wasted necessarily

GREAT! 😄 That's exactly what I want us to do here, then. 👍

Per #38.3.

but also if e.g. dependencies get stored in the same row as the flat json eventually, then it will result in easier querying of 'stuff in the row'

My thoughts exactly! :D

Something I didn't spot in the MR yet was an uninstall validator or similar to use the new component dependency information

That is listed in the proposed solution:

4. update FieldTypeUninstallValidator(Test) to leverage this new infrastructure

Same for themes, same for deletion of config-defined components. Is that all in scope here?

I think not; proving it for one thing is sufficient for this issue IMHO, because then at least the quite complex code that currently exists as a PoC should become simpler. That's enough proof for this MR I think.

There will need to be similar logic for config entities too, validators or onDependencyRemoval() etc.

Yep. And we have some issues for ::onDependencyRemoval() already:

1. #3484682: Handle update and delete of Block `Component`s, plus react to dependency removal
2. #3519168: Handle components provided by ComponentSources EXPLICITLY disappearing — enables deleting JS components that are in use

In #19 I wrote about which additional data we know we'll need to store in XB fields, and that I'd create a follow-up issue to capture all that.

I did that, in this new meta issue: #3520449: [META] Production-ready data storage. Please read the overview, and the proposed resolution, especially the 1. Changing/adding field props on the XB field type part.

#43 makes me think a separate table and computed properties is the easier path

Although a query on a JSON array can use an index, unlike LIKE %foo%, you still need to create the index, and core does not support this yet.

Similarly, core does not support a condition syntax for querying JSON.

Argh! 😬

To keep the multi-DB-table-joined-against-fields complexity out of this issue/MR (unless @catch or @larowlan have a nice reference/sample to look at? 🤞), let's:

1. keep this issue scoped to storing space-separated strings (with slow performance because no indexes) as additional columns
2. a subsequent issue to convert it to a joined-table approach (with good performance because indexes)

The intent for me in this issue was to really focus on getting the necessary information stored and be made queryable. Performance is crucial, but we don't need to do everything in a single MR.

P.S.: @catch, I'm so grateful to have you in this issue!!! ❤️🙏🙏

https://dbfiddle.uk/L3hiiLn2 is promising

Walked @penyaskito through this issue, next steps for this specific MR, and the big picture at #3520449: [META] Production-ready data storage.
In having him start on

3. update FieldTypeUninstallValidator(Test) to leverage this new infrastructure

we basically immediately got stuck, because the necessary information is missing. So AFAICT there's no way around what I wrote in #25:

P.S.: IMHO core's config dependency infrastructure is missing a type in addition to config, content, module and theme: plugin. A module still being present is a very weak guarantee for a plugin being present. I think XB should lead the way here, but I'm curious what @larowlan and others think.

What I wrote was too rough/inaccurate though! 😅 The Component config entity has a PluginExists constraint for the field types XB's "prop shape matching" assigned to it:

experience_builder.generated_field_explicit_input_ux:
<snip>
          field_type:
            type: string
            label: 'Default field type'
            constraints:
              PluginExists:
                manager: plugin.manager.field.field_type
                interface: '\Drupal\Core\Field\FieldItemInterface'

So that already prevents uninstallation of a field type. But … that chosen field type can change in the Component config entity, and then existing revisions will not have been updated (we can't go in and update millions of rows). So for component trees in content entities, we need the plugin information to be explicitly queryable. See detailed comment.

Otherwise we won't be able to remove

      // @todo The "$.*.*.expression" wildcard key matching works in Mariadb but not Sqlite.
      //   We need figure out a way to make this work in Sqlite and PGSQL in https://drupal.org/i/3452756.
      $select->where("JSON_EXTRACT($column_name, '$.*.*.expression') LIKE '%$field_expression%'");

which is the goal here.

Note that even long-term plugin dependencies likely make sense, because they allow for better error messages that are more actionable for end users. It's likely to help with Display actionable resolution steps for each error when e.g. a block plugin or an SDC disappears from a module (that is quoted from #3517941: [META] Robust component instance error handling during hydration+rendering).

Alright, extracting the "store in an efficiently queryable manner" part into #3521202: Store XB field type's "deps_*" columns in separate table to allow efficient querying.

#50: it's something that can currently happen if you implement hook_storage_prop_shape_alter() or if XB decides to change its default decisions in \Drupal\experience_builder\JsonSchemaInterpreter\SdcPropJsonSchemaType::computeStorablePropShape().

Uninstallation of the module providing the field type that is:

• currently referenced by a Component config entity is already validated, thanks to config dependencies.
• currently in use in historically created component instances using a field type NOT matching the current Component config entity is already validated, thanks to \Drupal\experience_builder\FieldTypeUninstallValidator

when you restore one of these revisions because presumably it would mis-match?

They don't need to match. The component tree is entirely self-sufficient. It contains all the metadata for reconstructing a field item object out of thin air.

For example, imagine a event SDC, which has a duration prop (shape: an object with two key-value pairs, named start and stop, both type: string, format: date). Then for that component instance, the following StaticPropSource is stored in the component tree:

{"sourceType":"static:field_item:daterange","value":{"value":"2020-04-16T00:00","end_value":"2024-07-10T10:24"},"expression":"ℹ︎daterange␟{start↠value,stop↠end_value}"}

Doing PropSource::parse(json_decode(<json blob>))->evaluate() yields

   [
      'start' => '2020-04-16T00:00',
      'stop' => '2024-07-10T10:24',
    ]

aka the value to pass to the SDC.
(FYI: copy/pasted that literally from \Drupal\Tests\experience_builder\Kernel\PropSourceTest::testStaticPropSource() — check it out, step through it, and it'll likely make more sense then.)

From there, XB can reconstruct the field type, field storage settings, instance settings, (default) widget, etc.

IOW: restoring old revisions would indeed bring back whatever field type was used at the time. That could mean that in theory you're restoring a revision using a field type that doesn't exist anymore. So we'll either need to prevent such a revision from being restored, or we'll let #3517941: [META] Robust component instance error handling during hydration+rendering handle that one component instance gracefully failing to render.

Fix adding deps_plugin to IS.

Actively reviewing this — @penyaskito did the bulk of the work here, I mostly am

• updating the MR after #3467870: Support `{type: array, …}` prop shapes landed yesterday , which introduced the need for many bits in this MR
• scrutinizing and tightening up as I go :)

👏👏👏 Fantastic work here, @penyaskito! 🤩

Technical debt resolved

The improvements to FieldTypeUninstallValidator on top of the new infrastructure that this issue is introducing allow us to close:

• #3452756: Ensure querying JSON nested values when parent keys are unknown is possible in all supported databases
• #3478866: FieldTypeUninstallValidator's error message needs to include the entity type and bundle of the default value field
• #3471028: Expand test coverage in FieldUninstallValidatorTest

Unblocked

Unblocked next step for #3520449: [META] Production-ready data storage: #3521202: Store XB field type's "deps_*" columns in separate table to allow efficient querying 👈 this is what @catch and I discussed in #45 and #46.

Unblocked issue not directly tied to #3520449:

• #3500074: Usage info for code components with unpublished changes
• #3518336: When a field instance used by a `ContenTemplate` is deleted, or a field-type providing module is uninstalled, replace affected `inputs` with default `StaticPropSource` (which now has far less to do)
• #3519168: Handle components provided by ComponentSources EXPLICITLY disappearing — enables deleting JS components that are in use

Follow-ups

Out-of-scope follow-ups:

• #3522198: [PP-2] Support translations tracking on Component "Audit" operation content listing when using asymmetric translations
• #3522199: Consider component trees in config translations when uninstalling the module providing a field type
• to be tackled way later: #3522196: [later phase] [PP-2] Add paging to Component "Audit" operation content listing
• to be tackled way later: #3522953: [later phase] Make Component `audit` operation performant/scalable.

Oops!

I forgot to actually run the test suite against the other DB back-ends, and turns out they all fail:

•        ├ Drupal\Core\Entity\EntityStorageException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'field_xb_demo_deps_config' at row 1: INSERT INTO "test66013041node__field_xb_demo" ("entity_id", "revision_id", "bundle", "delta", "langcode", "field_xb_demo_tree", "field_xb_demo_inputs", "field_xb_demo_deps_config", "field_xb_demo_deps_content", "field_xb_demo_deps_module", "field_xb_demo_deps_theme", "field_xb_demo_deps_plugin") VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11); Array    
  …                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
  

  — MySQL CI job

• rupal\Core\Entity\EntityStorageException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'field_xb_demo_deps_config' at row 1: INSERT INTO "test38671787node__field_xb_demo" ("entity_id", "revision_id", "bundle", "delta", "langcode", "field_xb_demo_tree", "field_xb_demo_inputs", "field_xb_demo_deps_config", "field_xb_demo_deps_content", "field_xb_demo_deps_module", "field_xb_demo_deps_theme", "field_xb_demo_deps_plugin") VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11); Array                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  …
  

  — PosgreSQL CI job

MySQL is particularly picky apparently, but … still, trivial fix :)