[PP-1] Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree, and to reference the field_union type of the prop values

Thanks for writing this up! I'm still digesting the proposed schema. Two minor things:

so this proposal creates the possibility of a multi-valued field item with more items in it than we're typically used to in Drupal

So I haven't actually used paragraphs, but I assume there's a single paragraph reference field that can have dozens if not hundreds of deltas in it referencing the paragraph entities. If so then we already have an equivalent example in the wild (except no extra entities for every row here). Also having written the last couple of sentences, I wonder if this starts to make an actual data migration from paragraphs more feasible.

I recommend doing that, and solving the querying use case by changing core's config table from serialized PHP to json.

We started discussing that in one of the JSON database support issues, it would allow us to remove the key value config stuff (which supports some limited querying now).

However, I think we could workaround not having that yet, just by running extra queries. e.g. if we want to find out whether a field type is used, we can get a list of field unions that use it, then a list of components that use those field unions, then run an IN(). Given the main current use-case for that sort of querying is auditing, it should be OK.

Hi, I just wanted to comment on this:

I hope people don't actually put hundreds of component instances on a node. That wouldn't lead to a good authoring experience. A good design system, even if it includes some small components (atoms), should also include larger components (molecules, organisms) that content authors work with, so that content authors aren't in practice putting every atom one-by-one on a page.

We use a lot of low-level elements on a page, so we have a lot of freedom. Yes, we also have some elements like molecules, but we usually do that with templates that we can then modify. And this templates are a group a single atoms.

And a landing, por example, can be very, very, very long.

So actually the hundreds of components that @Wim Leers was talking about can be real in a lot of cases.

It would let us take a bunch of stuff out of the component config entity and instead move that information to the field_union config entity.

In general that sounds like a great idea, it would mean the component config entity only needs to hold the things that are unique to the concept.

I had wondered whether we actually need two config entity types at all - i.e. could field union directly use a component config entity type instead of using its own, or could XB directly use field unions without an extra entity type in-between, but... no idea whether that would even be desirable even if it's possible.

@catch thinks it's needed to support #3462219: [META] Support alternative renderings of prop data added for the 'full' view mode such as for search indexing or newsletters, but @lauriii thinks those use cases could be solved in better ways by XB directly.

Are there use cases outside of the use cases that have been already identified that this would help with? So far I've not heard compelling reasons to do this. I'm pretty strongly -1 to supporting the workflow proposed in #3462219: [META] Support alternative renderings of prop data added for the 'full' view mode such as for search indexing or newsletters out of the box because at least as I understand it, it would result in a extremely convoluted UX. As a fairly technical user, I'm having hard time imagining working with several lists of components and figuring myself how to build anything meaningful out of it. I believe there should be an easier way for managing the challenges related to the search indexing.

Unless we can define what's the value we get out of this, I don't see why we would prioritize working on this over other work, especially because it sounds like that there's risk associated to introducing this. If I also understand correctly, this also means that there's additional complexity going forward because we support multiple data models out of the box (one for config, one for content).

I hope people don't actually put hundreds of component instances on a node.

I checked a sample front page I had built on another page builder and I had 129 components/elements on that page. This was still a fairly simple page using a mix atoms and organisms. I would have to do some more research to define what a reasonable upper bound would be, but it seems that the architecture should definitely be able to handle at least some hundreds of components.

Change the field type from single-item to multi-valued. Each item would be for a single component instance.

If we move from JSON structure to a multi-valued field (where each delta represents a component), how do we handle scenarios where there are overrides on top of the desktop breakpoint (e.g. for the mobile breakpoint)? This is requirement #20 from the original product requirements for Experience Builder.

Example scenario would be that I want larger margin and padding on desktop than on mobile and I want to display a block recommending to install an app on mobile.

How would this be represented in this data model? Would this still all be stored in the single list or would we have separate lists for different breakpoints?

RE: issue summary

Alternatively, we could omit this from here and instead add the field_union reference to the component config entity.

It should definitely be present in the Component config entity, because that'll allow a config dependency on the FieldUnion 👍

RE: @catch in #7

I assume there's a single paragraph reference field that can have dozens if not hundreds of deltas in it referencing the paragraph entities. If so then we already have an equivalent example in the wild (except no extra entities for every row here).

Very good point! 👏

🤔 Does anybody in this issue have access to a Paragraphs-heavy complex site with good performance, so that we can get some statistics? 🤓🤞

@effulgentsia in #8

[…] the component config entities have essentially all of the information that would need to be in the field_union config entity […]

Exactly!

XB currently basically implements the functionality that Field Union provides (I wrote that previously ~4 months ago at the top of #3440578-52: [PP-2] JSON-based data storage proposal for component-based page building), but without another config entity; the metadata that defines what field types to use (just like a FieldUnion config entity does) is captured by the Component config entity type. See how similar these are (the most notable difference being the absence/presence of config validation):

FieldUnion config entity

https://git.drupalcode.org/project/field_union/-/blob/acf17007546dcb76cd...

XB's Component config entity

https://git.drupalcode.org/project/experience_builder/-/blob/0.1.0-alpha...

(It's rather unlikely that two SDCs would point to the exact same FieldUnion, because even just naming the SDC's props differently would result in different FieldUnions.)

Given that, I wonder if making the field_union module a hard dependency wouldn't actually be that bad. It would let us take a bunch of stuff out of the component config entity and instead move that information to the field_union config entity.

• Theoretically: absolutely.
• Practically/devil's advocate: what would the benefit be?

@catch in #10

I had wondered whether we actually need two config entity types at all - i.e. could field union directly use a component config entity type instead of using its own, or could XB directly use field unions without an extra entity type in-between, but... no idea whether that would even be desirable even if it's possible.

HAH! 😅

Maybe … maybe the existing Component config entity with its (validated) config schema (see links above) actually sufficiently addresses all those needs already then? 😄 Back when we were actively going back-and-forth on #3440578 (~4 months ago), a lot of this was much vaguer, less fleshed out. Now it is in a more complete (but nowhere near done!) state. If you look at the <dl> above … does that already do what you're suggesting/thinking here? 😊 🤞

@lauriii in #11

If we move from JSON structure to a multi-valued field (where each delta represents a component), how do we handle scenarios where there are overrides on top of the desktop breakpoint (e.g. for the mobile breakpoint)? This is requirement #20 from the original product requirements for Experience Builder.

I imagine 41. Conditional display of components?

Hm, requirement #20's story says For example, I want to specifically change the padding for a specific component when the component is displayed in the mobile breakpoint., which requires storing additional values, not merely conditionally displaying (which is yes/no based on some condition, whereas the example in the story is more nuanced) … not sure. None of that is specced out nor estimated though. It sounds more like a "CSS-per-component instance" thing than a "prop value per component instance" thing though.

That's why I doubt the delta/multi-value change this issue proposes would affect this product requirement; it'd likely be a new style or css field property on the field type, to allow per-component-instance (responsive aka media query) styles.

In my prior comment I caught up on the issue and replied to things that stood out. @effulgentsia captured my first concern at the bottom of the issue summary ("what about hundreds of component instances"). @lauriii confirmed this must be supported. @catch suggested that Paragraphs likely already hits that scale. We still need numbers to gain confidence.

In this comment, I point out concerns that have not been raised before.

I'll use the same example in both concerns: suppose a PROVIDER:heading SDC that contain 2 props: a string (title) and a heading level (enum integer). Then a matching field union (first see the docs at https://git.drupalcode.org/project/field_union/blob/8.x-1.x/Readme.md) would be:

$union = FieldUnion::create([
  'id' => 'xb_component_PROVIDER_heading',
  'fields' => [
    'text' => [
      'field_type' => 'string',
      'label' => 'Heading title',
      'name' => 'heading_title',
      'required' => TRUE,
      'translatable' => FALSE,
      'default_value' => ['whatever the example value is in the *.component.yml file'],
      'settings' => [
        'max_length' => 255,
      ],
    ],
    'level' => [
      'field_type' => 'list_integer',
      'label' => 'Heading level',
      'name' => 'heading_level',
      'required' => TRUE,
      'translatable' => FALSE,
      'default_value' => ['whatever the example value is in the *.component.yml file'],
      'settings' => [
        'allowed_values' => [
          ['value' => 1, 'label' => '<h1>'],
          ['value' => 2, 'label' => '<h2>'],
          ['value' => 3, 'label' => '<h3>'],
          ['value' => 4, 'label' => '<h4>'],
          ['value' => 5, 'label' => '<h5>'],
          ['value' => 6, 'label' => '<h6>'],
        ],
      ],
    ],
  ],
]);

Concern 2: product requirement 7.1 Tokens aka Reusing values in the host entity's base/bundle fields

What if I want to populate one of the SDC props using a value from a host entity base/bundle field?

(This is called a dynamic prop source in current XB terminology because its value is dynamic: the value changes when the host entity's field values change. This is in contrast with a static prop source, where the value is manually/explicitly entered by the Content Creator, where the value that was entered is static: it will always evaluate to the same result. See XB terminology docs.)

For example, I want to populate a component instance that uses the heading SDC in part with the label of the single-cardinality "Category" taxonomy term reference of my host entity type+bundle "News item". (Or, simpler example: the "News item" entity's "Title" field .)

So, my heading component instance would be claiming to be using this xb_component_PROVIDER_heading FieldUnion, but … actually only the level prop would be populated by the field union, the text SDC prop would be populated by the "Category" taxonomy term reference!

This is what I was referring to in #3440578-30: [PP-2] JSON-based data storage proposal for component-based page building. That's what product requirement 7.1 Tokens refers to.

(The above interpretation AFAICT accurately/reasonably interprets the product requirement. @lauriii, please correct me if I'm wrong.)

Concern 3: How will this work for SDC props that themselves are type: object-shaped?

An SDC's props is always type: object. But what if some prop foo also is type: object-shaped?

This is not yet supported in XB yet (issue: #3467890: [later phase] Support `{type: object, …}` prop shapes with single level that require *multiple* field types: use `field_union`? — OUT OF SCOPE: nested components/component reuse), but I know/I'm confident it's possible.

This is a common need, and a number of SDCs in https://www.drupal.org/project/demo_design_system had to be refactored to not use that because #3467890 is not yet fixed! 😅

An example in the XB codebase itself is the shoe_details component, which contains:

…
props:
…
    expand_icon:
      title: Expand icon
      $ref: json-schema-definitions://experience_builder.module/shoe-icon
      # @todo slot prop on the icon should always be expand-icon
      type: object
    collapse_icon:
      title: Collapse icon
      $ref: json-schema-definitions://experience_builder.module/shoe-icon
      # @todo slot prop on the icon should always be collapse-icon
      type: object
…

I've found #3170831: Support nested union fields in the field_union issue queue. I have no idea yet how much work it'd be to support that. I bet @larowlan can speak to that 🤓

But this would make concern 2 above more complicated: what if it's the "title" of the expand_icon that you want to populate using a base/bundle field value? Then it'd be a token that needs to be resolved for one of a nested field union.

Maybe … maybe the existing Component config entity with its (validated) config schema (see links above) actually sufficiently addresses all those needs already then?

I think to answer this, we would need to figure out what addressing #3462219: [META] Support alternative renderings of prop data added for the 'full' view mode such as for search indexing or newsletters looks like with field_unions + components vs. just components or at least enough direction and agreement on the use-cases to be able to talk about it with a common understanding of the need, currently that does not seem to be the case.

The big difference with a field_union is that it results in field data that can be used outside XB (i.e. regular manage display), or which could be configured as the optional source for 'dynamic props' inside XB, for a different view mode. Just this morning I tried to type up some thoughts on how a component-only solution to that issue theoretically possibly could work to have something to compare to.

(This is called a dynamic prop source in current XB terminology because its value is dynamic: the value changes when the host entity's field values change. This is in contrast with a static prop source, where the value is manually/explicitly entered by the Content Creator, where the value that was entered is static: it will always evaluate to the same result.

But if we used field unions, then everything (or at least everything that looks like field data using field types) would be dynamic - the difference would instead be whether field deltas can be created/re-ordered/edited directly within the XB interface or not.

I just saw @lauriii's DrupalCon presentation, where he walked through designs that show the 7.1 Token functionality I mentioned in concern 2 in #14. He showed DrupalCon before everyone else 😄

That enables me to actually illustrate the problem:

👆 That shows the props form for an PROVIDER:event_card SDC. It has 9 props:

1. Variation aka #3390712: Add component variants to SDC
2. Primary title
3. Logo
4. Secondary title
5. Body
6. Address
7. Price
8. Image
9. Button

For each of the 9, I drew an arrow on the screenshot of the design:

• 3 big blue arrows: static information stored for this component instance, "in the field union" if you will.
• 6 small green arrows: for these, no information is stored, only an expression that describes how to retrieve this information from the host entity's bundle/base fields

How do you map that onto a Field Union? The Field Union metadata would still be relevant (allowing "regular manage display" as you say), but the field data itself would empty for 6/9=66% of the fields in the field union.

But if we used field unions, then everything (or at least everything that looks like field data using field types) would be dynamic - the difference would instead be whether field deltas can be created/re-ordered/edited directly within the XB interface or not.

I don't understand this paragraph in two ways:

• everything would be dynamic: I've been using "Dynamic" in the sense that what we store is a token/expression, and its value is dynamically retrieved.
• the difference would instead be whether field deltas can be created/re-ordered/edited directly within the XB interface or not. → no idea what this refers to 🙈

Could you rephrase that? 🙏

(Could be me I've had a terrible night with our RCBO/GFCI interrupting twice in the middle of the night 😬, so I'm not at 100% brain capacity.)

I don't understand this paragraph in two ways:

everything would be dynamic: I've been using "Dynamic" in the sense that what we store is a token/expression, and its value is dynamically retrieved.

Currently dynamic is 'referenced from entity fields' and static is 'stored directly in the xb field', with field union, everything becomes a field reference/dynamic.

the difference would instead be whether field deltas can be created/re-ordered/edited directly within the XB interface or not.

We discussed this during the Barcelona meeting - let's say you have five images + description components back by field unions, when re-ordering them, we might want to re-order the field union deltas too so that the XB order and the field order stays in sync. This is opposed to say a single standard description field which doesn't have a delta order.

How do you map that onto a Field Union? The Field Union metadata would still be relevant (allowing "regular manage display" as you say), but the field data itself would empty for 6/9=66% of the fields in the field union.

Even with the diagram I still don't understand what's going on here unfortunately. Personally it seems odd to me that you would place a single component then have to individually map what comes from there in it. Why is the content editor making all those 7-9 decisions about each component they add? For entity references I can see you would want selection/create etc., but not for little bits of text.

with field union, everything becomes a field reference/dynamic.

🤔 Aha! Maybe I have fundamentally misunderstood how Field Unions work. This is not mentioned in https://git.drupalcode.org/project/field_union/blob/8.x-1.x/Readme.md. I'll read the underlying code instead.

Personally it seems odd to me that you would place a single component then have to individually map what comes from there in it. Why is the content editor making all those 7-9 decisions about each component they add?

That is a very fair point! 👍

I wonder if this is solely because we don't have #3455629: [PP-1] [META] 7. Content Templates — aka "default layouts" — affects the tree+props data model built yet (where it'd indeed be a Site Builder decision), or whether @lauriii truly intends for Content Creators to (be able to) decide this.

That'd definitely change this conversation!

🤔 Aha! Maybe I have fundamentally misunderstood how Field Unions work. This is not mentioned in https://git.drupalcode.org/project/field_union/blob/8.x-1.x/Readme.md. I'll read the underlying code instead.

Sorry I may not be doing a good job explaining what I mean, it should not be necessary to look at the current field union module.

If we store XB-entered data in Field Union, then XB will be writing that data to field union field values. This means that the field union data is field data, same as any other field (except the extra stuff it adds on top).

XB's static vs. dynamic distinction is for field API vs. non field API data.

If all (or all*) entity-content data entered via XB is field API data, then I would assume that XB would switch to treating the field union data as field data and referencing it the same way that it does other field types. Albeit a more complex field type but one nonetheless.

If that doesn't clarify things, we should grab each other on slack, figure out the disconnect, then report back here.

I wonder if this is solely because we don't have #3455629: [later phase] [META] 7. Content type templates — aka "default layouts" — affects the tree+props data model built yet (where it'd indeed be a Site Builder decision), or whether @lauriii truly intends for Content Creators to (be able to) decide this.

OK let's please try to clarify that asap. If content creators cannot do this manual mapping, (I agree it's something site builders might do with bundle-level fields similar to layout builder view mode config now), then each component added within XB by site editors will be 'coherent' in that its data will come from the same place and no need for 'partial field unions' which I agree would be weird.

OK let's please try to clarify that asap. If content creators cannot do this manual mapping, (I agree it's something site builders might do with bundle-level fields similar to layout builder view mode config now), then each component added within XB by site editors will be 'coherent' in that its data will come from the same place and no need for 'partial field unions' which I agree would be weird.

💯 — made @lauriii aware in a meeting an hour ago 👍

I wonder if this is solely because we don't have #3455629: [later phase] [META] 7. Content type templates — aka "default layouts" — affects the tree+props data model built yet (where it'd indeed be a Site Builder decision), or whether @lauriii truly intends for Content Creators to (be able to) decide this.

The main goal isn't necessarily to make content creators do the mapping themself because the task of mapping fields to properties would be quite challenging for most content creators to manage (as @catch is arguing in #17). That said, the aim has been to include this capability consistently across the system for site builders to utilize. This would also enable us to build capabilities where site builders could pre-configure mappings to components, which would make it easier for content creators to utilize this capability.

Something to note is that the field mappings are not conceptually supposed to be restricted to Drupal fields. The plan is to eventually allow site builders to connect components to data from external APIs and other pre-configured integrations (e.g., Shopify, Zapier, Airtable, etc).

I think if I'm understanding #21 correctly, the use case that needs to be kept open is to have a component where the site builder has pre-configured mappings from different Drupal fields (or elsewhere), for say 3/5 of the sources, and then 2/5 are entered by the content editor. But then in that case, you could have a field union with two field types providing those two values and it would still be internally consistent. Can't think of a good concrete example but something along those lines?

The other case that maybe applies here is a component where the fields are mapped for everything (maybe with a hard-coded string value on the bundle layout level), but the content editor can provide overrides on a specific content item. For example a section heading that is the same on 90% of entities but can be customized for the other 10% of entities.

But in that case, the override can be an optional field value anyway, and the override just depends on whether it has content or not. And that would still be internally consistent too.

But it sounds like it's not necessary to support a use case where the site builder sets up a component, and the content creator can unilaterally remap where things come from arbitrarily.

But it sounds like it's not necessary to support a use case where the site builder sets up a component, and the content creator can unilaterally remap where things come from arbitrarily.

A content creator wouldn't necessarily map properties to fields but a site builder could do this even in the context of a single page. This is also one way in which the site builder could build these pre-defined mappings in the first place.

The goal is for the framework to behave similarly regardless if you're editing a component or a page. This way you get a consistent experience across the system, and can for example start building a component while you are building a page. This is a workflow that tools like Figma have popularized.

This way you get a consistent experience across the system, and can for example start building a component while you are building a page.

OK but if you do that, then the component that you're building would eventually get saved as config, and then if there is mappings involved, the source for those mappings would eventually get saved as field values - so you would still not have a page-unique field mapping? Or would you?

#19 @catch

If we store XB-entered data in Field Union, then XB will be writing that data to field union field values. This means that the field union data is field data, same as any other field (except the extra stuff it adds on top).

So this is different from what is proposed in the summary of this issue, correct?

In the summary there is static_values with the example

{
  "prop2": "Hello, world!"
}

But in what you wrote in #19 it seems like this would be written to the field_union field values instead. Otherwise you wouldn't get the benefit of using it in manage display or views.

Yes what effulgentsia wrote in the issue summary is not the same as what I was suggesting in #3440578: [PP-2] JSON-based data storage proposal for component-based page building option 3. It would be useful if he could explain what he thinks the pros/cons are and why he diverged in this issue because for me I can't really see the benefit of the specifics here where the field union is an extra config entity but doesn't do anything else.

But also I really think sorting out the use cases and desired functionality in the view modes issue needs to happen alongside this - see the discussion over there about partial field unions vs not from yesterday.

@effulgentsia I think it might stem from:

"The field table.." in that paragraph.

What I mean here is;

"The field union table (as distinct from current field union which does not use JSON) would store the field values, distinct/independent from the XB storage.

Instead of:

"The XB field table" would store field unions.

I thought this was a brand new approach, not a misunderstanding!

Yes something like that.

Define the XB field type as a subclass of dynamic_field_union and add the extra columns. Just like how in core FileItem subclasses EntityReferenceItem.
Define the XB field type as a field union of the first 5 columns plus a dynamic_field_union. This might actually be nice in terms of making the component column a full-fledged EntityReferenceItem (sub)field in its own right.

How I had it in my head is that the XB data would only reference the field union data (similar to how it does other fields on the entity), not incorporate it as such.

he mental model should be that a component instance is its own thing, where one of the things that it has is static values and those static values can be modeled as a dynamic field union.

Or that the component instance is its own thing, and it can reference dynamic values which happen to be in a field union.

would a stable item ID serve other use cases as well

I think it could help for translation, diff, and conflict resolution potentially? e.g. it would help to detect when field unions have been re-ordered as opposed to edited. If so, it seems as applicable to dynamic field unions as non-dynamic field unions.

Field union for me is 'field collections or paragraphs (or custom blocks) without the extra entities', so if we give those a uuid or similar then it should cover any latent use cases where having an individual identifiable thing was relied upon.

Another possible use case is for things like the featured image in an image gallery - if there's a way to select the featured image, and this is done by 'field union uuid' then that would persist across re-orderings. I would normally try to persuade someone that instead of selecting they should just automatically select the first delta instead, but if the requirements are specific it would enable use cases like that. I haven't personally used paragraphs (at least not for site building, I've seen it in performance audits...) but given it's theoretically possible to reference an individual paragraph entity now, I imagine a 'field union uuid' could be used to do similar things when they come up.

Postponed #3511852: Experience Builder support (default content dependency support) on this issue.

tagging for findability

Current issue summary: data_sources + static_values + field_union
First, the simple part. Evolution since this issue summary was created, but fits in existing proposal: data_sources must be able to store DefaultRelativeUrlPropSource. I don't see why it couldn't.

Then, the harder part. For a @ComponentSource=block component instance, we wouldn't use neither data_sources nor field_union. We'd be storing something like

        [
          'label' => '',
          'label_display' => FALSE,
          'use_site_logo' => TRUE,
          'use_site_name' => TRUE,
          'use_site_slogan' => TRUE,
        ]

… because block plugins have their own explicit input ("block settings") input UX + storage mechanism. Only the sdc and js ComponentSource plugins use the shape matching infrastructure and StaticPropSources. Which is why they're the two that subclass GeneratedFieldExplicitInputUxComponentSourceBase, which provides a "generated field-based explicit input UX" for any future component type as long as it has (JSON) schema information for each explicit input they accept.

@effulgentsia: how do you propose that to be represented?

One row per component instance per revision 🤔

Change the field type from single-item to multi-valued. Each item would be for a single component instance.

This would definitely easily result in millions of rows (one row per component instance per content entity revision) — because it's quite literally this excerpt from #3468272: Store the ComponentTreeStructure field property one row per component instance:

parent,slot,delta,uuid,component
ROOT_UUID,root,0,uuid-root-1,provider:two-col
ROOT_UUID,root,1,uuid-root-2,provider:marquee
ROOT_UUID,root,2,uuid-root-3,provider:marquee
uuid-root-1,firstColumn,0,uuid4-author1,provider:person-card
uuid-root-1,firstColumn,1,uuid2-submitted,provider:elegant-date
uuid-root-1,secondColumn,0,uuid5-author2,provider:person-card
uuid-root-2,content,0,uuid-author3,provider:person-card

would grow to something like the following based on the current issue summary + #3457504: XB field type: calculate all dependencies, store them, surface in new Component "Audit" operation:

delta,instance_id,component,parent,slot,uuid,data_sources,static_values,field_union,component,deps_config,deps_content,deps_module,deps_theme

… which in turn means querying dependencies (happening in #3457504) would have to match many more rows. Not necessarily a problem, but definitely a consequence to keep in mind.

Or … is @effulgentsia's idea to use instance_id to actually end up with

delta,instance_id

and then a separate DB table with

instance_id,component,parent,slot,uuid,data_sources,static_values,field_union,component,deps_config,deps_content,deps_module,deps_theme

to allow #3469082: Add way to "intern" large field item values to reduce database size by 10x to 100x for sites with many entity revisions and/or languages? 🤔

Also, when this issue was created, ContentTemplate config entities were a distant reality. Now they're becoming a close reality, with key pieces having already landed!

Thoughts on how you'd want #3519352: Content templates, part 3b: store exposed slot subtrees on individual entities to be reflected here?

… which in turn means querying dependencies (happening in #3457504) would have to match many more rows. Not necessarily a problem,

It's a benefit rather than a problem relative to all the other options in #3457504: XB field type: calculate all dependencies, store them, surface in new Component "Audit" operation because we know that relational databases can very efficiently query an indexed varchar even when there are millions of rows. Whereas even if there less rows, we either have unknown performance (indexed JSON queries across the three core-supported database types), or ones known to be bad (LIKE'%foo%').

Reviewed this and associated issues and created the following spikes to explore some of the options and be able to size and break this up

• #3523841: Versioned Component config entities (SDC, JS: prop_field_definitions, block: default_setting, all: slots for fallback) + component instances refer to versions ⇒ less data to store per XB field row
• #3468272: Store the ComponentTreeStructure field property one row per component instance
• #3523842: Spike: Explore storing a hash lookup of the ComponentInputs field property: one hash per component instance
• #3523844: Spike: Explore adding configuration options to the tree item formatter to support alternate use-cases
• #3523846: [PP-1] Spike: Explore storing component inputs in separate columns (aka field union)

At the moment we are using JSON storage for the Experience builder. This was done, because it is best solution from a performance prespective. In this issue we want to change that to a standard relational database structure. The result will be that, just like with the paragraph module, the performance will drop when we have complicated pages and/or a lot of pages. With JSON storage the page is stored in a single row in a single table. With a standard relational database structure the same data is split up in a lot of pieces (1 piece is a single row from a single table). When you load or update a page, every piece of the page needs to be read from the database or updated in the database. The more pieces of data that you have, the slower it will get. The same with the total amount of data. The more there is, the slower it will get. So, yes you can split up the page in a lot of pieces, just like was done in the paragraph module, and the result will be that you will get the same performance problems as with the paragraph module. The main difference with the paragraph module will be that the UI of the Experience Builder will be a lot better.

My educated guess is that we want to change the storage to a standard relational database structure, because we want to do things with the JSON data in the database that is not supported by MySQL or MariaDB. If I am wrong then please say so.
Support for JSON storage in MySQL and MariaDB is, and how do I say this in a polite way, pretty basic. Drupal core has another database that it supports and that is PostgreSQL. PostgreSQL has a for more advanced support for JSON storage. Without knowing what kind of functionality the Experience Builder needs for JSON objects in the database, I am pretty sure that PostgreSQL can do it. To be fully sure, I will need a list of all the things we need from the database for JSON storage to support the features that you want to add to the Experience Builder.

We have 2 options:
1. Change the storage to a standard relational database storage and keep support for MySQL and MariaDB. The main drawback will be the less then ideal performance.

2. Keep the JSON storage and have a great performance. The main drawback here is that it only works with PostgreSQL.

As I am not designer myself, so I have asked a couple of designers about the importance of performance with the Experience Builder. When I start talking about the Experience Builder their faces light up and they very much like the demo's that they have seen. When I then ask how much they like it when the performance will be the same as that of the paragraph module, they get a very disappointed look on their face. For them the performance is super important. I am not the product owner or the project owner, but for me the option to go for is very much the PostgreSQL only option.

I know that most of you have none or very little experience with PostgreSQL. That is fine. It is a change and people do not like change. From a strategic standpoint are MySQL and MariaDB just not the right choice. The owner of MySQL is the Oracle corporation. As the owner for the last 16 years, they have done very little to improve MySQL. They would very much like you to change you database to OracleDB. MariaDB is also supported by a single company. The problem here is that the company hasn't made a profit in a lot of years. Threfore they have very little money to improve MariaDB.
PostgreSQL is however supported by a large community. Just like the Drupal project. PostgreSQL and their community is been great run for a lot of years now. From a technical standpoint is PostgreSQL by far the superier database when compared to MySQL and MariaDB. PostgreSQL can be extended just like you can do with Drupal. Drupal modules are called extensions and there are a lot of them. They offer a lot of functionality that is not available in MySQL or MariaDB.

A lot of what Drupal can do is the result of the database it is using. Yes, the PHP part is very important, but so is the used database. With PostgreSQL you create far more advanced solutions with Drupal.

@daffie Thank you for the detailed analysis! I totally agree that performance is critical for Experience Builder. Improving performance over the existing solutions was one of the goals since the beginning.

Are you aware of the discussion in #3468272: Store the ComponentTreeStructure field property one row per component instance? The issue includes most of the intended changes. In #3468272-33: Store the ComponentTreeStructure field property one row per component instance there was an assessment from @catch where he stated that the expected performance impact would be largely neutral. It would be great if you could look at that issue and provide any further insights there!

What comes to relying on PostgreSQL, the challenge is that if we introduce changes to the hosting requirements, it will slow down adoption. Because of that, it seems unlikely that Experience Builder would introduce a dependency on PostgreSQL.

With a standard relational database structure the same data is split up in a lot of pieces (1 piece is a single row from a single table). When you load or update a page, every piece of the page needs to be read from the database or updated in the database. The more pieces of data that you have, the slower it will get. The same with the total amount of data. The more there is, the slower it will get. So, yes you can split up the page in a lot of pieces, just like was done in the paragraph module, and the result will be that you will get the same performance problems as with the paragraph module. The main difference with the paragraph module will be that the UI of the Experience Builder will be a lot better.

As @lauriii mentioned, the current proposed new schema is in #3468272: Store the ComponentTreeStructure field property one row per component instance. It's very different from paragraphs though so it feels like #46 is based on a misunderstanding.

Paragraphs stores each 'component' as a separate entity. This means each row has to be a separate entity save (and that will likely involve multiple tables for different fields too. This gets considerably worse with nested paragraphs too.

e.g. one node with 50 paragraphs:

node + node_field_data + node_field_revision
paragraph + paragraph_field_data + paragraph_field_revision * 50

= ~153 database queries / tables.

And there's not only the overhead of the queries themselves, but building the database queries in PHP, all the entity presave/update/insert hooks etc.

The proposed schema in that issue is for each 'component' to be a delta of a single field on the main entity, with the values themselves still in JSON. Field storage already writes all deltas of a field in a single query, so it's no more database queries than a single JSON blob to write or read each time. 1 entity save = 1 entity save still.

node + node_field_data + node_field_revision + node_field_xb = 4 queries / tables.

It should actually reduce write queries compared to the current schema by making #3521202: Store XB field type's "deps_*" columns in separate table to allow efficient querying partially or wholly redundant. That issue is mostly a workaround for lack of JSON index support in core, but I personally don't think that #3468272: Store the ComponentTreeStructure field property one row per component instance is a workaround at all, it should also help with implementation of issues like #3469082: Add way to "intern" large field item values to reduce database size by 10x to 100x for sites with many entity revisions and/or languages.

Also just to expand on:

with the values themselves still in JSON

This is what allows all the data to remain in deltas of a single field instead of separate tables.

We don't expect the field values of XB deltas to need to be queried on (by views or entity query) very often if at all - if/when they do, then once core supports JSON queries a bit better it will be doable.

What it does help with though, is querying on what the component for a specific delta is, or which components are in use across all entities and things like that - this will be just a regular varchar in its own column.

#3468272: Store the ComponentTreeStructure field property one row per component instance landed. Which is what prompted @catch to RTBC #3440578 at #3440578-87: [PP-2] JSON-based data storage proposal for component-based page building.

That means the
first half of this issue's title is done: Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree. 🎉

Work is under way over at #3523841 that will perform the second half of this issue's title: , and to reference the field_union type of the prop values — see @larowlan at #3523841-35: Versioned Component config entities (SDC, JS: prop_field_definitions, block: default_setting, all: slots for fallback) + component instances refer to versions ⇒ less data to store per XB field row.

AFAICT that means this issue should be postponed, to make sure that after #3523841 is done, no additional concerns are lingering.

@catch: agreed?

@Wim Leers, yes! afaict that issue covers everything that's in here, so this is hopefully mostly a reference point at this point until that one gets resolved.

👍

Let's ensure that we don't forget to revisit this after #3523841: Versioned Component config entities (SDC, JS: prop_field_definitions, block: default_setting, all: slots for fallback) + component instances refer to versions ⇒ less data to store per XB field row lands — prefixing with PP-1 :)

Captured in the meta as of #3520449-31: [META] Production-ready data storage. 👍

This has been covered by the issues referenced here.