[PP-1] Decorate the SDC plugin manager and allow components defined in code

Thinking about this some more, after having created two issues with very granular first steps towards multiple component types:

1. #3469609: Prepare for multiple component types: ComponentTreeStructure should contain Component config entity IDs, not SDC IDs
2. #3469610: Prepare for multiple component types: prefix Component config entity IDs with `sdc`

Then looking at this issue made me realize that #3469610 is seemingly at odds with this issue. (Postponing that issue on getting a conclusion in this issue.)

We want to be able to expose blocks (and layouts, paragraphs for BC) as components.

Do we want to expose all of those as SDCs (meaning: wrap anything that is not an SDC into an SDC), or do we want to expose them as their own thing?

This is a crucial decision, both from a product level POV ("everything is an SDC" is nice and simple, likely helping facilitate a consistent UX) and an architectural POV!

We only need #3469610: Prepare for multiple component types: prefix Component config entity IDs with `sdc` if we make the latter choice. That's the direction I imagined this to be going. But this is proposing something different. (I know I gave a thumbs up at #3454519-22: [META] Support component types other than SDC, block, and code components for ComponentSourceInterface and this direction, but thinking about it more makes me wonder whether that is truly feasible.)

However … if we truly could model everything as SDCs, that'd be very interesting+elegant, of course! 🤠

I'm just not sure if that's realistic, per #3454519: [META] Support component types other than SDC, block, and code components. For example, for blocks:

1. we'd need to transform each block setting to an SDC prop, which is possible thanks to the schema information in the config schema (block.settings.*)
2. we could then generate a form for it, using exactly the same mechanisms as we currently use for "real SDCs" … but then we'd just not use the existing form definitions for block plugins — is that intentional?
3. what about blocks' support for the "context" system? SDCs have no such concept.

IOW: AFAICT block plugins can do more than SDCs, which is why "wrapping an SDC in a block plugin" seems easy (hence: https://www.drupal.org/project/sdc_block), but the inverse (which is being proposed here) seems hard?

That's exactly why I started the table in the issue summary at #3454519: [META] Support component types other than SDC, block, and code components, and why the Inputs + Required context columns are so important. We didn't expand that table yet for Paragraphs.

Are we really confident that we'll be able to do this too for Paragraphs, for which we have product requirement 42. Paragraphs migration? In https://git.drupalcode.org/project/experience_builder/-/merge_requests/68, you seemed to convey confidence about this, @larowlan, but we didn't discuss that in detail.

So: let's double-check before we go down this path. I'd love for this to be true though, and would love to close #3469610: Prepare for multiple component types: prefix Component config entity IDs with `sdc` :)

Discussed #2 with @f.mazeikis and @lauriii. No outcomes, just made them aware, to ensure they start thinking about this and participate here 😊

This is a crucial decision, both from a product level POV ("everything is an SDC" is nice and simple, likely helping facilitate a consistent UX) and an architectural POV!

Can we keep SDC API for pure UI Components (card, button, menu, slider, steps, breadcrumb...), as designed by UI & UX designers, published in design systems and implemented by the front developer, instead of trying to fit everything as SDCs?

As a product, XB will benefit of keeping its use of SDC as a clean, single purpose and well defined API, which is doing one thing and is doing it well, without application state nor business logic.

#4: That's the direction I'm thinking would be best too. It's why I tried to capture the consequences (pros and cons) of going with @larowlan's proposal (if I understood it correctly) in #2.

It's why I'm leaning towards #3469610: Prepare for multiple component types: prefix Component config entity IDs with `sdc`.

Can we keep SDC API for pure UI Components...instead of trying to fit everything as SDCs?

What do you mean by the "SDC API" in that comment? For example, a key way of using an SDC is in a render array:

$element = [
  '#type' => 'component',
  '#component' => 'some_sdc',
  '#props' => [
    'prop1' => 'foo',
    'prop2' => 'bar',
  ],
  '#slots' => [
    'slot1' => [...],
    'slot2' => [...],
  ],
];

Why should '#type' => 'component' mean only an SDC? Why not be able to think of a menu block as a component and render it like:

$element = [
  '#type' => 'component',
  '#component' => 'block::system_menu_block:main',
  '#props' => [
    'level' => 1,
    'depth' => 2,
    'expand_all_items' => true  
  ],
];

This is where my approach of adding a component source plugin to the config entity came from. We add one for each type (Block, paragraph, SDC) much like media types. Maybe we revisit that?

I mostly like this. I just don't like it being coupled to XB. But perhaps it makes sense to start with this and later see how we can abstract the concept of a "component" somewhere more centrally so that "component" can mean any kind of component, and SDC is just one kind of component?

#7: because component means "SDC" as in "Single-Directory component".

The pseudo code you posted in #7 would mean "menu block represented as an SDC", i.e. the inverse of https://www.drupal.org/project/sdc_block.

Auto-transforming block plugin's settings' config schema + ::defaultConfiguration() to SDC props with schema

I think for that particular example it's feasible (see what I wrote ~3 weeks ago in #2: we'd need to transform each block setting to an SDC prop, which is possible thanks to the schema information in the config schema (block.settings.*)), because we could indeed auto-convert the menu block config schema to a JSON schema that would represent each of the key-value pairs in the block settings as SDC props.

The only problem would be the absence of default values. We could require block.settings.* schema to be enriched to provide that information. We can get that information from ::defaultConfiguration(). For example, \Drupal\system\Plugin\Block\SystemMenuBlock::defaultConfiguration():

  public function defaultConfiguration() {
    return [
      'level' => 1,
      'depth' => 0,
      'expand_all_items' => FALSE,
    ];
  }

i.e. we could auto-transform SystemMenuBlock::defaultConfiguration() +

block.settings.system_menu_block:*:
  type: block_settings
  label: 'Menu block'
  mapping:
    level:
      type: integer
      label: 'Starting level'
    depth:
      type: integer
      label: 'Maximum number of levels'
    expand_all_items:
      type: boolean
      label: 'Expand all items'

to

$schema: https://git.drupalcode.org/project/drupal/-/raw/HEAD/core/assets/schemas/v1/metadata.schema.json
name: Menu block
props:
  type: object
  required:
    - level
    - depth
    - expand_all_items
  properties:
    level:
      type: integer
      title: 'Starting level'
      examples: [1]
    depth:
      type: integer
      title: 'Maximum number of levels'
      examples: [0]
    expand_all_items:
      type: boolean
      label: 'Expand all items'
      examples: [false]

Immediate UX consequences

🚨 That will inevitably lead to less-than-great UX because actually:

1. level may not make sense because only SystemMenuBlock::blockForm() knows that it should inspect $this->menuTree->maxDepth() to determine what the actually valid range for this menu block's menu is!
2. depth: same exact challenge.

The UX inside XB (based on the SDC metadata) would be that any integer can be specified, including negative integers (which would never make sense) as well as e.g. 3 as the level even if it's a menu that has no hierarchy at all. Which points to another problem: neither level nor depth nor expand_all_items make sense for "tagging"-style vocabularies, where there simply is no hierarchy!

Extrapolated UX consequences

Because the existing block system has different restrictions (fewer), boundaries, different trade-offs, it is a huge challenge to make the UX inside XB for placing blocks as simple as that for placing SDCs.

What does it mean to "be an SDC"?

IMHO the two most defining characteristics of SDCS are:

1. they have clearly defined inputs, that are specified by the user of the SDC
2. they have no logic

Representing blocks as SDCs would violate both:

1. blocks can consume not just user input ("block settings"), but also global context (grep for context_definitions: in Drupal core — see https://www.drupal.org/node/3016699 and https://www.drupal.org/node/3029856)
2. for SDCs, the inputs can be directly traced to Twig template inputs thanks to the absence of logic, for blocks ::build() might "consume" those inputs (i.e. they may simply not appear in the AST at all!)

Consequences:

1. How does the UX convey what the user has just built? Why a block will appear/disappear based on circumstances?
2. It'd be impossible to do actual real-time preview updates for "Block components" once we start parsing SDCs' templates into an AST (see @larowlan's #3453690-13: [META] Real-time preview: supporting back-end infrastructure). Why? Because Block plugins can have arbitrarily complex render logic (as opposed to a static template!) that may consume the inputs (level, depth, expand_all_items) in that logic. That is actually the case for
   See how complex \Drupal\system\Plugin\Block\SystemMenuBlock::build() is for example — it calls many other services and then returns a render array like this:
   

It's also why the inverse direction is totally feasible: https://www.drupal.org/project/sdc_block is trivial because SDCs are defined in more detail and have fewer capabilities (mostly: zero logic), so exposing SDCs as blocks takes only ~20 LoC.

Now, it may be acceptable that real-time preview updates are impossible for "Block components", i.e. that a server round-trip is required. But that still leaves the first UX consequence.

Thanks Wim for this deep and helpful analysis.

Block plugins are stateful, context-aware, applicative objects.
SDC components are stateless, context agnostic, UI objects.

Different beasts.

Let's re-assess this after #3475584: Add support for Blocks as Components.