ComponentSource plugin for code components

This should update the table at the top of #3454519: [META] Support component types other than SDC, block, and code components.

@balintbrews: Did you intend for this issue to deliver end-to-end working code components, in the sense that:

• my code components appear in the left-hand Library in the XB UI
• I can drag them onto the canvas and see a preview
• I can populate their inputs using a field widget, and the preview updates
• I can save it

?

#7: Yes to all of those, with the following notes.

1. From the issue summary of #3499927: Config entity for storing code components:

   The status property could potentially be used in #3498889: ComponentSource plugin for code components to decide which code component to pick up and present as a component that can be used for creating content. Learn more about the distinction in #3500071: Adding code components to components.

   I don't yet see that property added in the config entity. It's a different issue which is still in progress, but I wanted to highlight it, because it will be relevant here.

2. Before we land #3500058: Hydration library for code components, workarounds will be needed in order to see a preview of the components. From the issue summary of this issue:

   Follow the steps in this comment to attach the appropriate JS files that you need to manually place in your files folder: #3483267-17: [exploratory] PoC of Preact+Tailwind components editable via CodeMirror. We'll create a proper library to use in #3500058: Hydration library for code components.

   If this issue lands before the hydration library, we'll adjust the component source plugin as part of the hydration library issue to remove the need for the workarounds.

Okay! That then shows that the

      // @todo Make actual render array.
      preview: [
        '#markup' => '@todo Make something 🆒',
      ],

bit in #3499927: Config entity for storing code components should indeed point here 👍

1. I don't yet see that property added in the config entity.

   It is — it's inherited from \Drupal\Core\Config\Entity\ConfigEntityBase::$status (and prescribed by type: config_entity in config schema).

2. 👍

Updated issue summary to include items from a call with @effulgentsia

#3499927: Config entity for storing code components is in! 🥳

Note: as described in #8 (and the issue summary of #3499927) only if JavaScriptComponent::status() === TRUE should it get a Component config entity generated.

Spent some time reviewing the original POC and how it wires up Astro
I think there's probably scope to split out some smaller stories from this one and make this a parent story as follows:

* Astro render element, this is reasonably straight forward in so far as it needs to take some #keys and render the twig like seen in the demo
* A library alter hook that should dynamically build library definitions based on defined code component entities - we will probably need to think about dependencies between components when one component makes use of another, but we can equally punt that to a followup
* Something similar to core's AssetControllerBase that serves CSS/JS from the assets:// stream wrapper to serve the compiled css and js as stored in the config entity. As mentioned we will likely need support for a unique hash of the contents and a preview flag. The preview flag will read from auto-saved entries. The hash will ensure we don't serve stale items. When we auto-generate libraries using the hook alter (item above) we will resolve these to the URLs for this controller. If the file exists on disk it will be served by apache. If it doesn't we'll write it to disk and then return it. This should be able to make use of a lot of the lazy asset work that landed in core in 10.1. The urls could be something like

    $directory_path = $this->streamWrapperManager->getViaScheme('assets')->getDirectoryPath();
    $routes['/' . $directory_path . '/xb-css/{js_component}/{hash}'] //...

like we have in core in \Drupal\system\Routing\AssetRoutes::routes but rather than passing a filename in the route we could pass the machine name and hash. We can set the hash as the version in the library alter and use it in the library definition URLS.

We can then append ?preview=1 to the URL for the preview version and likely add some access controls. At some point in the future this could use the workspace ID in the URL too, to allow discrete versions per workspace. The preview version most likely wouldn't dump to file as we would expect this to be regularly updated.

For now I'll start on these in this story but it might be worth splitting them off into smaller issues if the MR size grows too large

we will probably need to think about dependencies between components when one component makes use of another, but we can equally punt that to a followup

+1, because that'd also require the config entity to know about those dependencies. That was not in scope for #3499927: Config entity for storing code components (nor mentioned as future work, actually), and this is in the critical path for many other things. So let's stick to just individual code components in this issue 🙏

Something similar to core's AssetControllerBase that serves CSS/JS from the assets:// stream wrapper to serve the compiled css and js as stored in the config entity. […]

I think it could be much simpler:

• Generate those CSS and JS assets on disk, regardless of whether config is saved through user interaction or through config deployment — similar to https://git.drupalcode.org/project/component_library/-/commit/a78f3af960...

— #3499933: Storage for CSS shared across in-browser code components (and other use cases in the future)

[…] We can then append ?preview=1 to the URL for the preview version […]

Why do we even need to load these assets while previewing? That'd require going from client to server and back to the client? The client-side already has literally all the information it needs (source+compiled CSS, source+compiled JS), I don't see why the server needs to be involved at all? 🤔

Un-assigning @larowlan because it's a holiday in Australia on Monday. He'll likely be the one to push this forward further, but that shouldn't prevent somebody from doing so until he returns :)

[…] We can then append ?preview=1 to the URL for the preview version […]

Why do we even need to load these assets while previewing? That'd require going from client to server and back to the client? The client-side already has literally all the information it needs (source+compiled CSS, source+compiled JS), I don't see why the server needs to be involved at all? 🤔

I agree that we can keep the preview client-side. It's also how we implemented it in #3483267: [exploratory] PoC of Preact+Tailwind components editable via CodeMirror.

In #3500034: Preview for code components we need to do compilation with @swc/wasm-web and tailwindcss-in-browser in the browser anyway, at which point we have everything to display a preview. Like Wim said. 😊

Added the library hook and attached it to the island render array.

Wondering whether, similar to core, the paths should be something like assets://xb/css/HASH.css?component=BLAH - this way if two components happen to share identical CSS they will share the CSS file as well. Core uses the query string to build and then the stored file doesn't otherwise care.

For the preview CSS at least, I think we might need a separate library definition and separate route/controller to serve the CSS. Otherwise we have to invalidate library definitions every time the hash changes? If this is the case maybe we should do the same with the JS for consistency. Unless we don't need the preview at all as per #15/16.

Implemented asset generation as per #15.

My thinking around the client side/server preview comes down to import maps. If we need an import maps it needs to be in an IFrame. But if it's already solved, that's awesome

Everything in this MR so far has focused on the astro island/rendering/assets aspects.

But there's more to this issue:

1. The issue summary states clearly that there must be a new ComponentSource plugin.
2. I asked @balintbrews this in #7 and he responded affirmatively:
   

   my code components appear in the left-hand Library in the XB UI

Working on those two aspects.

Delivered on much of #21, but not everthing yet. EOD, so passing context + next steps on to the next person:

1. Managed to lift 600 LoC out of \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\SingleDirectoryComponent into GeneratedFieldExplicitInputUxComponentSourceBase. I kept it as simple as possible, and resisted doing significant refactoring 🤓
2. This made SingleDirectoryComponent merely ~275 LoC, and there's likely more still to be done.
3. Similarly, extracted the prop_field_definitions setting for the SingleDirectoryComponent ComponentSource plugin out of its config schema type into a new config schema base type: https://git.drupalcode.org/project/experience_builder/-/merge_requests/5...
4. Which then allowed me to quite simply/elegantly introduce the JsComponent ComponentSource: https://git.drupalcode.org/project/experience_builder/-/merge_requests/5...

Next steps:

1. auto-creating Component config entities based on JavaScriptComponent config entities. @longwave suggested automatically doing that whenever a JavaScriptComponent config entity is saved. I agree that makes sense. This would be a ::postSave() override in \Drupal\experience_builder\Entity\JavaScriptComponent.
2. The easiest way to test is to expand \Drupal\Tests\experience_builder\Kernel\Config\ComponentTest — which was previously expanded in #3475584: Add support for Blocks as Components to test Block-sourced Component config entities.
3. #3501290: Introduce unit test coverage for both ComponentSource plugins (Block + SDC) would make writing tests for this a hell of a lot simpler. ⚠️ But I think we explicitly should keep that out of scope to ensure we land this in the next few days rather than growing it. Let's add unit tests in #3475584.
4. Docs: updating docs/components.md!
5. I'm sure there's more, but those are the first steps :)

P.S.: I did not test \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\JsComponent::renderComponent() at all, I just matched what @larowlan did in \Drupal\Tests\experience_builder\Element\IslandTest. It likely needs updating 😅

Also merged in upstream and resolved conflicts, because this conflicted heavily with #3500994: Remove references to 'props' outside of SDC - use 'inputs' instead.

Should make for a smooth start for the next person 😊

SO great to see @larowlan continued where I left off 🤩

• Favorite changes are the ComponentSourceInterface API tightenings by @larowlan — best example: https://git.drupalcode.org/project/experience_builder/-/merge_requests/5... — done in a way that does not balloon scope and https://www.drupal.org/project/experience_builder/issues/3498889#:~:text... remains true 👍
• Worried: https://git.drupalcode.org/project/experience_builder/-/merge_requests/5... and https://git.drupalcode.org/project/experience_builder/-/merge_requests/5... implement the majority of #3473275: Support disabled Block Components + multiple reasons for incompatibility — I'd definitely have considered that out-of-scope 😅 That's how we went from 14 files changed, 855 insertions(+), 742 deletions(-) to 30 files changed, 1632 insertions(+), 915 deletions(-) 😳 Postponed the other issue: #3473275-13: Support disabled Block Components + multiple reasons for incompatibility.
  (And related: https://git.drupalcode.org/project/experience_builder/-/merge_requests/5....)

But it looks like @larowlan tackled things I had not even identified in #22, so I can literally work on the list I created there 😄

It looks like I was wrong:

• @larowlan did implement the "auto-create upon save" functionality, just not in ::postSave(), but in the new JavascriptComponentStorage 👍 Only thing that needed changing: respect the status flag that #3499927: Config entity for storing code components assigned a particular meaning to.
• He also added explicit test coverage for it, in JavascriptComponentStorageTest. Still, updating testComponentCreation() the way that I did provides value: it ensures that identical props for an SDC and a "code component" result in the exact same prop_field_definitions 👍

I did find some problems though, and I've fixed those.

I think I won’t be able to push this forward the next few hours. The only remaining thing I have partially (locally) is schema refinements, but there’s plenty more to do that won’t touch that.

Finished everything in #22, except for the docs (only a stub was added).

Iterated on everything @larowlan did overnight, and between his work and mine, we surfaced a bunch of oversights in both #3499927: Config entity for storing code components and the way that even the foundations of Component config entity type's support for ComponentSource-specific settings were not quite right yet. Which is understandable, because it's this issue that's introducing a 3rd ComponentSource plugin (and as the adage goes: an abstraction requires 3 concrete uses), and one that is quite different in some ways and almost the same in others!

I've spent most of today increasing test coverage, and in turn revealing weaknesses in our validation and hence the reliability of the Component config entity type in combination with the new ComponentSource this MR is introducing. Those weaknesses are important to eradicate to remain on track for requirement 14. Configuration management.

The one bit where I'm going quite strongly against what @larowlan did last night is https://git.drupalcode.org/project/experience_builder/-/merge_requests/5..., because by using ContribStrictConfigSchemaTestTrait, it became clear that all of those edge cases @larowlan was testing … actually are made impossible by the validation for the JavaScriptComponent config entity, and for good reason! (See the commit message.)

I believe that now, especially after the source-specific test coverage (which revealed missing validation) and its fixes (A — note how this causes the exact same fails + B) everything is in place 🤓

The only thing I have not reviewed at all is all rendering aspects: the astro island and asset generation. But for those, I'm happy to defer to @longwave & @larowlan. I want to get the config bits to be as robust as possible as soon as possible because they affect the data model/update path/overall stability — tweaks to the rendering can easily happen in subsequent MRs.

So: approved this MR! Hopefully @larowlan & @longwave think this is RTBC by tomorrow 🤞😊

FWIW

@larowlan did implement the "auto-create upon save" functionality, just not in ::postSave(), but in the new JavascriptComponentStorage

I did it in the storage handler because we have DI in storage handlers, in Entity::postSave we have to use \Drupal. We use this pattern fairly regularly for client projects and I think it is cleaner.

#28:: ah, of course! I somehow always forget that’s an option 😇🙈

Green and lots of commits overnight 🤩

Reviewing!

@larowlan opened #3502982: Rename ComponentSource settings' `plugin_id` to `local_source_id` or similar to not bias towards source plugins that don't use plugins under the hood and #3502988: Move more `ComponentPluginManager` methods to the SDC `ComponentSource`.

I opened #3503038: Enable candidate `DynamicPropSource` suggestions for code components: refactor `GeneratedFieldExplicitInputUxComponentSourceBase` and `FieldForComponentSuggester` to need only SDC's ComponentMetadata, not SDC plugin instances.

The goal for this issue, confirmed by @balintbrews in #8, is this:

@balintbrews: Did you intend for this issue to deliver end-to-end working code components, in the sense that:

• my code components appear in the left-hand Library in the XB UI
• I can drag them onto the canvas and see a preview
• I can populate their inputs using a field widget, and the preview updates
• I can save it

?

Ideally, we'd have an end-to-end test. But I think given that 99% is identical to SDCs, we could get away with not having that, and a future issue could do that — the scope of this one is already enormous.

I think that this MR could land if:

1. manual testing shows it works
2. the path is paved for a future end-to-end test, by ensuring that a JS component is already present and available for placing in e2e tests

Thanks to @longwave for getting the missing bits for Astro assets in place 👍 He also tells me he has essentially only nits. 😊

Good news all around, because:

1. Manual testing shows it works:
   
   (for this demo, I only commented out the if ($this->configInstaller->isSyncing()) early return in JavascriptComponentStorage — to force a corresponding Component config entity to be created 👍)
2. Pure back-end test
3. End-to-end test that verifies a code component can appear in the primary panel's "Library" tab and renders a preview on hover: (failed upon adding 👍)
4. End-to-end test that verifies a code component can be added to the canvas (failed upon adding 👍)
5. Made the end-to-end tests pass again by actually creating the expected JS component: https://git.drupalcode.org/project/experience_builder/-/pipelines/409263

Given how many front-end issues this blocks (virtually everything in #3499919: [Meta] Plan for in-browser code components), plus it following the exact same pattern as the pre-existing test logic, plus the vastness of this MR (~2500 LoC added, net 😳), I'm not awaiting FE approval for the end-to-end tests. It's too important to land this sooner rather than later!

So: RTBC'ing. Only thing outstanding: docs. First: belated lunch.

Crediting @effulgentsia, who had the vision for this plugin and using Astro islands.