Data surfaces: a definition layer between Form API and storage

Summary

Drupal has no general API for "what values does this thing accept right now, on this site." Content entities come closest: typed data definitions and entity validation serve forms and JSON:API alike, though the definitions themselves are static. But for configuration, plugin settings, and any value whose valid range depends on other values or site state, that contract lives as private knowledge inside form classes, and everything that is not the form (config sync, recipes, Canvas, agents) either re-derives it or goes without.

This proposes data surfaces: runtime groups of data definitions that can be refined by the values of their siblings (choose an entity type and the bundle definition narrows to that entity type's bundles), from which forms are generated, validation runs identically in every consumer, and an honest JSON Schema can be emitted or ingested. Form API keeps rendering; it stops being the only place the rules live. A working proof of concept exists, adopted progressively on plugin types that have never heard of it, with no core changes.

The gap

Drupal has a strong story for describing stored data and a strong story for collecting data, but very little connecting the two.

On one side there is storage: config schema, typed data, entity storage definitions. On the other side there is Form API: elements, validate handlers, submit handlers. The layer in the middle, "what values does this thing actually accept right now, on this site, in this state," does not exist as an API. It exists as the private knowledge of individual form classes.

Look at almost any settings form, config entity form, or plugin configuration form and you will find the same pattern: the form builds elements by hand, options are computed inline in buildForm(), validation lives in validateForm() (when it exists at all), and submitForm() reshapes the values into whatever the storage side expects. The real contract for the data is spread across three form methods and is invisible to everything that is not that form.

The smallest version of the problem makes it concrete. A setting stores 0, 1, or 2, and a constraint dutifully checks that the value is one of those three integers. What do they mean? The only place that knows is a form's #options array: 0 is "Disabled", 1 is "Optional", 2 is "Required". The constraint validates values it cannot explain, the labels exist only in presentation code, and every other consumer (an API response, a config export, a recipe author reading YAML, an agent deciding what to send) sees three bare integers. Even where core does carry labels, in OptionsProviderInterface, they hang off the instantiated data object rather than the definition, so nothing that works from definitions alone can reach them. Validation without meaning and meaning without validation, held by different layers.

The systems that could not live with this built their own bridges, and even the best bridge only covers half its territory. Field API is the clearest example: on the content side, field types define property and storage definitions with constraints, and widgets map them to elements. But those definitions are static, and the settings side got no definitions at all; a field type's own configuration is described by nothing beyond the form that collects it and the config schema that stores it. Views has its own approach. Plugin configuration (defaultConfiguration(), a hand-built form, and config schema that may or may not match) amounts to three separate spellings of the same shape, with nothing enforcing that they agree.

Config schema was intended to be the middle layer for config, and it deserves more credit than "static YAML." It supports dynamic type references: a mapping's type can be selected from the value of a key, a sibling, or a parent ([%key], [%parent.field_type]), which is real value-dependent shape resolution and has carried systems like field settings storage for years. The precise gap is narrower, and stronger for being stated precisely: config schema cannot compute runtime options or constraints from services, permissions, site state, or arbitrary sibling values. It can select among pre-declared types based on a stored value; it cannot say "this is one of the bundles of the entity type selected above," because answering that requires asking a service. So the interesting constraints, the ones that depend on the live site, still exist only in form code. Everything downstream of the form (config sync, recipes, REST, tests, tooling) has to trust that whatever got stored was valid at the time.

Core has recognized this and has been building toward the middle from both ends. On the form side, #config_target (in ConfigFormBase since 10.2) binds form elements directly to config properties: defaults load from config, submitted values validate against config schema constraints, and violations map back to the originating elements. On the schema side, the config validatability effort is making constraints complete enough to trust. Both are real progress, and both stop short of the same line: the form is still hand-built, the element-to-property mapping is still declared inside the form class, and there is still no machine-readable answer to "what does this accept" that exists outside PHP. #config_target bridges an existing form to existing schema; it does not remove the need to write and maintain the form.

Meanwhile the number of parallel answers keeps growing. Single-directory components describe their props in actual JSON Schema, validated by a bundled JSON Schema validator, and Canvas builds on that. So core currently maintains three languages for "what values does this accept": config schema for config, typed data for content, and JSON Schema for component props. They share no constraints, no labels, and no validation code.

That is the gap this proposal intends to fill.

The proposal

Introduce data surfaces: a runtime group of data definitions that describes the values something accepts, including how definitions depend on each other. The name is meant in the "API surface" sense: the contract a thing exposes about the values it accepts, as opposed to how it stores them or how a form happens to render them.

A data surface consists of:

• An ordered set of data definitions (label, description, data type, constraints, default: the typed data vocabulary we already have, extended where it falls short).
• A refinement map: which definitions depend on the values of which sibling definitions.
• Refiners: handlers that, given the dependency values, return a narrowed version of a definition. Tighter constraints, a concrete option list, a sharper description. Refinement may only narrow what the surface advertised, never widen or change type. That contract is what keeps a refined surface trustworthy: any value valid under the refined definition was already legal under the advertised one.
• An alter/build step where other modules can extend the surface (add namespaced definitions, append refiners) before it is sealed.

The surface becomes the single integration point between intent and storage:

• Forms are generated from the surface. Each definition maps to elements; each refinement dependency becomes dynamic behavior (an AJAX rebuild, or #states when the narrowing is client-computable). The form and the validation cannot disagree, because they are derived from the same object.
• Validation runs against the refined surface, everywhere: in the form, in REST/JSON:API writes, in config actions, in tests, in tools. One set of constraints, one set of violation messages, path-aware so an error attaches to the exact nested value that caused it.
• Storage receives values through one visible pipeline: accept, validate, prepare for storage. The translation from submitted values to stored values happens at a named point instead of inside a submit handler.

The canonical example

A surface accepts entity_type_id, bundle, and field_name. bundle depends on entity_type_id; field_name depends on both.

The generated form asks for an entity type first. Once one is chosen, bundle refines from an open value to a select of that entity type's bundles. Once both are set, field_name refines to a select of that bundle's fields. Because refinement is strictly narrowing, optional dependencies degrade gracefully: if bundle is optional and empty, field_name remains an open text field validated against the entity type's field list, and tightens the moment a bundle is picked.

No form code was written for that behavior, and no AJAX callback. The dependency structure is the form behavior, and the same structure is what a REST consumer, a recipe, or an agent sees.

This example is implemented in the data_surface_demo submodule of the PoC, as a block plugin whose entire settings form is generated, with the refiners as methods on the plugin. A second submodule, data_surface_demo_extras, shows the alter story: a third-party module mounts its own namespaced setting onto another module's surface and appends a refiner to one of its definitions. The generated form, validation, and summary all pick it up without a form_alter anywhere.

About the PoC

The data_surface module (github.com/mlander/data_surface) is a vibe-coded proof of concept, not an architectural recommendation. It exists to show that the model holds together end to end on today's core: surfaces generate real forms inside real hosts (block placement, Field UI formatter rows) that core knows nothing about, refinement drives AJAX correctly, alters compose, values round-trip to storage, and all of it is covered by kernel tests rather than functional ones. The specific class names, the attribute mechanics, and where things live are all open questions. The shape of the idea is what I am proposing.

A note on the name: the PoC started config-focused, because plugin and settings configuration is where the pain is most visible, and the module name reflects that starting point. Nothing about the mechanism is config-specific. A surface describes values being collected, wherever they end up: config, state, third-party storage, a tool invocation, or a form that never persists anything. That is why this proposal says data surfaces.

The result worth calling out is that none of it required touching core or the host plugin types. The demo includes a field formatter that declares its whole surface in a PHP attribute on the plugin class and answers Field UI's existing settingsForm()/settingsSummary() protocol from it. Field UI has no idea surfaces exist; the formatter adopted them anyway. That is the adoption model in miniature: individual plugins can start declaring surfaces on plugin types that do not support them yet, and when a plugin type gains native support, the per-plugin shim is exactly the code that gets deleted. The declarations survive unchanged.

Why refinement is the important part

Static definitions force a choice between two bad options: enumerate every possible combination up front, or under-describe and let validation (or the user) discover the rules later.

Enumeration does not scale. A tool that sets a field value takes an entity, a field_name, and a value. The honest static description of value is "anything," because what it really is depends entirely on the first two inputs. Pre-generating a definition for every field of every bundle of every entity type is not realistic; sites are too large and change too often. With refinement it is unnecessary: value starts as any and refines to the exact property structure and constraints of the chosen field once entity and field_name arrive. The Tool API project does exactly this today, and it is the difference between a usable contract and trial and error.

Under-describing is what most of core does now, and it is why so much validation exists only inside form classes. The dependency information is not representable as data, so it gets written as imperative form code and lost to every other consumer.

Refinement makes the dependency structure data. That is the core of the proposal.

Alternatives considered

Refineable is not the only way to express value-dependent shape, and the alternatives are worth taking seriously, both because they inform the design and because two of them already exist in some form.

Declarative conditionals. JSON Schema expresses conditionality statically: dependentRequired ("if this key is present, these keys are required"), dependentSchemas ("if this key is present, apply this subschema"), and if/then/else branches, composed through allOf when there are multiple conditions. This is genuinely useful, and for closed, small variant spaces it is the right tool: "if casing is uppercase, variant is one of bold or strong" is a perfectly good dependentSchemas clause. But it has a hard ceiling, and it is the same ceiling config schema's dynamic types hit. First, every branch must be enumerated in the document. The canonical example in the JSON Schema documentation is postal code formats switched by country, written as an allOf of if/then pairs, one per country, and it covers three countries. Extending that pattern to "bundle depends on entity type" means one branch per entity type on the site, per site, regenerated whenever site structure changes. The variations multiply out instead of being computed. Second, a static document cannot consult anything: no services, no permissions, no site state. The conditional can only rearrange information that was known when the document was written. So conditionals cannot replace refinement, but they make an excellent projection of it: a refinement whose dependency space is closed and enumerable can be compiled to dependentSchemas/if-then for static consumers, while site-state-dependent refinements resolve live. Refinement is the source of truth; conditionals are one of its outputs.

Config schema's dynamic type references. The [%key]/[%parent...] mechanism is prior art for value-dependent shape inside core, and it should be acknowledged as such: it is a discriminator lookup, where a stored value selects among pre-declared types. Seen from the surface side, that is a special case of refinement, specifically discriminated narrowing through a static mapping table. It works because the discriminator's value space is closed at the time schemas are compiled (there is a schema per field type because field types are enumerable). It cannot compute, only select, and its resolution vocabulary is limited to key and parent traversal. Rather than compete with it, a surface layer should subsume it: existing dynamic type references map naturally onto surfaces as pre-declared refinements, which would let current config schema participate without rewrites.

Per-definition option callbacks. Core also has allowed_values_function and OptionsProviderInterface: a callable computes live options for one definition. This solves the "ask a service" half of the problem but not the other half; there is no way to declare that the options depend on sibling values, so the callable either ignores context or reaches for it through globals, and either way the dependency is invisible to forms, schemas, and consumers. Refinement is essentially this mechanism with the dependencies made explicit and the narrowing contract making it safe to trust.

Extending config schema itself. The most natural counter-proposal is to teach config schema what it is missing: let YAML reference services for options and constraints, and the gap closes without a new layer. I do not think that holds up, for a few reasons that compound. First, it means writing behavior into YAML: strings naming services and methods, with no types, no refactoring support, and no discoverability, which is the pattern core has been steadily moving away from everywhere else. Second, config schema is scoped to config by design. The same gap exists for plugin configuration wherever it lives, for tool inputs, and for values that never touch config storage; solving it inside config schema solves it for one host. Third, and most fundamental, config schema describes stored data and resolves against complete values. Refinement operates at the collection moment, against partial, in-progress values: a form mid-edit, an agent mid-conversation, a recipe mid-application. Config schema has no concept of "the values so far," and bolting one on would strain its compile-and-cache model, which exists precisely because schemas are static. Finally, the input contract and the storage shape are not always the same thing; the accept/validate/prepare pipeline in this proposal exists because the translation between them deserves a visible home. Extending config schema to cover input would fuse the two shapes back together, which is the conflation this proposal is trying to undo. The relationship should run the other way: config schema remains the storage description, a surface can be derived from it as a starting point (the same ingestion direction described below for JSON Schema), and the runtime layer adds what a static document cannot know.

The pattern across these mechanisms: each covers one fragment (static branching, discriminated type selection, live options for a single value) and refineable is the primitive that generalizes them, while remaining exportable back into the static forms where those suffice.

Prior art

Form generation from definitions is not a new wish, and the existing attempts map neatly onto the two halves of this proposal.

Much of this exists in partial form in contrib, and this proposal descends from it. The Typed Data API Enhancements module provides TypedDataFormWidget plugins (the definition-to-element bridge), and Rules' context system has refineContextDefinitions(), where a plugin narrows its definitions from the data selected at configuration time; that is refinement, working in production for years. What is missing is the layer that would make it general: dependencies declared as data rather than discovered by executing plugin code, a narrowing contract so refined definitions can be trusted, an alter step, schema emission, and integration with the host protocols real settings forms live inside. This proposal promotes those plugin-internal conventions into an inspectable, shared layer. Tool API also introduces automatic form generation and refineable data definitions, inspired by the above and Dynamic Data Properties in Active Pieces

The schema-driven half keeps getting rebuilt too. At least 3 contrib modules generate config forms from config schema (Schema Form, Schema Based Config Forms, Automatic Configuration Form), and core has an active issue proposing the same for simple config on top of #config_target (#3477363: Add a way to automatically generate basic config forms). The repeated reinvention is the demand signal, and the shared limitation is the point made above: config schema can carry the static shape, so these tools handle simple settings forms well and stop exactly where site-state-dependent options and cross-value constraints begin. Surfaces are what that whole family of tools is reaching for.

Where this pays off

Settings forms, config entity forms, plugin forms. The bulk of Drupal's admin UI is "collect a shape of values, validate, store." Generated forms eliminate the boilerplate, and more importantly they eliminate the drift: no more defaultConfiguration(), form defaults, and config schema each saying something slightly different. One declaration, everything derived.

Field API, retrofitted. Field API is actually two different gaps. On the content side, field items have real data definitions (property definitions, constraints), but they are static: nothing can express that a property's valid values depend on another value, so refinement is the missing piece. On the settings side there are no data definitions at all: field storage and instance settings are described by nothing more than the form that collects them and the config schema that stores them. As an example, an address field's configuration (which sub-fields are shown, which are required, per-country behavior, defaults) is currently constrained almost entirely inside Form API. The stored settings structure is very difficult to describe from outside, as anyone who has tried to expose it through an API can attest. A refineable surface for those settings can fully explain itself: which keys exist, what each accepts, how country selection narrows the rest. Just as importantly, it can return precise constraint violations when inputs do not line up, so the person or program on the other end can correct course instead of guessing.

Canvas. Canvas is what the gap looks like at scale. To let someone edit a component's props, it has to chain across the description languages: JSON Schema props are matched to prop shapes, mapped to field definitions, rendered through field widgets and Form API, then adapted for the client-side editor. Every hop in that chain exists because no single layer carries the full contract. A surface is that single layer: one description that a visual builder, a plain form, and the validation pipeline can all consume directly.

ECA and Rules. Both are in the business of configuring actions and conditions with values that depend on other values, and both have built their own partial answers. A shared surface layer is the general version of what each maintains locally.

Recipes and config actions. A recipe applying config input has no form to lean on; it needs to validate input programmatically and fail with useful errors. Surfaces provide that validation directly, with the same messages a human would have seen in the form.

Decoupled admin. If the surface owns definitions, validation, and the storage translation, Form API stops being the keeper of any of those and becomes what it is genuinely good at: rendering and presentation. A JS frontend can fetch the surface (as JSON Schema, see below), build its own UI, post values back, and receive the identical validation the Drupal-rendered form would apply. That is the piece previous decoupled-admin efforts have been missing. The hard problem was never rendering forms in JavaScript; it was that the contract never left PHP form classes.

AI, agents, MCP. An agent configuring Drupal needs exactly what a JS frontend needs: a machine-readable description of accepted values, and violation feedback it can act on. Surfaces provide both without building anything AI-specific. More on this below, because it matters to both sides of that conversation.

JSON Schema, both directions

Core recently gained JSON Schema emission (the json_schema serialization format). That work keeps running into the same wall: typed data cannot express things JSON Schema considers basic, including defaults, examples, dynamic-keyed maps, unions, and conditional narrowing. Issues are open or in progress for several of these (#3556242: Add support for patterns and additional properties to Map and MapDataDefinition for patternProperties and additionalProperties on maps, plus defaults and examples on data definitions).

Surfaces are the consumer that makes finishing that alignment worthwhile. A surface whose definitions carry the full vocabulary emits as an honest JSON Schema object, and simple refinements can even compile to dependentSchemas/if-then for static consumers, with site-state-dependent refinements resolved live. The reverse direction follows naturally: a JSON Schema document maps onto a surface, which means external schemas can be ingested into typed data and validated with Drupal's constraint system. Import a schema, get a working form and validation pipeline. That is a genuinely new capability, and it becomes inexpensive once the layer exists.

One pipeline, visible

Currently the journey from submitted values to stored config is private to each form: reshaping in submit handlers, further reshaping in plugin setters, and occasionally the silent dropping of keys nobody declared. Debugging "why didn't my setting save" means retracing that private path.

Surfaces give the pipeline names: values are accepted against the surface, validated against the refined surface, and prepared for storage at one point where the settings-to-config translation is explicit and alterable. That last step is also where a common third-party-settings strategy can finally live. Today every namespaced-extension mechanism (third_party_settings on config entities, ad hoc keys elsewhere) is bespoke per host. A surface with a standard mount point for third-party definitions makes "extend someone else's config, with validation, without a form_alter" a single pattern instead of a per-subsystem negotiation. The PoC's extras submodule demonstrates this end to end.

Because altering happens at the definition layer under the narrowing contract, we keep what form_alter actually provides, which is extensibility, while dropping what it costs us: silent constraint mismatches between what a form enforces and what the owner intended. An alter that tries to widen past the advertisement fails loudly instead of shipping a form that accepts values the storage layer will reject.

Humans in the loop

This is the framing I care most about, and it is aimed at both camps in the AI conversation.

If we build one set of tools for AI and a separate UI for humans, we have made everything worse: two contracts to keep in sync, and AI operating in a lane humans cannot observe. The alternative is to describe the shapes of data we want to capture once, and let every consumer (Form API, a JS admin UI, a recipe, an agent, MCP w/ tools) work through the same descriptions with the same validation and the same visible history.

For those enthusiastic about AI: this is how agents get truthful contracts and correctable errors instead of guesses about form structures.

For those skeptical of AI: this is how AI stays inspectable. Every surface an agent can touch is a surface a human can open as a form. Same constraints, same violations, same audit trail. A rule worth adopting from the start: do not create data surfaces that are not accessible to humans. The moment machine-only configuration paths exist, observability is gone. This architecture is how we avoid ever needing them.

And if AI disappeared tomorrow (or for those who want it to), everything in this proposal would still stand on its own: less form boilerplate, no schema/form/defaults drift, validation that exists outside browsers, decoupled admin finally viable, recipes that fail informatively, kernel-testable forms. The machine-readability that agents need is the same machine-readability every other item on that list needs. That is not a coincidence; it is what a good separation of concerns looks like.

Open questions

Permissions. Forms bake access into elements ad hoc (#access, conditional element building), which means access logic is as invisible to non-form consumers as validation is. Surfaces need a deliberate answer here: per-definition access metadata, or an access handler per surface following the entity/field access pattern. Whatever the shape, the win mirrors validation: one access decision that the generated form, the JSON representation, and any tool all enforce identically, instead of the current situation where API exposure has to re-derive (or forget) what the form was hiding.

Performance and caching. Surfaces are built at runtime, which is the point, but hosts build forms often. The PoC has not needed caching yet; a core version needs a real story for when surfaces are constructed, what is cacheable (the advertised surface probably is, refinements mostly are not), and how that interacts with form cache. A related lesson learned in the PoC: anything a surface holds must survive form-cache serialization, so refiners and builders need to be services or serializable objects, by contract.

Semantics typed data cannot express yet. Required versus nullable versus absent, unions, dynamic-keyed maps, and labeled choices. The last one deserves emphasis because both form generation and machine consumers need it: an allowed-values constraint should be able to carry a label (and ideally a description) per value, so a generated select renders "Disabled/Optional/Required" instead of 0/1/2 and an agent understands what each value does before choosing one. JSON Schema shares this weakness (enum is labelless; the community convention is a oneOf of const entries with title), which makes labeled choices both a gap to close and an emission pattern to standardize. Some of this has open issues; unions in particular are their own architecture conversation. Surfaces do not need all of it to be useful, but the proposal should be honest that the definition vocabulary needs to grow alongside.

Adoption path. This cannot be a big-bang replacement of Form API and should not try to be. The progressive-adoption result described above suggests the path: land the layer, adopt it opt-in from the edges (plugin settings forms first, where the pain is worst and the blast radius smallest), and let plugin types grow native support at their own pace. The open question is really about ordering and endorsement: which core plugin types convert first, and whether the shim pattern is blessed as the official on-ramp.

What an initiative would look like

1. Finish the typed-data vocabulary gaps that surfaces need and that JSON Schema parity justifies on its own (defaults, examples, map key patterns; issues exist or are drafted).
2. Land the surface layer: definitions, refinement map, refiners, the narrowing contract, and the build/alter event. No form integration yet; validation-only consumers (config actions, recipes) can already use it at this stage.
3. The form bridge: generated elements, AJAX/#states derived from refinement dependencies, path-aware violation mapping. Opt-in per plugin or form.
4. JSON Schema emission and ingestion for surfaces, building on the existing json_schema work.
5. Retrofits where the payoff is proven: field settings forms (address as the flagship), a settings-form pattern for config entity forms, and adapters for ECA, Rules, and Tool-style consumers.

The PoC and its kernel tests are available as a working reference for all of it, as evidence the model works rather than as the code that should land.

AI usage (if applicable)

• [x] AI Assisted Issue: This issue was generated with AI assistance, but was reviewed and refined by the creator.
• [ ] AI Assisted Code: This code was mainly generated by a human, with AI autocompleting or parts AI generated, but under full human supervision.