Allow forms to be defined in three segments: schema, UI, data

I can't read.

I totally love this separation. Its how all modern form system work, unlike Drupal which has merged everything together historically.

One question I ask myself though: Does this mean forms will be always defined in PHP? Where is the line drawn between forms in JS and in PHP.

I have a form factory I wrote in d7...that could be ported to d8 easily

This is how you build a form for a plain old php object.

use FormFactory\Annotations\Field\FieldType;
use FormFactory\Annotations\Field\Group;
use FormFactory\Annotations\Field\HiddenTitle;
use FormFactory\Annotations\Field\Options;
use FormFactory\Annotations\Field\Title;
use FormFactory\Annotations\Field\Weight;
use FormFactory\Annotations\Field\WithEmpty;

class Distance {

  /**
   * Distance unit.
   *
   * @var string
   * @Title("Units")
   * @Weight(2)
   * @HiddenTitle(TRUE)
   * @Group(
   *   class = "container-inline",
   *   group = "Distance from Nearest Cross Street or Landmark"
   * )
   * @Options("distanceUnits")
   * @FieldType("select")
   * @WithEmpty(TRUE)
   */
  protected $unit;

  // More properties as required.
}

then the fapi array is just

$form['distance'] = [
  '#type' => 'object',
  '#default_value' => $distance,
 ];

Where $distance is an object of class Distance.
@options supports both an array and a reference ID that looks up a list in an options provider (they are defined dynamically).

The element value and process callbacks take care of making $form['distance'] into a tree of fields, and then converting the value into an object - which relies on a normalizer/denormalizer for converting to and from and array/object.

I think it would allow us to take some steps along this path.

+1 to this idea, exciting stuff.

I experimented some with using a similar model in Contenta Angular, where the schemas exposed by Open API would automatically generate editorial forms for creating / editing nodes in Angular. It worked alright for simple forms, but of course choked on custom Drupal field types like entity reference. Deep integration with FAPI would solve that issue, and also expose all forms, not just entities.

With regards to #4, in my view this is an incremental step towards opening up possibilities down the line. By moving forward with this separation, field types and forms defined in Drupal's backend can reliably maintain their status as the canonical driver of UI's, without client side applications having to write code every time a field is added or modified.

Once this separation exists, down the line the responsibility for defining and configuring these field types and forms could theoretically be shifted to JS. And of course whenever it's desirable a client side application can still customize their forms and simply rely on posting back through Drupal's existing APIs (while assuming the burden of extra lift when data models change).

+1 for this idea as well. For whatever time I can spare, you have my sword.

I would like to propose the Form.io Vanilla JS renderer @ https://formio.github.io/formio.js. This will work in all frameworks so does not require React, but one could easily use https://github.com/formio/react-formio for a React.js wrapper.

A Form builder is also provided @ http://formio.github.io/ngFormBuilder

All libraries are MIT licensed, stand-alone, and do not require the hosted Form.io service to work.

Disclaimer: I work for Form.io, and myself and my team wrote these libraries.

Nice to see some movement here

1. +++ b/core/lib/Drupal/Core/Form/JsonSchemaFormBuilder.php
   @@ -0,0 +1,358 @@
   +      case 'textarea':
   +      case 'textfield':
   +      case 'token':
   +      case 'item':
   +      case 'date':
   +      case 'datelist':
   +      case 'datetime':
   +      case 'color':
   +      case 'email':
   +      case 'path':
   +      case 'url':
   +      case 'search':
   +      case 'tel':
   +      case 'machine_name':
   +      case 'markup':
   +      case 'password':
   +      case 'link':
   +      case 'submit':
   

   i think embedding knowledge of the element types here is not the right approach. I think we should add a new interface JsonSchemaEnabledFormElementInterface with the required methods, and then add to each Element plugin as required. If an element doesn't support it, its dropped from the built schema or we fall through to the exception. There are a wide range of Element plugins in contrib that we cannot know about in advance. If those elements don't support it, they'd fall through to the exception like you have here. tl;dr invert this - make the elements tell the builder about their features and not the other way

2. +++ b/core/lib/Drupal/Core/Form/JsonSchemaFormBuilder.php
   @@ -0,0 +1,358 @@
   +      if (!isset($element['#type']) || !in_array($element['#type'], ['checkboxes', 'radios'], TRUE)) {
   

   same here, we can't special case just those two - there might be others in contrib

3. +++ b/core/lib/Drupal/Core/Form/JsonSchemaFormBuilder.php
   @@ -0,0 +1,358 @@
   +      case 'link':
   +        $row['ui:widget'] = 'link';
   +        $row['ui:options']['label'] = FALSE;
   +        if (!empty($element['#url'])) {
   +          $row['ui:options']['title'] = (string) $element['#title'];
   +          $row['ui:options']['url'] = $element['#url']->toString();
   +        }
   +        break;
   +
   +      case 'checkboxes':
   +        $row['ui:widget'] = 'checkboxes';
   +        break;
   +
   +      case 'radios':
   +        $row['ui:widget'] = 'radio';
   +        break;
   +
   +      case 'radio':
   +      case 'checkbox':
   +        $row['ui:widget'] = $type;
   +        $row['ui:options']['label'] = FALSE;
   +        break;
   +
   

   same deal here

4. +++ b/core/lib/Drupal/Core/Form/JsonSchemaFormBuilder.php
   @@ -0,0 +1,358 @@
   +  protected function buildFormData(array $parent, array $result = []) {
   +    foreach (Element::children($parent) as $key) {
   +      $element = $parent[$key];
   +
   +      $value = isset($element['#value']) ? $element['#value'] : [];
   +
   +      if (!isset($element['#type']) || !in_array($element['#type'], ['checkboxes', 'radios'], TRUE)) {
   +        $value = $this->buildFormData($element, $value);
   +      }
   +
   +      // Retain 0 and FALSE, but ignore other empty values.
   +      if (!in_array($value, [NULL, [], ''], TRUE)) {
   +        $result[$key] = $value;
   +      }
   +    }
   

   in my opinion this should be separated out of this class into a separate service - i've done similar in d7 and it would be useful as a stand alone.

I totally agree that conceptually, even just on the php side, separating the data, schema and UI makes more than sense.

One question I have is: Do you plan to implement all the feature of FAPI, like multi-page forms, or would you rather say: There is a limited scope where using a non custom form written in JS makes sense.

Cool stuff.

I think the biggest challenge is validation. We'd need to be able to make all our validation logic work on the client side somehow. For some things that will be easy (min/max value, list of allowed values), but for lots of things, that will be very hard.

IOW, I completely agree asking this question is important:

or would you rather say: There is a limited scope where using a non custom form written in JS makes sense.

I think the biggest challenge is validation. We'd need to be able to make all our validation logic work on the client side somehow. For some things that will be easy (min/max value, list of allowed values), but for lots of things, that will be very hard.

a) I question whether the additional roundtrip for validation is a huge issue for generic forms. The time needed to validate could still be within a reasonable user experience when done serverside.
b) If we accept its a problem,. using formapi as base could be a bit of a problem, due to its insane flexibility. If we otherwis start with typed data, we would already have a way of validation. If you look at the structure of a constraint definition:

 *   constraints = {
 *     "ComplexData" = {
 *       "value" = {
 *         "Length" = {"max" = 12},
 *       }
 *    }

you could replicate that in JS. Mapping from the form elements to the data could be easier at the end of the day.

#15: agreed on both counts. Especially b) is the direction I've been thinking in: declarative constraints rather than code (i.e. a layer of indirection), which allows for equivalent JS implementations.

#16: you mentioning config forms here makes me ridiculously happy/excited, because #2300677: JSON:API POST/PATCH support for fully validatable config entities is blocked on:

1. #2907163: Split validation-related methods from FieldableEntityInterface into a separate ValidatableInterface
2. #1818574: Support config entities in typed data EntityAdapter
3. #2869792: [meta] Add constraints to all config entity types

to become a reality. And with @tim.plunkett helping out with that stuff, things could go much faster.

Separating these concerns will be essential to moving any form rendering to the client side.

Raising to Major for this.

It seems that this has not been mentioned in the issue, but we have a dedicated tool that generates JSON Schemas for entity types. Granted these will not be the only schemas we'll need to generate, we still want arbitrary front-end forms to be generated. However, I feel that schemas should describe data types, and forms should be generated from Schema + Form definition. See slide #46 in a presentation I did in DrupalCon Dublin for more context of what I mean.

The module that generates schemas is called Schemata and is used to generate automatic documentation for the API-First initiative via OpenAPI. Note that OpenAPI/Swagger generates forms for the API resources.

In any case, I'm not opposed to changing the way the API-First initiative recommends generating schemas, but I would really like to have a single and robust schema generation tool that can be used in different scenarios.

@e0ipso's comment is interesting.
It basically comes down to, due to treat Forms as the starting point or do we put our data as starting point aka. our entities. This issue would allow any form to be exposed, while something like schemata totally just works for entities, and is in that scope basically what I was talking partially about in #15

If we think it's a good direction we can put the data at the starting point even for non-entities. If a user is filling a form with the intention to upsert some data, then we should start by describing that data. Even if that is not an entity.

If the form has nothing to do with data in the server we can either have the form completely managed in React, or still send a pair of Schema + Form definition with an empty schema. I think I like the form managed in React better, because the submission may not need to interact with the server.

If we think it's a good direction we can put the data at the starting point even for non-entities. If a user is filling a form with the intention to upsert some data, then we should start by describing that data. Even if that is not an entity.

I tend to agree with this. At the end of the day, we are working with data. It's not necessarily as complex as entity data, but it's data nonetheless. I think a data-first model is appropriate.

The module that generates schemas is called Schemata

Looks like that module already bases its work on TypedData. Via code we already have in TypedConfigManagerInterface and Drupal\Core\Config\Schema, I think that means it would work for all our config data as well (both config entities and single configs). And for forms that upsert data that is neither content entities nor config, we could probably create TypedData objects to represent whatever their data is as well. Based on current experience with Schemata, have any problems surfaced with TypedData as the foundation on which to generate JSON schemas?

[…] we could probably create TypedData objects to represent whatever their data is as well

Yeah, I was thinking about the same thing. That may not be the most pleasant DX for writing a form, but I guess that we can work on DX at some point later.

Two month ago last comment for this issue.
Does it mean discussion closed?
What have been done in this direction?
Global codedesprint should bring some progress.

I'm writing following on from a short discussion in Slack about how we should replace all form element arrays with Objects, so instead of:

$form['text_field'] = [
  '#type' => 'textfield',
  '#title' => $this->t('Subject'),
  '#default_value' => $node->title,
  '#size' => 60,
  '#maxlength' => 128,
  '#required' => TRUE,
]

We could have:

$form->add('title')->(new TextField()
  ->title($this->t('Subject'))
  ->defaultValue($node->title)
  ->size(60)
  ->maxLength(128)
  ->isRequired() // could also accept a Boolean as an argument, this would allow for changing that later in a hook but makes the initial creation cleaner and would default to true if the method is called without any parameters
);

// A shorter version could also be created for those who just want the bare minimum and even cleaner code:
$form->add('title')->(new TextField($this->t('Subject'), $node->title));
// with the first argument being the title and the second being the default value, both of these arguments can be optional so one could simply just spefiy the title or as seen above only use the method-chainng approach

The two main advantages to this is that it aligns with the move to OO and for those like myself who use an IDE like PhpStorm it makes life easier by allowing the IDE to suggest the various properties that you can use to build the form.

So I think the first thing we need to do is agree that switching to OO would be the best starting point, possibly create a child issue to establish the standards for how OO Form Elements will be constructed.

Moving forward with that in mind we can look at the separation of Schema, UI and Data.

Fundamentally when it comes down to it, yes forms just handle data and yes we could do something with the headless/API side of stuff, but ultimately defining a form and the data it accepts is so intertwined with what the actual form element is and looks like, so there might be challenges to overcome there. You could abstract radio buttons, select lists, maybe even check boxes into some kind of "choice" class but we'd have to be very careful to make sure we don't end up breaking some form logic.

Going on that line of thinking you could break this "choice" class down into two classes: "Single Choice" and "Multi Choice". The "Single Choice" class could be displayed as a set of Radio Buttons or a Select List, but the key factor is it will only accept one selection. The "Multi Choice" class could be displayed as Check Boxes or a Select List (which accepts multiple selections), but the key factor is will accept more than one selection.

From a headless perspective that could work if implemented correctly in the API, but we should be careful not to implement something like this in a way that negatively impacts developers, the last thing I want is to have to write more arrays or hooks or create more instances of classes just to add a text box to a form.

To achieve this level of abstraction while still maintaining simplicity my OO example from earlier could be adapted to look like this:

$form->add('flavour_of_icecream')->(new SingleChoice()
  ->title($this->t('What flavour of Ice Cream would you like?'))
  ->isRequired()
  ->options(new OptionsSet()
    ->add('vanilla', $this->t('Vanilla'))
    ->add('chocolate', $this->t('Chocolate'))
    ->add('strawberry', $this->t('Strawberry'))
  )
  ->renderAs(Radios::class) // would cause the form element to render as Radio Buttons, equily you could specify SelectList::class to have it render as a select list, or if you changed the object decleration to be new MultiChoice() you could use ->renderAs(CheckBoxes::class) or SelectList::class to have it render differently
);

There are two key differences here:

1. The object that is defined (SingleChoice) is more generic and relates to the type of data that the form (or for that matter API) expects and defines what criteria the data being received validates against.
2. While the "->renderAs" method tells the Drupal Rendering Engine what HTML elements are to be rendered on the front end, in this case the HTML would be Radio Buttons (<input type="radio">). In fact if you were building a truly headless site in theory you could just omit the "->renderAs" call and the form would still work, it wouldn't render any HTML for that element on a page, but it would still work fine by submitting data over an API using JSON, XML or something else.

The advantages to this approach are:

• You abstract the data type from the way the element is rendered, and allow a theme or another module to change the way elements are rendered using hooks later on. Want my "flavour_of_icecream" element to render as a Select List instead of Radio Buttons in your theme? No problem just change it using a hook such as hook_form_alter().
• You maintain the benefits describe above of moving to OO based forms.
• The effort required to add form elements is no different, and code is still clean, yet allows for a much more powerful approach to the way forms and data are handled.
• You can abstract the rendering of form elements and pass that off to React or another JavaScript Framework, meaning the developer can focus on building clean code and does not have to worry about the process of rendering the element or which JS Framework is in use, if any. Remember Drupal does still plan to support multiple JS Frameworks on the front end even though the Admin Interface is standardising on React.
• From an API perspective this allows the form and validation logic to still function without having to involve rendering.

Some other examples of this:

$form->add('icecream_scale')->(new Range()
  ->title($this->t('How much do you like ice cream (0 being not at all, 10 being very much)'))
  ->minValue(0)
  ->maxValue(10)
  ->renderAs(Slider::class)
);

$form->add('icecream_username')->(new TextInput()
  ->title($this->t('Enter a username to store your ice cream'))
  ->maxLength(32)
  ->renderAs(Text::class)
);

$form->add('icecream_password')->(new TextInput()
  ->title($this->t('Enter a password to secure your ice cream'))
  ->renderAs(Password::class)
);

Notice I did not include a ->size() (the equivalent of '#size') this is because we should simply leave that up to the theme to define in CSS. To be honest the HTML "size" property should really just bee removed, not sure why it's still around, but that's a conversation for another day and place.

I'll leave it at those examples but you get the idea.

In summary:

• Abstraction and separation is good
• But we shouldn't make code more complex as a side effect of doing so
• Adopting this approach will achieve both of these things
• Forms will be in favour of ice cream going forward :P

Thanks
Aaron

@AaronMcHale well written.
We also have a very good entity api in d8 core with almost completely separated layers. Each entity/field has data storage(Data), form(Schema) and display(UI). I wonder if we can leverage that and extend it to all forms. I'm talking about fields without storage and ability to create a form using only OO like what explained by @AaronMcHale. I think it would be much easier to implement and lots of code can be reused. I think there has been a discussion about using field api in form, i haven't been able to find it yet.

@AaronMcHale well written.
We also have a very good entity api in d8 core with almost completely separated layers. Each entity/field has data storage(Data), form(Schema) and display(UI). I wonder if we can leverage that and extend it to all forms. I'm talking about fields without storage and ability to create a form using only OO like what explained by @AaronMcHale. I think it would be much easier to implement and lots of code can be reused. I think there has been a discussion about using field api in form, i haven't been able to find it yet.

Interesting idea, I like the idea of being able to reuse code between the Entity/Field and Form APIs, we'd need to first figure out if that would be feasible and what challenges would need to be overcome. As you pointed to, and for me ultimately my main concern is that anything like that doesn't create more work for developers like ourselves when it comes to simply creating or modifying forms, so we should keep that in mind.

What I suspect we might find or at least the recommendation would be that the base code goes into the Form API and the Field/Entity APIs just builds on top of the Form API. To me that also makes the most sense since the Filed API is basically just dealing with forms but providing a more managed approach and a UI.

I've also been thinking about how we'd go about implementing something like this, as forms are probably one of the most widely used parts of Drupal so changing their structure presents two possibilities: either the work would need to be done in a major release like 9.x, or we'd need to find a way to maintain the current system while introducing this new one, then deprecating the old system.

Personally I'm in favour of the latter option as it means we can stay aligned to the philosophy of a constantly evolving Drupal where upgrading between versions is seamless for developers and site builders.

To achieve the latter option though we may also want to do it in tandem with introducing a new OO based Hooks or Events system. So if you're using the old hooks you use the old Form Arrays and if you're using the new OO Hooks/Events you use the new OO Forms. Then we just link the two together on the backend and have form arrays converted to objects as and when the system finds them.

I think this approach would work well and would be the least disruptive to current development and Drupal sites, since it allows for a period of time for people to migrate. If we did want to push for this we'd need to spin off some new issues as this is beginning to look like a rather large project.

we'd need to find a way to maintain the current system while introducing this new one, then deprecating the old system

Yep, the new system can be added in any Drupal 9 minor. The old system can be deprecated once all of core is converted to using the new system. And then the old system can be removed from the following major version (e.g., Drupal 10 if all this is done by one or two minors prior to Drupal 10's release).

Related: #2949888: Enhance config schema for richer default experiences.

Re #16 (I only just noticed this issue). One thing I've been doing for a number of years, inspired by Config Inspector indeed is use the schema label for the config form element title AND, where available after a delimiter split, for the config form element description.

It would be nice if schema could be extended to support description separately from label, for this kind of autogeneration of edit forms.

RE #14:

I think the biggest challenge is validation. We'd need to be able to make all our validation logic work on the client side somehow.

I'm about 6 years late to reply, but I've been a maintainer of the Clientside Validation module. Admittedly, I haven't been active on that project for quite some time, others have taken over, but it might me a good starting point/inspiration for the validation of forms on the JS side.