Problem/Motivation

Good data contracts let one application act on another’s answer without guessing what it means. Two systems can exchange perfectly valid JSON and still make the wrong business decision.

One of Drupal's problems is that a required value answers three separate questions: is the data present, may it be null, and may it be empty.

JSON Schema keeps all three apart (required for presence, "null" in type for nullability, minItems/minProperties for emptiness).

Drupal's typed data answers all three with a single isRequired():

  • TypedDataManager::getDefaultConstraints() adds NotNull to every required definition, so a required value can never be NULL.
  • NotNullConstraintValidator converts an empty list or map to NULL, so a required list or map rejects [] with "This value should not be null." Only config schema's mappings and sequences (ArrayElement) are exempt.
  • Typed data has no method that returns whether a value is NULL: a list's getValue() returns [] for both NULL and [].

This matters more and more, both internally and externally: the serialization module publishes JSON Schema for typed data, single-directory components define their props using JSON Schema, Tool API (https://www.drupal.org/project/tool) validates tool inputs and outputs against it, ECA/FlowDrop/Maestro rely on JSON Schema, OpenAPI module generates incorrect schema, and more

When typed data validation and the published JSON schema disagree, a client can't trust the schema and it makes interacting with Drupal more difficult.

The coercion dates to #2343035: Upgrade validator integration for Symfony versions 2.5+, and it is the semantic Field API wants: a required field means "at least one item". But it leaks onto every other typed data consumer, and core has already acknowledged that once: #3364109: Configuration schema & required values: add test coverage for `nullable: true` validation support added a hardcoded exemption for config schema's ArrayElement, with the comment "TRICKY: ... configuration schema sees a substantial difference between an empty sequence/mapping and NULL." That fixed one subsystem by class check and left the conflation in place for everything else.

Related: #3347033: NotNullConstraintValidator doesn't catch empty strings, only list/complex data types documents the mirror-image inconsistency (empty strings are NOT coerced). One interaction to account for: on maps, the coercion currently doubles as a stopgap for #3256536: Map data definition properties are not validated if root value is empty - an empty map's required properties are never visited by RecursiveContextualValidator, so the NotNull rejection of [] is the only thing catching a required property missing from an empty required map. Any consumer switching the option off for maps needs #3256536: Map data definition properties are not validated if root value is empty fixed or its own property walk; the Tool API workaround was scoped to lists for exactly this reason.

Steps to reproduce

$definition = \Drupal::typedDataManager()->createDataDefinition('map')->setRequired(TRUE);
$violations = \Drupal::typedDataManager()->create($definition, [])->validate();
// One violation: "This value should not be null." - for a present, empty map.

Proposed resolution

The MR is fairly large at around 850 lines. However, less than 100 lines are production code. The rest is tests and documentation. We built up a very robust set of tests that should prevent regressions in the future. Important for anything that touches typed data.

As stated in comment #22:

I think it's really important that TypedData and JSON Schema agree, so Drupal can be compatible with external tools and other platforms. [...] JSON Schema gives us a shared reference point or shared contract. It provides direction, allows us to focus on the details we actually need to decide, and avoids us having to bikeshed nuances.

So as a principle, we follow JSON Schema's meaning of "required" and "nullable", and we did so in four parts:

  1. Required and nullable are separate. New: DataDefinitionInterface::isNullable() and DataDefinition::setNullable(). By default, a value is nullable exactly when it isn't required, as before.
  2. Typed data knows whether it is NULL. New: TypedDataInterface::isNull(), which returns FALSE for '' and []. NotNull calls it.
  3. For a map property, required means present. NotNull rejects a required property that is absent, and a property that isn't nullable when it is NULL. So a required, nullable property may be NULL but not absent.
  4. A list got a "null mode", proposed in #20. NullMode::Loose, the default, keeps today's rule, which treats an empty list as NULL, so it also rejects [] for an optional list that isn't nullable. NullMode::Strict, set with setNullMode(), treats NULL and [] as different values: a required list accepts [], and Count with a minimum of 1 requires an item.

Entity fields are always Loose: storage has no rows for an empty field, so a field given [] loads the same as one never given a value. Maps have no mode yet, because a map created with NULL has [] as its value (see #3256536).

The tests for required and nullable data, TypedDataTest::testRequiredNullable*(), don't hardcode expected results: each case asks JSON Schema's validator whether the data is valid, and asserts that typed data validation agrees.

Each commit makes one change, with its tests, so it can be reviewed commit by commit:

  1. isNullable() and setNullable(): https://git.drupalcode.org/issue/drupal-3625989/-/commit/5637b60dd857d6c...
  2. isNull(): https://git.drupalcode.org/issue/drupal-3625989/-/commit/b908c84c73df295...
  3. NotNull calls isNull(), with a deprecation that commit 7 removes: https://git.drupalcode.org/issue/drupal-3625989/-/commit/dc35a9be63f5e43...
  4. Reworded comments: https://git.drupalcode.org/issue/drupal-3625989/-/commit/b554b49d22020b0...
  5. Required and nullable map properties: https://git.drupalcode.org/issue/drupal-3625989/-/commit/10228dcd293ce9e...
  6. Map properties that are maps: https://git.drupalcode.org/issue/drupal-3625989/-/commit/a9bda492e962c0e...
  7. The null mode: https://git.drupalcode.org/issue/drupal-3625989/-/commit/157570e64349bf3...

Remaining tasks

  • Reviews.
  • Publish the change record.

Follow-ups: try moving entity fields from Loose mode to Strict mode.

User interface changes

None.

API changes

  • New: isNullable() and getNullMode() on DataDefinitionInterface, setNullable() on DataDefinition, setNullMode() on ListDataDefinition, isNull() on TypedDataInterface, and the NullMode enum.
  • A class that implements those interfaces without extending core's base classes has to add the new methods.
  • A subclass with its own untyped isNullable(), like component_schema's, needs : bool, or PHP stops with a fatal error.
  • New: Map::isAbsent(). Map::getValue() leaves out the key of an absent property that is required but nullable, or optional but not nullable.
  • Existing definitions validate as before, except that NotBlank now accepts NULL wherever NotNull also applies, so NULL gets one violation instead of two. Otherwise only a definition that calls setNullable() or setNullMode(), or overrides isNullable(), like Tool API's output definitions, changes behaviour.

Data model changes

None.

AI was used to assist in drafting this issue.

Issue fork drupal-3625989

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

michaellander created an issue. See original summary.

michaellander’s picture

Issue summary: View changes
michaellander’s picture

Issue summary: View changes
michaellander’s picture

michaellander’s picture

A note on the part this issue deliberately leaves out. The summary scopes presence versus nullability out as a separate conversation, since data definitions have no nullability concept. If the option here is agreed, that becomes the natural next step, and it would need its own issue:

  • DataDefinition::isNullable() / setNullable(), mirroring config schema's nullable, which config already keeps apart from presence (requiredKey).
  • TypedDataManager::getDefaultConstraints() adds NotNull only when a definition is required and not nullable.
  • Core's JSON Schema normalizers publish "null" in type from nullability instead of from required, so presence (the parent's required list), nullability ("null" in type) and emptiness (minItems / minProperties) each map to their own keyword.

With both in place, required means present, nullable says whether null is allowed, and "not empty" is an explicit constraint: the three questions from the summary, each answered separately.

One refinement to the long-term direction: for Field API the explicit requirement would need to be a dedicated constraint rather than Count, since a field item list can hold empty items that Count would still count. Plain typed data can use Symfony's NotBlank or Count.

Tool API is working through the same split on its side: tool#3583069 (a required map may be empty; Tool API walks declared map properties itself, which covers #3256536 for its own validation) and tool#3583043 (a nullable flag). A second real case since this was filed: canvas_tools' canvas_get_component_schema failed for every component without slots, because its required, empty slots map hit this coercion (canvas_tools#26).

AI-assisted comment.

michaellander’s picture

Issue summary: View changes
mglaman’s picture

Issue tags: +rosetta
dries’s picture

I've been digging into this as it feels foundational for Rosetta, and while I'm still coming up to speed, I'd like to propose a shorter route to the same destination.

The issue summary gets there in two stages: first a treatEmptyAsNull option on NotNull, then, as michaellander describes in #5, nullability on the definition.

I'd like to try and skip the option and build #5's nullability in this issue. I'm also happy to take a first crack at it.

Why skip the option and go straight to step 2? Because we need the full set of capabilities, and I personally think we should have a sense of urgency around it.

Say a map describes the input for updating a node's tags, with a required tags list. A Drupal developer would write something like this:

$definition->setPropertyDefinition('tags', ListDataDefinition::create('string')->setRequired(TRUE));

Leaving tags out should be an error, and sending "tags": [] should mean "remove all tags":

The caller Today Step 1 only Step 2
Leaves tags out "This value should not be null." Passes "This field is missing."
Sends "tags": [] "This value should not be null." Passes Passes


ItemList::setValue() stores NULL as [], so a missing list looks exactly like an empty one. So during step 1, turning the option off on a required list also turns off the check that the key is there. It likely adds intermediate complications.

Second, the summary calls the option "step one of a correction, not a permanent flag". It gets added, its default flipped in a major, and then removed, so code needs to be updated twice. Going straight to nullability might actually be less work.

Two additional thoughts. First, both routes probably need #3256536: Map data definition properties are not validated if root value is empty to land. Second, this would also mean we might want to resolve #3347033: NotNullConstraintValidator doesn't catch empty strings, only list/complex data types differently. I'll explain that in a comment #4 over there.

michaellander’s picture

Agreed: skip the option and build nullability directly. That's where we eventually landed on Tool API and I never went back and updated this issue accordingly. Tool API has run on that model since beta10 and can serve as the reference: a required list accepts [] with the minimum from Count (tool#3583040), a required map is present and may be empty (tool#3583069), and outputs carry nullable apart from required (tool#3583043).

One dependency and two additions for the rewrite:

  • Presence has to be checked by the parent. ItemList::setValue() stores NULL as [], so a list's own NotNull cannot tell "left out" from "empty". That is #3256536: Map data definition properties are not validated if root value is empty in fago's form: skip complex data only when its value is NULL, validate an empty one.
  • Field API keeps "at least one item" through an explicit constraint on FieldItemList in the same change, or dropping the coercion changes what every required field accepts.
  • Config schema's nullable: true maps onto the new flag, so the ArrayElement carve-out goes away.

Agreed on #3347033: NotNullConstraintValidator doesn't catch empty strings, only list/complex data types too. I'll rewrite the summary accordingly unless anyone objects.

AI-assisted comment.

fago’s picture

indeed, typed data conflates the two things isNullable() and isRequired() in one, since it does not make any difference on the runtime/in-memory level when operating with $typed_data objects - it allows chaining to any child property even if not yet existing, so an absent value and a NULL value becomes the same.

However, I agree it makes sense on schema level and thus the schema/data-definitions should respect it + as we see it matters for validation.

It's a quite foundational change though, so thinking through all the mentioned cases together for the proposed solution makes sense. What do you see needed to make the use-case work, would we have to change NotNull as well?

I must say having NotNull + treatEmptyAsNull seems like band-aid to me, if the constraint says "not-null" it would be good if it just mean that. But that seems to be hard with the pre-existing usage isRequired validation. Any ideas on the best way forward here?

dries’s picture

Assigned: Unassigned » dries
Status: Active » Needs work

I'm going to try and work through this in a series of commits, one step at a time, likely over multiple days.

I just made my first commit. It basically pulls apart the two things "required" means today: (1) a value has to be given and (2) it may not be NULL. To achieve (2), data definitions just got isNullable() and setNullable(), as @michaellander proposed in #5.

Without a setNullable() call, isNullable() returns !isRequired(), so core's data definitions behave exactly as before.

The flags combine like this:

Definition Means isRequired() isNullable() Gets NotNull
Default Optional FALSE TRUE No
setRequired(TRUE) Required, not NULL TRUE FALSE Yes
setRequired(TRUE)->setNullable(TRUE) Required, but may be NULL TRUE TRUE No, which is new
setNullable(FALSE) Optional, but not NULL when given FALSE FALSE No, because validation can't tell a value that's left out from NULL


The last two rows are basically new. In the third row, validation accepts NULL, and the serialization module's JSON Schema output allows null. In the fourth row, validation is the same as for the default but the JSON Schema changes: it no longer allows null.

@fago, re #10: yes, NotNull has to change too, and no treatEmptyAsNull is needed. My plan for my second commit is to give ItemList and Map an isNull() so we can distinguish between "never given a value" and "empty list" ([]).

Disclaimer: I built this with the help of an LLM but reviewed everything extensively.

dries’s picture

Ok, the first commit basically added isNullable() and setNullable() to data definitions without breaking anything in core. That is something ...

I just pushed my second commit. It adds isNull() to typed data so we can tell NULL apart from '' and [].

Until now, that wasn't possible for lists: a list set to NULL and a list set to [] both return [] from getValue(). The second commit also fixes Map::getValue(), which turned a NULL map into an array once one of its list properties was read.

Before this commit:

$list = $typed_data_manager->create(ListDataDefinition::create('string'));

$list->setValue(NULL);
$list->getValue();  // []
$list->isEmpty();   // TRUE

$list->setValue([]);
$list->getValue();  // []
$list->isEmpty();   // TRUE

After this commit:

$list->setValue(NULL);
$list->isNull();    // TRUE

$list->setValue([]);
$list->isNull();    // FALSE

This work is important for typed data that is validated against a schema, rather than stored in entity fields. This MR does not change the behavior of entity fields, which takes a lot of risk out of this MR.

Currently, Field API doesn't track whether a field was set: a field set to [] and a field that was never set are the same. So a required field still needs at least one item whose isEmpty() returns FALSE. This code doesn't change that, but could enable more nuanced behavior in follow-up work, if we wanted to.

The MR is important for contrib, though. Modules like Tool API work around these core bugs/limitations today. It's important that core helps specify its inputs and outputs correctly.

On its own, this second commit is low risk. Nothing in core calls isNull(), except Map::getValue(), and the only behavior that changes is that a NULL map stays NULL (small bugfix).

To my own surprise, the two commits are pretty much independent. The third commit will bring them together. With the first commit, NotNull is only added to data that is required and not nullable, and in the third commit, NotNull will check isNull() from the second commit.

If anything looks wrong so far, let me know.

Disclaimer: I built this with the help of an LLM but reviewed everything extensively.

dries’s picture

Alright, I pushed the third commit. As mentioned, this is where the first two commits come together in validation.

Building on the example from #13, this is what validate() returns with this commit:

$definition = ListDataDefinition::create('string')->setRequired(TRUE);
$list = $typed_data_manager->create($definition);

$list->setValue(NULL);
$list->validate(); // "This value should not be null."

$list->setValue([]);
$list->validate(); // "This value should not be null.", plus a deprecation:
// Relying on NotNull to reject a required list given [] is deprecated in
// drupal:12.1.0 and is removed from drupal:13.0.0. Add a Count constraint with
// a minimum of 1 to '' to keep rejecting it.

$list->setValue(['a']);
$list->validate(); // No violations.

For users, nothing changes. Developers, however, get a deprecation message that explains how to continue rejecting [] in Drupal 13. A definition that already rejects [] with Count(min: 1) or NotBlank gets no deprecation warning.

In Drupal 13, with the deprecated code removed, we'd get:

$list->setValue(NULL);    // 1 violation: "This value should not be null."
$list->setValue([]);      // 0 violations
$list->setValue([NULL]);  // 1 violation: "This value should not be null."
$list->setValue(['a']);   // 0 violations

In other words, the corrected behaviour doesn't land until Drupal 13, when the deprecated branch is removed. It feels far away but I believe that is how it's done? I'd love some guidance on that.

Overview:

Case Today This step, Drupal 12 Drupal 13
No value (NULL, or left out of its map) rejected rejected rejected
[], without Count(min: 1) or NotBlank rejected rejected, with the deprecation accepted
[], with Count(min: 1) or NotBlank rejected, 2 messages rejected by that constraint alone rejected by that constraint alone
[NULL] rejected rejected rejected
A map given [] rejected rejected rejected, until #3256536
A list with items accepted accepted accepted
An entity field with no item that isn't empty rejected rejected rejected
A config sequence or mapping given [] accepted accepted accepted


The new NotNullConstraintValidatorTest covers the Drupal 12 column for lists and maps. Besides testing the changes in this MR, it also fills a gap in existing core tests.

The behavior matches how Tool API already works, which @michaellander pointed to as the reference in #9: a required list accepts [], and Count enforces a minimum.

Perhaps most important, I checked, and this also matches what JSON Schema recommends.

The docblock of DataDefinitionInterface::isRequired() documents the recommended approach with the following example:

$definition = ListDataDefinition::create('string')
  ->setRequired(TRUE)
  ->addConstraint('Count', ['min' => 1]);

So once this lands, Tool API and other contrib modules should be able to clean up their workarounds.

And more broadly, core's internal data definitions will be able to specify what JSON Schema already can: whether a value is required, whether it may be NULL, and how many items it needs.

Entity fields still aren't affected. As mentioned, that should make this MR fairly risk-free.

@fago, @michaellander and others: could you review the three commits?

Disclaimer: I built this with the help of an LLM but reviewed everything extensively.

dries’s picture

Status: Needs work » Needs review

The tests pass, so this is a good point to pause for review. The diff is somewhat large, but much of it is tests. The three commits, each with their own comment, should hopefully make it easier to review in stages.

michaellander’s picture

Status: Needs review » Needs work

This is great! I reviewed the three commits, ran the suites and probed the cases before and after. Behavior matches #12-#14. Two things I'd fix:

  1. Optional data with setNullable(FALSE) gets no NotNull, so NULL still validates. But the normalizers now read isNullable() and drop the null branch. The "optional, may not be NULL" row in TypedDataDefinitionTest locks that in. Either isNullable() returns TRUE whenever the data is optional, or nullable FALSE adds NotNull.
  2. The omitted-key test in TypedDataTest passes with or without setNullable(). An empty map never recurses, so a required non-nullable property also gives zero violations. It needs a non-empty sibling key.

Nit: rejectsEmptyList() only sees the definition's top-level constraints. A Count nested in Sequentially, or passed to validate() beside an explicit NotNull, still gets the deprecation and a duplicate violation on []. The deprecation should link a change record. setNullable() on field definitions is silently ignored.

This matches what Tool API does for lists, so we can drop that workaround once it lands. Maps still wait on #3256536.

borisson_’s picture

I just want to point at #3560443: [Policy, no patch] Stop enforcing NULL for text config properties that do not need to differentiate from empty string, since that's also about null/blank values and is probably related to this?

fago’s picture

Thank you for the updates and the work, the comments summarize the changes and problem statement well.

I generally think the differentiation between NULL And empty is good and important, but I have concerns whether this is the right way forward.

The concerning issues I see here:

1. API-additions must work cleanly on entity fields

The MR does implement the new method isNull() for field-item-lists with isEmpty(). This is important for BC, but as the comment describes it, the result is: a field set to [] is NULL. This is exactly what we are trying to work out here to be not true. But we cannot make this work without breaking isNull() or the pre-existing NotNull constraint.

Another example is this comment
# A list given [] is not NULL, while an entity field without a non-empty item is.

That basically documents that the entity field stops implementing the (updated) contract of typed data correctly. Typed data may not special-case entity fields, the API it implements must be valid and implemented by entity fields also, else the abstraction layer becomes worthless.

2. No forward-adoption of API-change is possible
With an API deprecation you can implement the new API before the old one goes away. Here the behaviour simply flips between D12 and D13: a required list given [] is rejected, then accepted. A module cannot opt into the new verdict early; all it can do is pin the old one with Count(min: 1).

Meanwhile, the API for modules becomes inconsistent. For D12 isNull() already says [] is not NULL, while NotNull still reads [] as NULL. So you cannot rely on getValue() === [] to be validated consistently with NotNull if you want to support APIs at both D12 and D13 at the same time. That is worse than the status quo, where [] at least means one thing.

Whenever we change what [] versus NULL means in Typed Data, we change the contract for every existing consumer at once. There is no gradual path, because a definition cannot say which meaning it uses.

My conclusion
Unfortunately Typed data has not been built with the separation of NULL, required and empty. I think changing that is quite a foundational, which unfortunately we cannot do cleanly with BC - since the API can have only one behavior.

This is why I think in this case gradually evolving APIs is not the right approach, if we want to fix foundational things like this. Instead, I think the right approach would be to come-up with a better replacement API and building a bridge from typed data. The replacement API can be more expressive and be capable of things like NULL and empty separation, we just need to map typed-data definitions cleanly to it. Then, we can start adopting the replacement-API for new things, and gradually replace typed-data definitions with it. We could start extending our APIs to support both, then start deprecating the acceptance of typed-data-defintions.

dries’s picture

@michaellander, I compared all four combinations of required and nullable against what JSON Schema does. I think both of your suggestions would move core further away from JSON Schema so I think it warrants more discussion.

The comparison shows other problems with the merge request in this issue, which I might come back to after fago's #18. (As an aside, I also wondered how the comparison below fares against Tool API.)

To make it easier to grok, I used an example of a person with a required first_name and an optional middle_name.

Each row in the tables below declares middle_name differently: required or optional, and nullable or not. The person is given ['first_name' => 'Ada'] (column 2) to leave the middle name out, or ['first_name' => 'Ada', 'middle_name' => NULL] to give it as NULL (column 3).

1. JSON Schema

Checked with justinrainbow/json-schema, the validator core's tests use.

middle_name is Key left out Given as null
required, not nullable rejected rejected
required, nullable rejected allowed
optional, nullable allowed allowed
optional, not nullable allowed rejected

2. Core's validation with this merge request

On main, only the first and third rows exist, because setNullable() is new.

middle_name is Key left out Given as NULL
required, not nullable rejected rejected
required, nullable allowed allowed
optional, nullable allowed allowed
optional, not nullable allowed allowed

This is appears to be a bug in the MR! Both differences seem to have the same cause: typed data stores a value that was left out as NULL.

3. Core's outputs with this merge request

For a person without a middle name, core always outputs {"first_name":"Ada","middle_name":null}.

middle_name is Schema allows null? Does the output fit?
required, not nullable no no, but the data fails validation anyway
required, nullable yes yes
optional, nullable yes yes
optional, not nullable no no

The last row is your finding. Its schema matches table 1: JSON Schema rejects null for that combination too. What doesn't match is the output, which writes "middle_name": null where JSON Schema expects the key to be left out.

dries’s picture

@fago, you raise some big questions. Thank you! It's what we need.

I agree with both of your principles: (1) typed data must not special-case entity fields (at least long-term), and (2) changing what [] and NULL mean shouldn't change the contract for every consumer at once.

That said, I'm not sure about "There is no gradual path, because a definition cannot say which meaning it uses.".

What if a definition could? It could declare a mode. Something like this:

ListDataDefinition::create('string')
  ->setRequired(TRUE);
  ->setNullMode(NullMode::Loose); // Default, if not specified

ListDataDefinition::create('string')
  ->setRequired(TRUE)
  ->setNullMode(NullMode::Strict);

// Any code can ask which meaning a definition uses.
$definition->getNullMode();

The loose mode is today's, where empty and missing are the same as NULL.

The strict mode keeps NULL, empty and missing seperate, as JSON Schema does.

Entity fields would start in loose mode.

The question then becomes: what would fields need to do in order to move to strict mode over time, so loose mode can eventually be deprecated?

If that works, it could also be the bridge to a potential replacement API: definitions in strict mode would map onto it one to one. But it might also open up opportunities for a more evolutionary approach that moves field types one at a time, assuming that is possible.

f0ns’s picture

I also agree with the two principles that @Fago described above.

The solution that @dries proposes looks elegant. I can't think of a better way to do it myself and I like the idea that it opens the door for a more gradual approach if that would be needed.

Interesting stuff!

dries’s picture

My last commit fixes the two bugs from #19, where typed data and JSON Schema disagreed:

  • A property that is required and nullable is now rejected when it is absent. Before, it was allowed.
  • A property that is optional and not nullable is now rejected when it is NULL. Before, it was allowed.

I added a new test, TypedDataTest::testRequiredNullable(), that takes #19's person with a middle name and runs every combination of required and nullable, with the middle name absent, NULL and a string. For each row it compares the result with Core's JSON Schema validator's results.

I think it's really important that TypedData and JSON Schema agree, so Drupal can be compatible with external tools and other platforms. This new test will help enforce that.

I worked on this one commit for more than 3 hours, and it makes my head hurt. If you feel the same, you're not alone. Aligning with JSON Schema helps with this: JSON Schema gives us a shared reference point or shared contract. It provides direction, allows us to focus on the details we actually need to decide, and avoids us having to bikeshed nuances.

For this change to work, typed data needed a way to say that a property is absent. It now uses the same rule as JSON Schema: a property is absent when the map's values have no key for its name. I introduced a new Map::isAbsent() for that.

Since NotNull is now added to more data, NotBlank accepts NULL whenever NotNull is there, as #3404061: When setting `NotNull` constraint on type: required_label, two validation errors appear decided, so NULL causes one violation, not two.

Disclaimer: I built this with the help of an LLM but reviewed everything extensively.

mglaman’s picture

+1 to the null mode in #19. It answers fago's "a definition cannot say which meaning it uses" in #17 directly, without building a second definition API.

The mode does not pass down to children

Each definition declares its own mode. A strict map can have a loose list property, and the list follows its own rules. If a property looked up its parent's mode, the same definition would validate differently depending on where it sits.

Fields can move to strict

On the question at the end of #19: fields fit strict mode better than they look. An entity always has every field for its bundle, so a field is never missing. An empty field is stored as zero rows, so storage can't record NULL either.

What makes required fields reject [] today is NotNullConstraintValidator, which turns an empty list into NULL before it checks. So "at least one non-empty item" lives in the validator, not in the field.

That also narrows fago's first point in #17. Core has no isNull() today. The MR added FieldItemList::isNull() returning isEmpty() to keep fields passing, and that is the part that breaks the contract. With fields in strict mode, a field item list is never NULL:

public function isNull(): bool {
  return FALSE;
}

The "at least one non-empty item" rule then becomes its own constraint on required fields, as michaellander suggested in #5 and #9. It can't be Count, because Count also counts empty items like [NULL].

I'd call it NotEmpty: it fails when isEmpty() returns TRUE, for any ListInterface or ComplexDataInterface, not only fields. That is exactly the rule loose mode applies today, so every loose definition gets a strict equivalent:

// Loose, today.
ListDataDefinition::create('string')->setRequired(TRUE);

// Strict, same result.
ListDataDefinition::create('string')
  ->setRequired(TRUE)
  ->setNullMode(NullMode::Strict)
  ->addConstraint('NotEmpty');

That makes deprecating loose mode a one-line migration.

Proposed sequence

  1. Add NullMode with NullMode::Loose as the default. NotNullConstraintValidator applies today's empty-to-NULL rule to loose definitions only, so existing code sees no change.
  2. Add isNull() so strict definitions can tell NULL from [], and mark config schema's Mapping and Sequence strict. That replaces the instanceof ArrayElement check. Strict maps need #3256536 so the properties of an empty map get validated.
  3. Mark field definitions strict and add NotEmpty to required fields, in the same change. Required fields reject exactly what they reject today. The violation moves from NotNull to NotEmpty, so fields pass the old message to keep REST and JSON:API error responses unchanged: 'NotEmpty' => ['message' => 'This value should not be null.']. A new issue can move fields to the "This value should not be empty." default.
  4. Deprecate NullMode::Loose in a later major and make strict the default. The deprecation message tells developers to add NotEmpty.

Two things to settle in this issue before more code:

  1. @fago: would a strict mode that maps one to one onto the replacement API you described count as the bridge? If so, the two proposals fit together, and this sequence is the first part of it.
  2. Rewrite the issue summary around NullMode and this sequence. It still proposes treatEmptyAsNull, which everyone has moved past.

Generated with the help of an LLM.

dries’s picture

@michaellander and the Tool API team, while working on #22, I briefly looked at Tool API. I ran Tool API's tests against the merge request, and my latest commit doesn't seem to break anything. It might not be as simple. But I think this basically makes core match the design Tool API already chose in https://git.drupalcode.org/project/tool/-/work_items/3583043. Does the commit give Tool API what #3583043 needed from core?

dries’s picture

The commit in #22 made typed data validation follow JSON Schema's, and fixed the two cases that #19's table showed as wrong.

However, its tests only covered string properties. For a property that is a map, such as an address with a city, one of those two cases was still wrong.

So my last commit fixes that in Map::get(): it calls isAbsent() before creating the property's object, and sets an absent property's object to NULL.

It also adds tests for map properties. The tests don't hardcode expected results: as proposed in #22, each test case asks JSON Schema's validator whether the data is valid against the schema, and asserts that typed data validation returns the same result.

Much of what we're doing is building up a very robust set of tests that should prevent regressions in the future. Important for anything that touches typed data.

So far, I tried hard to make sure that each commit makes one change (always with tests), so the commits can be reviewed in order. The sequence:

A few more commits are needed to address the remaining feedback.

I'd still keep this as one issue, for three reasons.

First, it's one concept: what required and nullable mean for typed data, and how validation enforces them. It's hard to judge the API commits without the validation they exist for.

Second, the size is mostly tests. Of the roughly 840 lines, only about 100 are runtime code. The rest is tests and documentation.

Third, it's already split where it matters for review. Each commit makes one change with its own tests and can be reviewed in order.

dries’s picture

My last commit adds the null mode I proposed in #20.

Every data definition now has a getNullMode(), which returns NullMode::Loose by default for backward compatibility. A list definition can opt in with setNullMode(NullMode::Strict).

Fields are always in Loose mode. Moving fields to Strict mode is something we want to attempt in follow-ups, possibly one field type at the time. It's not clear yet how easy that will be, but this merge request is the foundation for it. It gives us the tools.

Because nothing changes unless a definition opts in, I also removed the deprecation that I added earlier. That feels better right now.

Maps can't be Strict yet until #3256536: Map data definition properties are not validated if root value is empty lands. I added a @todo on MapDataDefinition for this.

wim leers’s picture

As the person who led #3364109: Configuration schema & required values: add test coverage for `nullable: true` validation support 3.5 years ago, it's both baffling and beautiful to see so much attention on this issue! 🤩💙

Can we get a change record that summarizes the architectural decisions and changes, to allow me to efficiently catch up? (While the issue summary looks nice too, it was last updated in #8, and AFAICT #19 is the current direction.)

dries’s picture

Thinking a bit more about how we might move fields to Strict mode in follow-ups.

Let's take an email field, for example.

Today NotNull rejects an empty field by treating it as NULL. What we'd have to do to make it Strict:

  1. Add a NotEmpty constraint and put it on every required field alongside NotNull. Seems easy.
  2. Fix how we store the field. This is the difficult-ish part. The database stores "no value" and "empty list" the same way: no rows. So after a load, a field doens't remember if it was empty or null. That means we need a storage change?

It seems doable, and the benefit is that fields would then mean what their JSON Schema says, like the rest of typed data.

What am I missing?

dries’s picture

Issue summary: View changes
Status: Needs work » Needs review

I took a first crack at updating the issue summary. We're at a good stopping point to get some reviews.