Problem/Motivation

Primitive data types describe their normalized output with the JsonSchema attribute, and Serializer::getJsonSchema() exposes it through the json_schema format. See the change record.

Nothing describes a map. ComplexDataNormalizer does not opt in to the json_schema format, so the serializer falls back to TypedDataNormalizer, which returns a placeholder with only a $comment. Validators ignore $comment, so any data passes. A map's property definitions already describe the object, and each property can already describe itself.

Steps to reproduce

$map = \Drupal::typedDataManager()->create(
  MapDataDefinition::create()
    ->setPropertyDefinition('id', DataDefinition::create('integer')->setRequired(TRUE))
    ->setPropertyDefinition('label', DataDefinition::create('string'))
);
\Drupal::service('serializer')->getJsonSchema($map, []);
// ['$comment' => 'No schema is defined for property of type Drupal\Core\TypedData\Plugin\DataType\Map. ...']

Expected:

[
  'type' => 'object',
  'properties' => [
    'id' => ['type' => 'integer'],
    'label' => ['oneOf' => [['type' => 'string'], ['type' => 'null']]],
  ],
  'required' => ['id'],
  'additionalProperties' => FALSE,
]

Proposed resolution

Add MapSchemaNormalizer, which only supports the json_schema format and only the Map class itself. It builds an object schema from the same non-internal properties that ComplexDataNormalizer::normalize() iterates.

ComplexDataNormalizer stays out of the json_schema format. Field item normalizers extend it and add keys that aren't properties, such as target_uuid on entity reference items, so a schema built from properties would not match their output. Field items, and other Map subclasses, keep the fallback.

Decisions:

  • additionalProperties is FALSE. Map::setValue() accepts unknown keys, but normalize() only outputs defined properties.
  • Field items are out of scope.
  • Maps without property definitions, maps whose properties are all internal, and maps whose properties are named 0, 1, 2 keep the fallback. None of them normalize to a JSON object with known keys. Numbered properties describe a list, which should use ListDataDefinition.
  • isRequired() maps to the parent's required list. Separating presence from nullability is #3625989.

Remaining tasks

  • Address the MR review.
  • Subsystem maintainer review.
  • Decide whether to update the existing change record or write a new one.

User interface changes

None.

Introduced terminology

None.

API changes

New serializer.normalizer.map_schema service. Serializer::getJsonSchema() returns an object schema for maps with property definitions. Normalization output does not change.

Data model changes

None.

Release notes snippet

AI-Generated: Yes. I used Claude Code to draft this issue summary and to check it against the MR. I reviewed and edited the result.

Issue fork drupal-3627318

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mglaman created an issue. See original summary.

mglaman’s picture

Issue summary: View changes
mglaman’s picture

Issue summary: View changes
mglaman’s picture

Issue summary: View changes
mglaman’s picture

Issue summary: View changes
mglaman’s picture

Issue tags: +rosetta

dries made their first commit to this issue’s fork.

dries’s picture

Status: Active » Needs review

I took a crack at this and opened a merge request for this.

Disclaimer: I built this with the help of an LLM but reviewed everything extensively.

While working on this MR, I kept running into nuances: cases where a map didn't behave the way I expected. I wanted to share some of my learnings so any review goes faster, or you can correct me when I'm wrong.

The basics

A map is typed data for a set of named values. Think of it as a PHP array with string keys. Its definition can declare properties:

$definition = MapDataDefinition::create()
  ->setPropertyDefinition('title', DataDefinition::create('string')->setRequired(TRUE))
  ->setPropertyDefinition('count', DataDefinition::create('integer'));

The map's value is the PHP array it contains, like ['title' => 'Hello', 'count' => 3], and its keys are that array's keys. When the definition declares properties, the keys are the property names.

The serializer turns typed data into plain PHP arrays, which is called normalizing, and then into JSON as {"title": "Hello", "count": 3}.

The serializer can also describe data as a JSON Schema. However, before this MR, core could not describe maps as JSON Schema.

When core can't describe something as JSON Schema, it returns the fallback schema: {"$comment": "No schema is defined for property of type ..."}. Clients and JSON validators ignore $comment and, maybe no small detail, any data passes validation.

Every map gets the fallback today. The merge request adds MapSchemaNormalizer, which gives the map above the following JSON Schema:

{
  "type": "object",
  "properties": {
    "title": {"type": "string"},
    "count": {"oneOf": [{"type": "integer"}, {"type": "null"}]}
  },
  "required": ["title"]
}

So far so good, so now the nuances, in order of increasing head scratching:

Nuance 1: internal properties are left out

A definition can mark a property as internal, with setInternal(TRUE). Internal properties are left out of the normalized data, and we leave them out of the JSON Schema as well. (@michaellander, I believe that is different from Tool API, but it felt more correct.)

Nuance 2: a map without property definitions accepts any array

I didn't fully realize that you can have a map without property definitions. It makes sense now, but it's good to know that in this case, any array is accepted.

The challenge is that its keys aren't known in advance, and because of that, it gets/keeps the fallback schema. There is a test for that.

Nuance 3: properties named 0, 1, 2 become a JSON array

PHP has a function array_is_list(), which calls an array a list "if its keys consist of consecutive numbers from 0 to count($array)-1" (see https://www.php.net/manual/en/function.array-is-list.php). Interestingly, json_encode() encodes such a list as a JSON array and any other array as a JSON object:

  • [0 => TRUE, 1 => 'a'] becomes [true,"a"]
  • ['0' => TRUE, '1' => 'a'] also becomes [true,"a"], because PHP turns the keys '0' and '1' into the integers 0 and 1
  • [0 => TRUE, 'key7' => 'a'] becomes {"0":true,"key7":"a"}
  • [] becomes [], not {}

Long story short, some arrays are lists apparently, and lists gets encoded as JSON arrays. An object schema would reject that array, so I decided to keep the fallback schema. It's an edge case, but I added a test for it.

Strictly speaking, the fallback may not be the best we could do here. We know the map will be encoded as a JSON array, so we could describe it as one. Given that it is such an edge case, we could make this a follow-up issue, or try to include it later in this MR.

Nuance 4: field items are maps, but keep the fallback schema

Each value of an entity's field, like a node's author, is a field item, and field item classes extend Map. But field items have normalizers of their own, which can add keys to the output that aren't properties. Yes, you might need to read that twice. A node's author, for example, outputs one property, target_id, but its normalizer also adds target_type, target_uuid and url.

To be honest, I don't know why yet. And because of that, I decided that MapSchemaNormalizer describes only the Map class itself, and field items, like any other class that extends Map, keep the fallback schema. The test's last case checks that. The key to understand is that this isn't a regression.

For example, a node's author data is normalized to:

{"target_id": 1, "target_type": "user", "target_uuid": "7cc783fa-...", "url": "/user/1"}

Before this change and after it, its JSON schema is:

{"$comment": "No schema is defined for property of type Drupal\\Core\\Field\\Plugin\\Field\\FieldType\\EntityReferenceItem. ..."}

So this changes nothing for field items: they get the fallback schema today and still do after this MR. I need to dig into this more, but we may be able to improve this in a follow-up issue. This MR provides us the foundation for that.

Hope that helps! With that, check out the code.

nicxvan’s picture

It might be worth asking @bradjones1, he's the subsystem maintainer.

There are enough edge cases outlined here I'd be curious if the correct approach is cleaning them up or maintaining the current fallback behavior. L

mglaman’s picture

Thanks for this, Dries. I updated the issue summary to match the MR, and left a review on the MR.

On the nuances:

  • Nuance 3: a map with properties named '0', '1', '2' is a list. Typed data has ListDataDefinition for that, and config schema has sequence. The fallback is the right answer for that definition, so I don't think it needs a new issue to describe it as a JSON array.
  • Nuance 4: agreed. Field items keep the fallback until their normalizers describe their own output.
  • additionalProperties: my summary had this wrong. Map::setValue() keeps unknown keys, but Map::getProperties() only builds properties from the definitions, so normalize() never outputs them. The schema can set it to FALSE.
$map = $typed_data_manager->create($definition, ['id' => 3, 'zzz' => 9]);
$serializer->normalize($map);
// ['id' => 3]

Mapping isRequired() to required lines up with #3625989, where required means present and nullability gets its own flag. I added it as a related issue.

@nicxvan agreed, @bradjones1 should weigh in. The edge cases are narrow, though: every one of them keeps today's fallback, so nothing gets worse.

Generated with the help of an LLM.

mglaman’s picture

Status: Needs review » Needs work
dries’s picture

In the same spirit as #3625989-22: NotNullConstraintValidator conflates "required" with "non-empty" for lists and complex data; only config schema is exempted, my last commit refactors the tests I added earlier.

As I wrote in #3625989:

I think it's really important that TypedData and JSON Schema agree, so Drupal can be compatible with external tools and other platforms. This new test will help enforce that. [...] JSON Schema gives us a shared reference point or shared contract. It provides direction, allows us to focus on the details we actually need to decide, and avoids us having to bikeshed nuances.

The refactored tests stop comparing each schema with a hardcoded array written in the test. Instead, they use core's JSON Schema validator instead. This makes JSON Schema our contract and avoids drift in those hardcoded arrays.

To do: an empty person, with every value NULL is left out until #3256536: Map data definition properties are not validated if root value is empty lands.