[META] JSON API's normalizers support schema tracking, to guarantee comprehensive schema

#3014277: ResourceTypes should know about their fields paves the path to field level schema.

Field level schema is the fundamental part of resource schemas. That is because the resource schema is a composition of field schemas using the JSON:API schema.

I quickly scanned that link @Wim Leers, it looks like that schema is generated from Symfony's "models" (akin to our entity types) and it makes no mention of serialization. I suspect it suffers from the same problems as using Typed Data to generate output schema.

@e0ipso asked me to record a couple challenges that we need to consider that came up in our weekly API-First meeting...

It seems the only way to properly generate output schema is to have normalizers for every data type in Drupal that can be determined without an instance of the data to be normalized. That means that they need to be determined by class name, maybe field metadata (like field settings or machine names) or within JSON:API by the ResourceType (or a similar field-level value object).

Unfortunately, what we've promised is that Drupal's normalization system will continue to work on the Typed Data level, IOW, we've "cordoned off" most of the normalization system, except for field properties. So, I'm not sure that normalizers can ever be deterministic at that level.

Another challenge in config entities, which don't have fields. The best we've got is Typed Data and/or config schema, but even then, we're SOL because there are instances of config schema with runtime dependencies, like in the following example from block.schema.yml:

block.block.*:
  type: config_entity
  label: 'Block'
  mapping:
...
    plugin:
      type: string
      label: 'Plugin'
    settings:
      type: block.settings.[%parent.plugin]

As you can see, the type for settings depends on the plugin ID in the block config.

I think that it will be impossible in Drupal 8 to have 100% deterministic and static schema. IOW, we'll eventually have to accept that the output schema for some deep properties of a data structure might need to be any. That said, I'm sure that there will be ways to get as close as possible to perfect schema for 99% of uses cases.

Agreed on the configuration entities.

Unfortunately, what we've promised is that Drupal's normalization system will continue to work on the Typed Data level

Does this need to be true? Maybe we could put some useful constraints on that that work in our favor.

I had this relevant conversation the other day with @alexpott

e0ipso [4:41 PM]
I'm failing to find an easy way to retrieve the schema for a configuration entity type.
Has anyone dealt with that in the past? (cc @fubhy)

alexpott [4:41 PM]
@e0ipso it’s tricky without an ID

e0ipso [4:41 PM]
yeah, that's what I'm finding.
@alexpott ideally I'd like to determine the schema statically so I don't have a _representative entity_.

alexpott [4:43 PM]
Yeah there’s a big big but to that though. The schema can be dependent on the data. This is how plugins work

e0ipso [4:44 PM]
I see… thanks @alexpott. I knew I was going to find that problem eventually. :stuck_out_tongue:
(for context, I'm trying to build reliable schema for Open API for the auto-generated REST resources based on entity types)

alexpott [4:45 PM]
You can generally get the first level of keys. But not their type info :disappointed: - see \Drupal\Core\Config\Entity\ConfigEntityType::getPropertiesToExport()

e0ipso [4:47 PM]
I'm using that method. It must mean I'm on the right track. I'll share frustrations to see if we can improve this area.

alexpott [4:48 PM]
I guess you going to need to traverse the schema and look for all the ones of type ‘config_entity’ and then traverse the mapping and find the plugins and then somehow come up with all the possibilities for the plugins.
So something like `node.type.*` is relatively simple because the one complexity you have to deal with is `third_party_settings` so you’ll need to find anything of `node.type.*.third_party.%key` - for example `node.type.*.third_party.menu_ui` on the site (edited) 

e0ipso [4:50 PM]
Sounds like it
I'll be happy to have a first pass with correct schema on the top level properties
But eventually that's going to be a must

alexpott [4:52 PM]
Block might not be a bad place to start
You have `settings` which is completely plugin determined
and `visibility` which is determined by the sequence key
and there’s the `third_party_settings` from core config entity… so you have all the problems that something like views has but on a smaller scale. (edited) 

There is a proof of concept to statically extract data definitions out of content entities and config entities.

In the situations described in #10 we can only say: well, there is a property with a data definition that varies from config entity to config entity, so we cannot know in advance. Hence the data definition is undefined.

https://git.drupalcode.org/project/jsonapi_schema/blob/8.x-1.x/src/Stati...

Moving to core.