Generate JSON schema for content entity types

I like the name "schematic", especially alongside "deterministic". Very clear.

This is unblocked - #3252872: Use CacheableSupportsMethodInterface for performance improvement in normalizers replaced #3031214: Introduce "deterministic" normalizers, which was originally cited as the blocker. Woohoo.

I generally understand the interface proposed, except for:

$format is obviously the encoding format.

Finally, I think that we would put this method on the Serializer service so that normalizers will not need to specifically know which child normalizer services will be applied.

I want to gut-check how this would actually work. Assuming SchematicNormalizerInterface extends NormalizerInterface, a module that wishes to generate a schema, e.g. OpenAPI, would do something like (pseudocode):

$json_schema_formatted_schema = $serializer->getNormalizationSchema(Thing::class, 'schema_json', ['bundle' => $bundleName]);
return decorateForOpenApi($json_schema_formatted_schema);

And, under the hood, the serializer would call Serializer::getNormalizer() in basically the same way as if you were normalizing the data?

Would all normalizers (basically, anything implementing NormalizerInterface other than the serializer itself) then need to set

protected $format = ['api_json', 'schema_json'];

...in order to say hey, pick me when you're doing normalization of either data or schema?

OR does $format in the ::getNormalizationSchema() mean the format the data would be normalized in? Thus we are saying, each serialization format is coupled by definition with a schema format? E.g., api_json gives you schema in json-schema?

Mulling this over a bit more, I think my confusion stems from the basic underlying question: What do we mean by "schema," as far as what format is it represented in? In the example of OpenAPI, it is a superset of JSON Schema, so if I start with a normalized array of a JSON schema, I may not have all the information I need to add the OpenAPI metadata, for instance.

Maybe I'm misunderstanding. Currently, Schemata module generates a Schema object which basically looks like a collection value object for typed data definitions. This is helpful, but then you need to basically normalize the Schema object itself, which doesn't seem to be the intent here. Or perhaps I'm way off base.

I think schema should be defined as the definition of data. That should indeed enable other modules to take this definition and output that through a spec like json-schema or OpenApi.

The point made about 'deterministic' normalizers is that we need to know what normalizer we end up in. The fact we now define in core which normalizers do not change based on the data but are resolved by class and output format (the change in #3252872: Use CacheableSupportsMethodInterface for performance improvement in normalizers) means you should be able to determine the path the normalizers would take AND make sure the normalizer you get will always be the same.

*Thinking out loud here*
In the IS there is mention of using a parameter bag to determine some facts you cannot get from the class with bundle being the obvious example. I'm thinking though if the schema that can be provided uses the same routing as jsonapi is should be easy enough to determine which resource type we are talking about and going through the motions of generating the schema. My mind does wander towards endpoints with multiple resourcetypes though, this might end up being a pain, but something we should consider eventually.

Anyways, i agree a POC feels quite realistic at this point :)

Thanks for the reply.

I think schema should be defined as the definition of data. That should indeed enable other modules to take this definition and output that through a spec like json-schema or OpenApi.

Agreed, though that basically is a tautology - the schema is a definition of data, which is the definition of a schema. I might not have been entirely clear before, but I'm trying to get at - in what format? My example with OpenAPI is appropriate, I think, in so far as it's "like" json-schema but a bit extended. So it needs "more" raw information from which to pull.

I would need to do some research but I think the type for the return value of this method (if we're following the pattern of making this a new interface) needs to be determined first, and then everything else kind of flows pretty logically from there.

I'll need to do some more research with this in mind but I feel like json-schema might be one of, if not the only, options here for a well-known, interoperable schema definition that does not have Drupal reinventing the wheel or simply doing something like returning a bespoke value object.

From #contenta just now in Drupal Slack:

So per the late night brainstorming session with @bbrala pasted above:

• json-schema seems to be the de-facto standard to generate schemas; this is a mature, well-supported and portable format, and there really isn't any modern alternative.
• Earlier attempts at providing openapi schemas were limited by OpenAPI 2.x/Swagger's not-perfect intersection with the json-schema spec. That has since changed in OAI 3.1, and so the use case of generating an openapi spec for the json:api API is clear-cut. You can even mix and match schema styles if need be (but I don't think we'd need to.)
• Currently, modules like jsonapi_schema and openapi_jsonapi need to do normalization of the data type definitions; the idea here is that we would get schema from fields' respective classes (basically, the things that actually get normalized into json) instead of having to normalize the data type definition. There is a known universe of normalizers shipped with json:api, and they are @internal, so we can encapsulate this logic, there.

A bit more of an archaeological dive:

json:api had integrated schema generation at one point, which was removed in this commit, but it provides a good example of a more tightly integrated approach that was pulled out, and seeded Schemata.

This is all very content-entity centric thus far of course. We also need to think about config entities, although let's face it, most of the use case here is for content. So I think it could be appropriate for the config entities to be perhaps less robust. See #2994473-10: [META] JSON API's normalizers support schema tracking, to guarantee comprehensive schema

Very glad to see the movement here. This would help tremendously :)

The blocker is closed as outdated so IS needs update

Think this is the issue, we can use the symfony property for that I think.

#3252872: Use CacheableSupportsMethodInterface for performance improvement in normalizers

Updating IS.

👀👀👀👀

Changing the component to serialization because this is mostly not specific to JSON:API.

I am marking NR because I'm deep enough into this now that I would really like some maintainer/committer review to ensure this isn't totally off-base. I'm personally pretty proud of how this is implemented with the interface and the mostly-automatic integration of the base normalizer with the new Attribute.

So, yeah, this now requires a change to symfony/serializer which is a bummer, and I'm not quite sure what the work-around would be. I've chosen to just include the patch for now, in hopes that as we work on this that either 1) Symfony will make a minor change to two methods from private to protected, or 2) someone smarter than I comes up with a different alternative.

The issue is that we are adding a new method to normalizers to express information about their normalization, but the selection of said normalizer is hidden behind the serializer's ::normalize() method, where it finds the correct normalizer from all those registered. We need to be able to essentially statically-analyze the selected normalizer. We need to peer into ::getNormalizer(), which is currently private.

There is a dirty hack to invoke private methods with reflection, but I doubt that would pass a core quality gate and makes me feel very dirty.

Still leaving NR because I would love feedback on this and the overall approach. Not letting this slow down development, but it is a sticking point.

This should now be to the point of producing a spec-compliant schema for a resource object. There's plenty more work to be done, but I'm pretty happy with how relatively easy it was to implement the earlier generic work on schematic normalizers to JSON:API in basically an afternoon.

Re: our need for the Symfony Serializer to change two methods from private to protected, I've received some initial feedback from two Symfony maintainers and both asked good questions for follow-up.

The concrete bad news is that Symfony 6.4 and 7.0 are in feature-freeze, so the earliest this change would be accepted is 7.1.

Parallels were drawn between what we're doing and api-platform, which is an official-ish reference implementation of Symfony for doing fancy API things. They are generating JSON Schema using object reflection, and I laid out how this isn't much of a parallel to the Drupal entity and field APIs. We'll see if anyone's convinced by my reasoning enough to change the method visibility.

So optimistically, this would be blocked on Symfony 7. If this is a non-starter, there are two other options:

1. Most radical: Ditch the Symfony serializer/normalizer because we barely use it now and abuse it when we do. This would be a major lift/BC break, but it would also alleviate a lot of headaches encountered in the D8+ lifecycle.
2. Fork the Symfony Serializer and implement our own which implements SerializerInterface. This increases maintenance overhead and we are forked further from Symfony core, which we're trying to avoid. But there is precedence, e.g. we have our own YamlLoader.
3. (Ab)use NormalizerInterface::normalize()'s context parameter and allow normalizers to return a value object containing a schema. There is some precedence for returning a value object as JSON:API module needs cacheability metadata and so it passes around CacheableNormalizations. This is a bit of an escape hatch if we get cornered by other options. The downside is that is makes ::normalize() even more polymorphic and magical and also implies the value you pass is "real data," instead of potentially being "just" a supported class or interface name, which is all we might have during schema generation. Then again, that parameter is already mixed and so we could just define by convention that if the special context flag is passed, the value is to be regarded as the same as to ::supportsNormalization(). We could also use this approach to start and then deprecate it in favor of a more explicit method if we can ever make it publicly callable.

First off, im so happy to see you working on this issue. :)

I've been tring to wrap my head around this and am having a hard time hehe.

One of the things we might be able to do is perhaps use the serializer to keep track? We already overwrite the contructor for that and pass that up to the parent. Perhaps there is a way to add the required data to the normalizers there using our own interface. This also feels very tacky in a way, but could be a place where we can track that information and allow it to be pulled in the normalizer itself perhaps.

This is purely based on looking at code, have not tried and see if that would work and how that would look.

The suggestion to use another format for the json_schema in the github issue and go from there is also interesting, wouldn't that work? it might not be the fastest of things to serialize again and go through all the paces to get the schema and would add another whole normalization pipeline, but it could be cached pretty well I guess. This does seem to combine pretty well with the 'describeby' member that was added in jsonapi 1.1 though, which is a link. It could then be a link like '/jsonapi/node/article?format=json_schema'.

Basically, i have no real awnser here right now. I'm not convinced we will see Symfony change the visibility right now. Hopefully my thoughts are helpfull in a way.

Thanks @bbrala for the feedback. Sorry I couldn't make it to Lille this year to work on this in person! We got a lot done last time we pair programmed in Portland.

One of the things we might be able to do is perhaps use the serializer to keep track? We already overwrite the contructor for that and pass that up to the parent. Perhaps there is a way to add the required data to the normalizers there using our own interface. This also feels very tacky in a way, but could be a place where we can track that information and allow it to be pulled in the normalizer itself perhaps.

This is purely based on looking at code, have not tried and see if that would work and how that would look.

I think I follow what you're saying, and in theory yes we could do a lot of shenanigans within the @internal portion of JSON:API module. However there are two main gotchas to this approach: 1) despite these methods not being "technically" extend-able, modules like Extras violate that with imposters and so any normalizers from modules like that, while violating the internal rule, would need to also change. And 2) this would only address JSON:API's normalizers, but not those in the core typed data API, which provides all the schema for properties.

The suggestion to use another format for the json_schema in the github issue and go from there is also interesting, wouldn't that work?

I ran into your feedback while coming here to say that yes, I think this suggestion for treating JSON Schema as another "format" is going to be the path forward.

...it might not be the fastest of things to serialize again and go through all the paces to get the schema and would add another whole normalization pipeline, but it could be cached pretty well I guess.

Unless we're talking about different things here, and I don't think we are, I don't see a performance hit. The approach with ::getNormalizer() would depend on the same normalizer resolving process that was optimized recently with the help of upstream changes, and this is just another path for resolving a normalizer but for a different $format. Also as it stands I don't believe that performance should be a huge issue here, as the resulting schemas could be cached and won't vary much.

This does seem to combine pretty well with the 'describeby' member that was added in jsonapi 1.1 though, which is a link. It could then be a link like '/jsonapi/node/article?format=json_schema'.

That's an angle I hadn't explored yet, however I think it's really powerful and would open the door to a very low-code way for us to bring the functionality that currently lives in jsonapi_schema module into core. I almost wonder if that shouldn't be the near-term goal vs. the OpenAPI integration, though OAI is basically just formatting the same data in a slightly different way, so these are not in competition.

Basically, i have no real awnser here right now. I'm not convinced we will see Symfony change the visibility right now. Hopefully my thoughts are helpfull in a way.

Yeah, after the third maintainer basically said "no," I agree, and honestly this is how the process should work. I iterated on the initial idea (an interface for the normalizer), it turns out that won't work and is blocked on upstream cooperation anyway, but we land on a solution that might even be "better."

I am toying with different ways of implementing this, however, that would still perhaps leverage an interface and/or trait to do most of the heavy lifting for the normalizers. I have some ideas that I'll toy around with in the MR as I refactor out of this solution and into the alternative for $format.

Marking NW as I only really had it in NR to solicit this kind of help.

Putting this back to NR as I would still love feedback from maintainers and committers.

The MR now contains a refactor of the initial approach, which I think overall is an improvement. TL;DR, you now request schema from the normalization system by specifying the $format as json_schema.

The only real nit I can see with this approach is that normalizers are resolved based on the data to be normalized and the format, and so there is theoretically a different universe of normalizers selected for schema vs. a specific format. That's pretty much unavoidable with this approach. In theory we could get 100% the same normalizers by specifying the format same as the eventual normalization and hinting in $context that we actually want the schema. (This is in spirit what we wanted by having access to ::getNormalizer().)

I don't love this but don't hate it, in theory. The practical reason we can't take this approach is that normalizers which don't care about the $format (which is actually the majority of them) need to be aware of that context, and might return a "normal" normalization instead of schema. We have to address this as it is with this change to NormalizerBase::checkFormat() to special-case the json_schema format as an exception to the "if I don't specify any formats, I serve them all" default.

Another question would be if the JSON:API implementation should wrap schema in CacheableNormalizations or not.

One thought about type-safety on the returned value (since we are using ::normalize() for schema as well as normalizations) could be to require it to specify the JSON Schema meta-schema it implements in $schema, which would be a quick check for the calling code to say "this is for sure a schema." I'm not sure if that's necessary, but one idea to throw out there if people are worried about non-core normalizers somehow getting tricked into returning a normalization when we really want a schema.

Making title less technical and more accurate as to the current goals.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Good bot.

Config schema generation is very different and a path forward for them is not very clear yet. The recent work on validation for config entities will help unlock this in the future, however.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Is there a high level summary for this issue? I can't tell from the huge changeset what exactly is being proposed here. What's the before/after? "Content entities were serialized in BLOBs before (in which cases?) but now they are serialized in JSON?" What's the benefit? Which APIs are affected, etc? Especially of an issue of this magnitude I think it would be important to outline these. It should help with reviews as well :)

+1 to what @Gábor Hojtsy said. Change records would be really helpful to understand this functionality too. 😇

Draft CR added. IS updated. MR rebased. Back to NR.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Conflict was from the conversion of typed data plugins from annotations to attributes 💯

Related: #3426508: Generate JSON Schema for config entity types

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Rebased. Back to NR.

I went through the MR and have some comments. All and all it is a good implementation of what we talked about (quite) a while back.

Change record is available.
Issue summary seems up to date, although format is a little more freeform, would prefer to move to default setup. So keeping that tag.

Some small questions for now.

Related note, so close to having a proper JSON:API 1.0 and 1.1 schema available on jsonapi.org but it's blocked on infra: https://github.com/json-api/json-api/issues/1749

That doesn't mean we have to block this issue on that, and now that it's in a const it's easy enough to update.

I've gone through the changes and all previous comments. Think we got through everything and the feedback has been adressed. Remove NISU since the wording has changed referencing the other issue.

I'm gonna go out on a limb and push to RTBC. <3

I've added a couple of small comments to the MR that could be addressed.

I have addressed your comments.

A bit of a fat-fingered rebase but I am hoping that this is very close to RTBC still.

The comments by alex have been adressed. I agree with the argument for the array argument. Back to RTBC

Updated credits

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Putting this back to RTBC as it only needed a phpstan-related update relating to improved analysis as this awaits commit.

Updating tags. Marking this as a contrib blocker because new versions of OpenAPI and OpenAPI JSON:API depend on this.

Thank you!

I think the patch needs a reroll as it doesn't apply to 11.0.6.

I've rebased against 11.x, if it doesn't apply to a specific tag it would be because of a change in HEAD.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

This is likely due to #3100732: Allow specifying metadata on JSON:API objects going in which is a good thing. Needs a rebase then should be back to RTBC.

Rebasing took a little effort because we need to deprecate some extra arguments. I also had to undo contructor promotion since the merged json:api metadata issue changed the contructor parameter.

I think the changes are small enough no to warrent a new review, all is green still.

Some questions on the MR, will keep an eye out for when this is addressed, as getting this into 11.2 early is a priority

Back to NR.

All things have been adressed.

Exciting times :)

Committed to 11.x - thanks

We've got ~6 months to find any issues downstream from here - likely started with JSON API extras which is maintained by several of the people in this issue.

Great work all

Is this eligible to go into D10 LTS?

Under https://www.drupal.org/about/core/policies/core-change-policies/allowed-... - I don't think so.

Published the change record.

🤯 What a christmas present! 🤩

Questions:

• The change record doesn't mention it, but the commit does change the JSON:API version: JsonApiSpec::SUPPORTED_SPECIFICATION_VERSION = '1.1' is now a fact. So #3305324 is partially done? Completely? See #3305324-18: [Meta] JSON:API 1.1 spec compliance/support. It seems that meta indicates there's a lot more to be done to be 1.1-compliant, so it's a bit surprising to see that constant being changed. I think either an updated change record that clarifies it, or an additional change record would be appropriate?
• Next up: #3426508, and I posted some pointers at #3426508-4: Generate JSON Schema for config entity types.
• This seems to statically define JSON schemas, which is not good enough: many field types have settings that impact what the correct JSON schema would be. IOW: the JSON schemas have to be dynamically defined for them to be precise. For example: min/max for integers and numbers, a limited set of valid values (enum in JSON schema), many strings have a particular format (https://json-schema.org/understanding-json-schema/reference/string#built...), and so on. Are there plans/discussions for how to support those more precise JSON schema descriptions? Perhaps this needs a follow-up?

Either way: HUUUGE leap forward! 🤩👏

EDIT:

Can we use this to our advantage in https://www.drupal.org/project/experience_builder? This seems to have the potential to simplify some of the shape matching functionality in XB, which literally is matching Entity field properties’ Typed Data Definitions (including validation constraints) against the JSON schemas of SDC props:

• https://git.drupalcode.org/project/experience_builder/-/blob/18d36048e61...
• https://git.drupalcode.org/project/experience_builder/-/blob/18d36048e61...
• https://git.drupalcode.org/project/experience_builder/-/blob/18d36048e61...
• explained in https://git.drupalcode.org/project/experience_builder/-/blob/18d36048e61...

P.S.: first <hr> of the year 😁

Thanks, Wim!

I will reply to your specific points soon, but why is this now marked as to-be-ported? The reply above indicates this can't go into 10.x, so I think there's nothing to backport?

(To be fair, my main project in production is "still" D10 and this applies relatively cleanly, but I also understand if it can't go officially into D10 because of policy.) I would love for this to be supported in D10 as that would personally benefit me and make the new majors of OpenAPI/OpenAPI JSON:API D10 compatible but appreciate if that's just too much.

I think NW to add a follow up for at least the third of Wim's points is an appropriate status - tagging as such.

Getting this early into 11.2 allows us to smooth out any pain points so it would be great to see what the current approach is missing for XB and refine this before 11.2 comes out

Re: ##8 - thanks Wim for your very kind words.

The change record doesn't mention it, but the commit does change the JSON:API version: JsonApiSpec::SUPPORTED_SPECIFICATION_VERSION = '1.1' is now a fact. So #3305324: [Meta] JSON:API 1.1 spec compliance/support is partially done? Completely? See #3305324-18: [Meta] JSON:API 1.1 spec compliance/support. It seems that meta indicates there's a lot more to be done to be 1.1-compliant, so it's a bit surprising to see that constant being changed. I think either an updated change record that clarifies it, or an additional change record would be appropriate?

I bumped the version in this MR for a few reasons. One, our testing and validation features (e.g., that validates responses when assertions are enabled) depend on a schema for JSON:API itself, which is not exactly stable but the upstream maintenance of JSON:API overall is glacially slow. The schema file we had for 1.0 was draft and not that valid and so the updated 1.1 draft is much closer to truly representing the spec's requirements.

After reviewing the linked issue I would say that we very nearly support 1.1 at a basic level, though we "should" also finish #2955020: Spec Compliance: JSON API's profile/extention (Fancy Filters, Drupal sorting, Drupal pagination, relationship arity) needs to be explicitly communicated to convey our profile.

TL;DR, it's time to bump it because we were far ahead of 1.0 with implementing features that would later go into 1.1. We don't strictly validate, but it's much closer to the current spec than we've ever been.

This seems to statically define JSON schemas, which is not good enough: many field types have settings that impact what the correct JSON schema would be. IOW: the JSON schemas have to be dynamically defined for them to be precise. For example: min/max for integers and numbers, a limited set of valid values (enum in JSON schema), many strings have a particular format (https://json-schema.org/understanding-json-schema/reference/string#built...), and so on. Are there plans/discussions for how to support those more precise JSON schema descriptions? Perhaps this needs a follow-up?

Schemas _may_ be defined statically, but they need not be. In addition, JSON Schema generation has no caching layer (the assumption being that the caller is responsible for this, e.g. OpenAPI module, and invalidation is based on entity definitions or whatever) so the entire schema for the normalization you are introspecting is generated on-demand. It is true that included in the merged MR was a trait and attributes that help provide static schemas... but that's not a requirement. This allowed us to add a lot of default schemas for primitive types and make it easy to define them this way, but you can also return any valid schema generated any way you like. So I think a follow-up for option module to provide a ListStringItemNormalizer with its own dynamic logic would be amazing. But it requires no change to the underlying API. Perhaps we could introduce some additional traits or whatever to assist with code re-use, but yeah.