API-First Initiative: top priorities for Drupal 8.8.x

Removing #2856110: [PP-1] Expose entity validation errors in a machine readable REST API: , which is blocked on #1916302: RFC 7807: "Problem Details for HTTP APIs" — serve REST error responses as application/problem+json, because (quoting myself from #1916302-30: RFC 7807: "Problem Details for HTTP APIs" — serve REST error responses as application/problem+json):

FYI: The jsonapi.module effectively already implements this, because the JSON:API spec has absorbed the relevant portions from RFC7807. See https://jsonapi.org/format/#error-objects. Even the language is lifted from RFC7807, for example:

RFC7807

     "title" (string) - A short, human-readable summary of the problem
      type.  It SHOULD NOT change from occurrence to occurrence of the
      problem, except for purposes of localization

JSON:API spec

title: a short, human-readable summary of the problem that SHOULD NOT change from occurrence to occurrence of the problem, except for purposes of localization.

That doesn't change anything here, but I just wanted to call it out explicitly that this issue applies only to rest.module.

Given that, and the recommendation to use jsonapi.module instead of rest.module for interacting with entities, which is where validation occurs most frequently, I am removing #2856110: [PP-1] Expose entity validation errors in a machine readable REST API: from the top priorities. (It also saw very little interest; the status quo
seems to suffice.)

I'm marking rest.module-specific issues with Ⓡ and jsonapi.module-specific issues with Ⓙ.

I triaged the entire JSON:API issue queue once more.

I also took another look at the API-First Initiative Priority Matrix that @gabesullice started (thanks Gabe! 👍) and incorporated much of tha.

This issue summary is meant to evolve: feel free to reorder, add, and remove. As one thing gets blocked, others might move forward. If you're interested in helping out with something, come find us in #contenta on Slack.

There are so many things we would like to see happen. So many foundational improvements that have been holding back rest.module for so long (e.g. full config entity support), have caused countless hours spent on security mitigations for jsonapi.module (e.g. Entity Query Access API), and there are things that would help both (e.g. serialization gaps, better normalizer infrastructure). Besides those foundational problems, there are many features we would like to see added to JSON:API — from more complete revision support to a query builder UI and a full-featured JS client.

I know @gabesullice is going to be working on a PoC for a query builder UI soon, he's already been working on hypermedia support, and we all are already working on performance/cacheability improvements and working to complete revision support.

My focus this quarter (until the end of June) is going to be on Media embedding, so expect less from me. I will probably be back in full capacity in July.

Note that for the remainder of the 8.8.x cycle, for JSON:API development the following applies:

1. Commits must happen against Drupal 8.8.x core first, and are backported to 8.7.x when core policy allows this.
2. Only commits to 8.7.x may be backported to the https://www.drupal.org/project/jsonapi contrib module (8.x-2.x)
3. Therefore patches must be filed against the jsonapi.module component in Drupal core. But to speed up development, you can develop the patch in the https://www.drupal.org/project/jsonapi issue queue, since tests run much faster there.

#4 just appended list items to the existing lists.

I just moved them around (without changing any): now they're in the order I think we should tackle them. Looking forward to feedback from my fellow API-First Initiative coordinators @gabesullice and @e0ipso, but also from others of course!

Adding a few more things from the aforementioned API-First Initiative Priority Matrix that don't yet have issues, yet I think are important for 8.8.

Adding four missing "Ⓙ"s.

1. It doesn't matter which solution is better: the problem is that Drupal core doesn't provide image styles at all today. This is a basic expectation, having to install two Drupal contrib modules (Consumers + Consumer Image Styles) is perceived as "WTF, Drupal?". I'm happy to do whatever it takes to get something into Drupal 8, even if it's less than complete. Being able to access something is already better than today's nothing :)
2. This is not new, was already in #3002188: REST: top priorities for Drupal 8.7.x!
3. I don't know what that's about exactly, but I think what matters most is that the overall priorities are listed and agreed upon; we can always add follow-ups later :)
4. I actually did not forget about normalizer schema support! It's listed as:
   
   5. Normalizer infrastructure:
      1. schematic normalizers #3031367: Generate JSON schema for content entity types

   #2994473: [META] JSON API's normalizers support schema tracking, to guarantee comprehensive schema is apparently the meta and should've been listed after "Normalizer infrastructure:" — did that just now, thanks! 🤝

5. It's actually not comprehensive. It lists all remaining things that we think are especially important, highly impactful. It does not list things we necessarily work on, but also serves as a way to show the community at large which areas we feel we think are important to improve, and in which we'd certainly value more contributions.

   I went through the list and removed #2805279: Routing system + authentication system + format-specific routes (e.g. those in rest.module) = frustrating, unhelpful 403 responses instead of 406 responses, #2817737: A REST resource's "auth" configuration MUST be non-empty, which prevents configuring it for anon access only, #2135829: EntityResource: translations support and #2904423: Translated field denormalization creates duplicate values. Hopefully you agree with those deprioritizations. Which others would you like to remove?

More deprioritizations. #2820364-57: Entity + Field + Property validation constraints are processed in the incorrect order downgraded that issue from "major" to "normal" so removing #2820364. #2847041: Add a format and start/end validation constraints to DateRangeItem to provide helpful REST/JSON:API error message also seems too small to merit being listed here.

Thanks! I support those changes :)

• I don't think we should think about custom JSON:API routes before we've explored the full realm of what JSON:API hypermedia support enables. Put differently: let's first provide the extension point that does integrate deeply with JSON:API responses before we open the door to custom routes where it's harder to make that guarantee.
• Regarding #2577923: MenuLinkContent entities pointing to nodes are not deployable: LinkItem should have a "target_uuid" computed property: I added that in #2794263-11: REST: top priorities for Drupal 8.3.x, based on https://www.previousnext.com.au/blog/we-could-add-default-content-drupal..., more than 2.5 years ago. Since it hasn't gotten any attention ever since I posted a functional patch at #2577923-119: MenuLinkContent entities pointing to nodes are not deployable: LinkItem should have a "target_uuid" computed property >1.5 year ago, it seems interest in this has decreased, so I'm fine with deprioritizing it. Removed!
• I'm fine with adding OAuth. But that would mean bringing another module into Drupal core, and readying that for core inclusion. I think it'd be great to expand the capabilities of JSON:API before we start doing that. It's easier to search for additional authentication providers to install on your Drupal site than it is for alternative HTTP APIs (alternative to rest.module), especially one with a hard-to-search for name like "JSON:API" :)

Overall, I feel that this list is a bit too focused on solving edge cases rather than providing new capabilities. Do you feel that way? Maybe you do, but you think that's ideal or maybe it's just a symptom of the fact that it was derived from existing issues?

The intent has never been to focus on providing new capabilities. The intent is to solve challenges people are encountering in production — whether that's gaps in the existing functionality (which you rightfully describe as "edge cases" — the less edge casey things have been solved by now — but remember that your edge case may be my essential basic case!) or adding new functionality. In the past, there was almost nothing for adding new functionality. The current issue summary is for the first time listing new functionality as the majority actually! 💪

Again, adding it in scope here doesn't necessarily mean core.

That's a very good point. That is what it used to mean. But back then, in #2721489: REST: top priorities for Drupal 8.2.x, API-First Drupal was basically limited to the REST API that Drupal core contained. So I think you're absolutely right that we should change this.

I am curious about that expectation and that perception. This is totally normal and expected in other ecosystems like node.js.

Sure, but A) discoverability is a big problem, B) https://www.drupal.org/project/consumers has fairly far-reaching implications on how the provided HTTP API works: you have to set up consumers, update the API client to send a consumer ID, and so on. I don't think it makes sense that Drupal core's stance would be Oh you don't want to download original image uploads megabytes large and of arbitrary dimensions? Then you must first install these two contrib modules and modify your API client. — that's why I'm saying I think it's a basic expectation. Did that explanation result in you nodding with agreement? If not, I'm ready to give up on this and accept that Drupal core will never be able to provide image styles in its HTTP APIs out of the box.

I met with my fellow API-First Initiative Coordinators @e0ipso and @gabesullice to discuss this in some more detail and agree on a select few top priorities.

The outcome:

1. Ⓙ Provide interactive documentation/query builder à la GraphiQL in a stable contrib module: [issue to be created] (work already in progress with https://www.drupal.org/project/jsonapi_schema by @gabesullice and @e0ipso and a query builder UI by @gabesullice and @zrpnr, link to that to follow in next few weeks). Crucially, this involves improving normalizer infrastructure in Drupal 8.8 that will then be guaranteed to be available in Drupal 9, allowing us to have better HTTP API support in the future without breaking BC.
2. Ⓙ Hypermedia adoption #2994193: [META] The Last Link: A Hypermedia Story (https://www.drupal.org/project/jsonapi_hypermedia is already working sufficiently well, we just want to see more modules adopting it to help validate it further before considering making it part of the JSON:API module itself)
3. JSON:API core commit follow-up work: #3040025: [meta] Decide on and implement additional security hardenings for JSON:API and other HTTP APIs + complete revision support + complete translation support
4. JSON:API cacheability/performance improvements for interacting with huge collections and/or highly complex data models

Based on this, I think the theme for this release is Harden JSON:API and validate the big two future enablers: query builder and hypermedia.

Updated issue summary accordingly. More detail there.

To be clear: both the query builder and hypermedia support need must happen in contrib, not in core. But both will likely need improvements in core to reach their full potential (for example schematic normalizers would help the query builder, cacheability associated with links would help hypermedia).

Just talked to @Dries. He’s completely +1 to our 4 top priorities in #17, and agrees with our reasoning. 👍🥳

#3039730: Include paths are resolved for every resource in a resource collection, instead of once per unique resource type landed last week!

@justafish raised two more serialization gaps on today's API-First Initiative meeting: #3066751: Add resolvable_uri property to LinkItem for APIs + #3066768: `parent` field on MenuLinkContent entities uses `string` field type, which is semantically wrong and burdens HTTP API clients.

#2819335: Resource (entity) normalization should use partial caching landed!

I've been triaging the issue queue of the jsonapi.module component in Drupal core. I have good news and bad news.

• #3066202: Filtering multivalue reference fields to contain multiple specified values does not work is a problem in the Entity API component in core, not in JSON:API
  
• #3067770: Entity queries and hence JSON:API filtering based on comment count are not supported is a problem in the comment.module component in core, not in JSON:API
  
• #3067627: API users cannot access menu links is a problem in the menu_link_content.module component in core, not in JSON:API, and an issue for exists already: #2915792: MenuLinkContentAccessControlHandler does not allow "view" access without admin permission, making these entities inaccessible via REST, JSON API and GraphQL and entity reference fields

It seems the majority of bugs being reported against JSON:API are bugs elsewhere in Drupal core. This is actually encouraging, because it means it’s no longer the HTTP API-providing module that gets in the way, it’s the rest of Drupal that actually needs to be thinking of becoming actually API-First. It's also encouraging because it means more people are building more advanced things on top of Drupal core's JSON:API than they were on the REST API.

#3018287: ResourceTypeRepository computes ResourceType value objects on *every request* landed!

Added #2975217: Update the default comment entity owner to the current user.

#2994193: [META] The Last Link: A Hypermedia Story was completed in 2019!

With https://www.drupal.org/project/jsonapi_explorer having been released, Ⓙ Provide interactive documentation/query builder à la GraphiQL in a stable contrib module is also definitely completed. But the underlying infrastructure work has not yet been completed, which limits the reliability of the explorer: in case a site is customizing normalizers, it may break. Still, the current version of it is working and already is a huge enabler!

Significant progress was made on #3040025: [meta] Decide on and implement additional security hardenings for JSON:API and other HTTP APIs too, but it's been blocked on framework manager review for nearly 6 months now.

Shouldn't we just close this since #2757967: API-first initiative has been closed? 🤓

Hehe 😅

Restoring previous title; it's in line with all preceding sibling issues.

End of an era! 😊