Problem/Motivation

To explain why having something like a "JSON API" module in core makes sense, I first have to explain the process/choices that led to the current REST module in D8. I will explain later what "JSON API" is.

rest.module to Drupal core, inspired by the https://www.drupal.org/project/restws contrib module in Drupal 7.

The scope of that issue is clear: it's minimal. It only does DELETE, not yet GET, POST or PATCH. It hardcodes a single format: JSON-LD (the up-and coming standard back then). It only supports content entities and one other resource (watchdog/dblog), to prove that it's possible. This is a fine starting point!

The question of whether to support collections, paging and queries is raised at #1816354-22: Add a REST module, starting with DELETE, and is answered that collections and paging are a problem for Views plugins, because Views is how we build collections — this also makes sense.

This was done in #1816354: Add a REST module, starting with DELETE (committed November 2012).

2013: Default format: from JSON-LD to HAL+JSON
The default format was first JSON-LD. It was then decided to change that to HAL+JSON. Over time it had become clear that shipping a perfect "primary format" was unnecessary, because Symfony's Serializer component makes it easy to add new formats. The primary use case for the REST module was expected to be content deployment. HAL was the simplest format with hypermedia support. Because of the involvement of JSON Schema proponents, HAL+JSON was chosen, because JSON Schema can be layered on top of HAL+JSON.

This was done in #1924220: Support serialization in hal+json (committed March 2013).

2014: no significant changes (mostly updates for D8 API/best practice changes)
2015: no significant changes (mostly updates for D8 API/best practice changes)
2016: #2721489: REST: top priorities for Drupal 8.2.x + #2794263: REST: top priorities for Drupal 8.3.x
The test coverage for the rest, serialization and hal modules was insufficient to guarantee correct behavior and good DX for users of D8's REST API. See the numerous issues listed in those two plan issues.
This is fine and understandable — all of the above was built by volunteers and limited sponsoring. Drupal 8's APIs changed a lot since 2012! The popularity of the HAL format didn't take off as expected.

REST in D8: fundamental pain points after one year

After one year of having Drupal 8 released, some pain points have become clear:

  1. Difficult to set up. From the extra layer of permissions (fixed by #2664780: Remove REST's resource- and verb-specific permissions for EntityResource, but provide BC and document why it's necessary for other resources) to the complex configuration (simplified by #2721595: Simplify REST configuration): it was far too hard. We've made it better in 8.2, but even with https://www.drupal.org/project/restui to make it simpler, it's more hassle than necessary for the 90% case: REST access to entities.
  2. Collections: even harder to set up. For every single collection that you want to access, you need to create a separate view. If you need to access five entity types, you need to set up five views. As you can see in the history, this was intentional, but it leads to silly/frustrating situations. Like the very common case of having an article with comments, and it literally being impossible to GET all comments for an article, because you need to set up a (relatively advanced) view for that. Not to mention that even when you set up a view to provide access a collection, you still don't have pagination: have 1 million nodes? You'll get all of them in a single response, except the request will time out.
  3. Easy to shoot yourself in the foot. Due to combinations of circumstances, the URL structures are confusing (GET/PATCH/DELETE /node/42, but POST /entity/node), forgetting a particular request header or an entity field results in incomprehensible error responses, and figuring out capabilities (such as embedding related resources) requires debugging the PHP code.
  4. Incomplete normalizations. This means you simply cannot access certain information. For example, translated entities, modifying config entities, file URLs, file uploads, image style URLs, parent term, entity paths, and so on. We are fixing those, which will also help the REST module and any other module providing RESTful web services.

We can fix #4, and doing so will help all modules providing RESTful web services.

The first 3 are fundamentally unsolvable, because that's how it was designed.

JSON API: the spec

JSON API is a specification: http://jsonapi.org/.

Like HAL, it describes a particular normalization. But on top of that, it also prescribes:

  1. how to handle collections: http://jsonapi.org/format/#fetching-resources
  2. how to handle filtering: http://jsonapi.org/format/#fetching-filtering
  3. how to handle sorting: http://jsonapi.org/format/#fetching-sorting
  4. how to handle relationships: http://jsonapi.org/format/#fetching-relationships + http://jsonapi.org/format/#crud-updating-relationships
  5. how to handle errors: http://jsonapi.org/format/#error-objects
  6. an extension system is in development: http://jsonapi.org/extensions/ — but the spec already ensures that it's possible to layer this on top: http://jsonapi.org/format/#content-negotiation

In other words: it is more restrictive, it defines much more of how a RESTful web service using it should work than the HAL specification (zero mentions of "collections".

Put differently: JSON API has a broader opinion, which removes the need for Drupal to have an opinion. This in turn allows developers to use non-Drupal-specific tools to consume Drupal 8's APIs.

JSON API: module

e0ipso developed the JSON API contrib module: https://www.drupal.org/project/jsonapi. It's JSON API, applied to Drupal entities.

It made some choices:

  1. Only entities. Because only entities are guaranteed to have collections and relationships. And because it's the 99% use case.
  2. Rely on the Entity (Field) Access API. This means we can expose all entity types via JSON API by default: the responsibility lies with Entity API to ensure access is set up correctly — the rest of Drupal already relies on this too.
  3. All installed authentication providers are allowed.
  4. It doesn't expose the data structures that Drupal uses internally for entities + fields, unlike the REST module. See:

Together those decisions mean that installing the JSON API module means instant, format-standardized access to all entities in Drupal 8 without any configuration, via any of the available authentication providers.

Painless setup. Zero config (no need for documentation). A spec to look at to see what's possible (not Drupal-specific documentation).

Proposed resolution

As part of the API-first initiative (#2757967: API-first initiative), we'd like to propose to add JSON API to Drupal 8.3 core as an experimental module. This should come as no surprise because #2757967 has mentioned it since being published on June 29.

e0ipso is the maintainer of the JSON API contrib module. It's not an early-stages module:

  1. It's been under development since May 2016.
  2. It's had a fair number of contributors.
  3. It has had weekly meetings to coordinate progress for the last several months.
  4. Its seen some usage already, despite little publicity and despite D8 core's REST: https://www.drupal.org/project/usage/jsonapi. That usage includes production sites.
  5. Wim Leers has done a comprehensive review to evaluate core readiness (#2829327: Review JSON API module for core-readiness), some improvements have already landed, and the most important architectural issue is being worked on right now (#2829398: Clean up JsonApiResource: the annotation, plugin type, plugin manager, plugin implementations, the dynamic routes generator and the request handler).
  6. It has received a 1.0-alpha3 release — if it weren't for #2829327, it'd have already been in beta stage.

We believe that JSON API may be a natural complement to the REST module. (As does Dries: http://buytaert.net/improving-drupal-8-api-first-json-api-oauth2).

The REST module will always have its own place, with its ability to support multiple formats, any URL structure, and custom REST resources. JSON API aims to make the 90% use case better.

The JSON API module completely solves the first 3 fundamental pain points of the REST module! The fourth pain point is one that is the same for both the REST module and the JSON API module — fixing it for one also fixes it for the other! In other words: a step forward on all fronts for a very common use case.

Note that JSON API is layering on top of the Entity API (exposing all its data), the serialization API (for its serialization needs), the routing API (for its routes) and the authentication provider API (to allow additional authentication providers to be used). That's why it's able to have zero config. And that's why it's easy to to have it as an experimental module: it can be installed and uninstalled at any time.

Remaining tasks

  1. Build consensus that adding JSON API to core is a good idea. — see #18.
  2. Formulate detailed plan. — see #2842148: [PP-2] Next steps: the path to stability for the JSON API module — added in #20.

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

e0ipso created an issue. See original summary.

e0ipso’s picture

The above text was crafted by Wim Leers. I wish I had the skill to write issues as detailed as this one, … maybe some day.

Wim Leers’s picture

But I was only doing it for e0ipso, who is the one who got the JSON API module to the point it's currently at!

thebruce’s picture

This is very well put. Another thing I've seen and which I think there is some general agreement on about HAL is that it serves really well for read only scenarios but begins to fall down as soon as you need other methods than GET as the available HATEOAS descriptors in HAL support that less well. JSON API however is much friendlier in that respect.

I also agree with Matteu, or Wim, well both, that JSON API is going to be a great 90% solution and that REST will still have a place for its agnosticism in format output, which will allow us to grow as new formats come into prominence.

dawehner’s picture

One really huge point of us getting JSON API in is that it would allow us to dogfood our own APIs easier. When a module, let's say views_ui2, adds a dependency to jsonapi, it can rely on the resources being available and feature complete, as outlined in the issue summary. This should really enable us building different admin UIs on the longrun.

Fabianx’s picture

+1 for JSON API in core and yes for new front-end we need something like that.

fgm’s picture

It's a pity the current GraphQL module is not yet at a stage where it would be considered for the role : beyond the current state of the D8 implementation, it seems to have /lots/ more traction in the general world, and focusing on JsonAPI feels a bit like entering yet another not-so-popular solution just because we happen to have it right now. And it focuses on the same issue, but tackles it more in-depth (schema introspection, non-entities).

More generally, both modules have to expose the entity/fields system (actually the complete typeddata system for GraphQL) : IMHO the proper solution would be to have core provide the general lower-level ability to expose the entity/fields graph at the PHP level, and let contrib provide the serializations. I even seem to remember this being discussed at DDD Milan ?

andypost’s picture

@fgm There's typed data & entity query so introspection is here) but neither views nor other "serialization" modules able to map this knowledge somehow

IMO jsonapi query syntax is more human-readable so +1 here

e0ipso’s picture

@fgm I don't see this as GraphQL vs JSON API. As stated in #2757967: API-first initiative GraphQL is still under consideration at the same level of JSON API.

I am not sure about the adoption rates you state, since JSON API is huge in the RoR and Ember communities. I have personally only worked with JSON API + React, so I may be wrong. Comparing both adoption rates would require to define a metric (do we favor number of requests, number of apis, number of 3rd party libraries, do we include generic REST libraries or not, …), and then perform a measure. That sounds like a lot of work to measure something that will change very quickly. I would be curious to see the results, but I don't think that this should go in detriment of any particular option. I really don't want this to become the focus of the discussion here, because it may distract from the other points under consideration.

We believe that there are many benefits in having this as an experimental module (as stated in the issue summary and comments), beyond the popularity the spec versus other options.

My goal is that I'd like to see decoupled Drupal thrive like never before, because I really believe that it's where the future of Drupal is. For that GraphQL plays an immense role.

dawehner’s picture

More generally, both modules have to expose the entity/fields system (actually the complete typeddata system for GraphQL) : IMHO the proper solution would be to have core provide the general lower-level ability to expose the entity/fields graph at the PHP level, and let contrib provide the serializations. I even seem to remember this being discussed at DDD Milan ?

We have been there and we kinda agreed that you basically implement a system which is super generic and suiteable for GQL. Especially the complexity to achieve something like that is insane compared to solve concrete problem with jsonapi.

tkoleary’s picture

Re #2 and #3. Now that you're done patting each other on the back, let me pat you both on the back. :)

Great stuff.

redbrickone’s picture

We're using JSON API on a project right now and it's great. Planning on using it on all of our future headless D8 projects. Definitely add to core :)

postporncig’s picture

I love this module , would be nice if gets added to experimental core modules , i'm sure so many people going to adopt it

e0ipso’s picture

Completely unscientific poll: Check out @e0ipso's Tweet: https://twitter.com/e0ipso/status/815107293335404544?s=09

Wim Leers’s picture

In case the link in #14 stops working at some point:

  • the question was: If we add @JSONAPI in #Drupal core, will it increase the chances to choose it for your application's backend?
  • 70% voted yes, 30% voted no
  • 127 votes
Wim Leers’s picture

Issue summary: View changes

Adding a small clarification to the IS that I forgot when I initially wrote it.

Wim Leers’s picture

Issue summary: View changes

At the request of Josh Koenig, embedded a video showing JSON API's cleaner output.

This unfortunately requires me to use the Full HTML text format, meaning that only documentation maintainers are able to edit the issue summary now.

Wim Leers’s picture

Assigned: Unassigned » Wim Leers

The first remaining task is: Build consensus that adding JSON API to core is a good idea.

I think that the many supportive comments here combined with the Twitter poll at #14/#15 and its results are pretty strong signals that there is consensus that adding this to core is a good idea. 90 people saying that adding JSON API to core causes them to be more likely to pick Drupal 8 to power the back-end, that's pretty significant out of 127 respondents!

So, marking that task as done.

Next up: the final remaining task, Formulate detailed plan. — working on that now.

Wim Leers’s picture

Issue summary: View changes

Now actually marked Build consensus that adding JSON API to core is a good idea. as done.

Wim Leers’s picture

Wim Leers’s picture

Wim Leers’s picture

Status: Active » Reviewed & tested by the community

Apparently I need to move this to RTBC if we have community consensus. Per #18, that's done. And since we now also have a plan, I think it's totally fair to move this to RTBC.

Wim Leers’s picture

e0ipso opened the corresponding core issue that contains the patch that is adding the JSON API module to Drupal core: #2843147: Add experimental JSON API module.