Problem/Motivation

To explain why having something like a "JSON API" module in core makes sense, I first have to explain the process/choices that led to the current REST module in D8. I will explain later what "JSON API" is.

REST in D8: a short history

2012: Humble beginnings
Back in the days when Drupal 8 was in active development, we chose to add rest.module to Drupal core, inspired by the https://www.drupal.org/project/restws contrib module in Drupal 7.

The scope of that issue is clear: it's minimal. It only does DELETE, not yet GET, POST or PATCH. It hardcodes a single format: JSON-LD (the up-and coming standard back then). It only supports content entities and one other resource (watchdog/dblog), to prove that it's possible. This is a fine starting point!

The question of whether to support collections, paging and queries is raised at #1816354-22: Add a REST module, starting with DELETE, and is answered that collections and paging are a problem for Views plugins, because Views is how we build collections — this also makes sense.

This was done in #1816354: Add a REST module, starting with DELETE (committed November 2012).

2013: Default format: from JSON-LD to HAL+JSON
The default format was first JSON-LD. It was then decided to change that to HAL+JSON. Over time it had become clear that shipping a perfect "primary format" was unnecessary, because Symfony's Serializer component makes it easy to add new formats. The primary use case for the REST module was expected to be content deployment. HAL was the simplest format with hypermedia support. Because of the involvement of JSON Schema proponents, HAL+JSON was chosen, because JSON Schema can be layered on top of HAL+JSON.

This was done in #1924220: Support serialization in hal+json (committed March 2013).

2014: no significant changes (mostly updates for D8 API/best practice changes)
2015: no significant changes (mostly updates for D8 API/best practice changes)
2016: #2721489: REST: top priorities for Drupal 8.2.x + #2794263: REST: top priorities for Drupal 8.3.x
The test coverage for the rest, serialization and hal modules was insufficient to guarantee correct behavior and good DX for users of D8's REST API. See the numerous issues listed in those two plan issues.
This is fine and understandable — all of the above was built by volunteers and limited sponsoring. Drupal 8's APIs changed a lot since 2012! The popularity of the HAL format didn't take off as expected.

REST in D8: fundamental pain points after one year

After one year of having Drupal 8 released, some pain points have become clear:

  1. Difficult to set up. From the extra layer of permissions (fixed by #2664780: Remove REST's resource- and verb-specific permissions for EntityResource, but provide BC and document why it's necessary for other resources) to the complex configuration (simplified by #2721595: Simplify REST configuration): it was far too hard. We've made it better in 8.2, but even with https://www.drupal.org/project/restui to make it simpler, it's more hassle than necessary for the 90% case: REST access to entities.
  2. Collections: even harder to set up. For every single collection that you want to access, you need to create a separate view. If you need to access five entity types, you need to set up five views. As you can see in the history, this was intentional, but it leads to silly/frustrating situations. Like the very common case of having an article with comments, and it literally being impossible to GET all comments for an article, because you need to set up a (relatively advanced) view for that. Not to mention that even when you set up a view to provide access a collection, you still don't have pagination: have 1 million nodes? You'll get all of them in a single response, except the request will time out.
  3. Easy to shoot yourself in the foot. Due to combinations of circumstances, the URL structures are confusing (GET/PATCH/DELETE /node/42, but POST /entity/node), forgetting a particular request header or an entity field results in incomprehensible error responses, and figuring out capabilities (such as embedding related resources) requires debugging the PHP code.
  4. Incomplete normalizations. This means you simply cannot access certain information. For example, translated entities, modifying config entities, file URLs, file uploads, image style URLs, parent term, entity paths, and so on. We are fixing those, which will also help the REST module and any other module providing RESTful web services.

We can fix #4, and doing so will help all modules providing RESTful web services.

The first 3 are fundamentally unsolvable, because that's how it was designed.

JSON API: the spec

JSON API is a specification: http://jsonapi.org/.

Like HAL, it describes a particular normalization. But on top of that, it also prescribes:

  1. how to handle collections: http://jsonapi.org/format/#fetching-resources
  2. how to handle filtering: http://jsonapi.org/format/#fetching-filtering
  3. how to handle sorting: http://jsonapi.org/format/#fetching-sorting
  4. how to handle relationships: http://jsonapi.org/format/#fetching-relationships + http://jsonapi.org/format/#crud-updating-relationships
  5. how to handle errors: http://jsonapi.org/format/#error-objects
  6. an extension system is in development: http://jsonapi.org/extensions/ — but the spec already ensures that it's possible to layer this on top: http://jsonapi.org/format/#content-negotiation

In other words: it is more restrictive, it defines much more of how a RESTful web service using it should work than the HAL specification (zero mentions of "collections".

Put differently: JSON API has a broader opinion, which removes the need for Drupal to have an opinion. This in turn allows developers to use non-Drupal-specific tools to consume Drupal 8's APIs.

JSON API: module

e0ipso developed the JSON API contrib module: https://www.drupal.org/project/jsonapi. It's JSON API, applied to Drupal entities.

It made some choices:

  1. Only entities. Because only entities are guaranteed to have collections and relationships. And because it's the 99% use case.
  2. Rely on the Entity (Field) Access API. This means we can expose all entity types via JSON API by default: the responsibility lies with Entity API to ensure access is set up correctly — the rest of Drupal already relies on this too.
  3. All installed authentication providers are allowed.
  4. It doesn't expose the data structures that Drupal uses internally for entities + fields, unlike the REST module. See: https://www.youtube.com/embed/cDDSdIyjVkM

Together those decisions mean that installing the JSON API module means instant, format-standardized access to all entities in Drupal 8 without any configuration, via any of the available authentication providers.

Painless setup. Zero config (no need for documentation). A spec to look at to see what's possible (not Drupal-specific documentation).

Proposed resolution

As part of the API-first initiative (#2757967: API-first initiative), we'd like to propose to add JSON API to Drupal 8.3 core as an experimental module. This should come as no surprise because #2757967 has mentioned it since being published on June 29.

e0ipso is the maintainer of the JSON API contrib module. It's not an early-stages module:

  1. It's been under development since May 2016.
  2. It's had a fair number of contributors.
  3. It has had weekly meetings to coordinate progress for the last several months.
  4. Its seen some usage already, despite little publicity and despite D8 core's REST: https://www.drupal.org/project/usage/jsonapi. That usage includes production sites.
  5. Wim Leers has done a comprehensive review to evaluate core readiness (#2829327: Review JSON API module for core-readiness), some improvements have already landed, and the most important architectural issue is being worked on right now (#2829398: Clean up JsonApiResource: the annotation, plugin type, plugin manager, plugin implementations, the dynamic routes generator and the request handler).
  6. It has received a 1.0-alpha3 release — if it weren't for #2829327, it'd have already been in beta stage.

We believe that JSON API may be a natural complement to the REST module. (As does Dries: http://buytaert.net/improving-drupal-8-api-first-json-api-oauth2).

The REST module will always have its own place, with its ability to support multiple formats, any URL structure, and custom REST resources. JSON API aims to make the 90% use case better.

The JSON API module completely solves the first 3 fundamental pain points of the REST module! The fourth pain point is one that is the same for both the REST module and the JSON API module — fixing it for one also fixes it for the other! In other words: a step forward on all fronts for a very common use case.

Note that JSON API is layering on top of the Entity API (exposing all its data), the serialization API (for its serialization needs), the routing API (for its routes) and the authentication provider API (to allow additional authentication providers to be used). That's why it's able to have zero config. And that's why it's easy to to have it as an experimental module: it can be installed and uninstalled at any time.

Remaining tasks

  1. Build consensus that adding JSON API to core is a good idea. — see #18.
  2. Formulate detailed plan. — see #2842148: The path for JSON API to "super stability" — added in #20.

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

e0ipso created an issue. See original summary.

e0ipso’s picture

The above text was crafted by Wim Leers. I wish I had the skill to write issues as detailed as this one, … maybe some day.

Wim Leers’s picture

But I was only doing it for e0ipso, who is the one who got the JSON API module to the point it's currently at!

thebruce’s picture

This is very well put. Another thing I've seen and which I think there is some general agreement on about HAL is that it serves really well for read only scenarios but begins to fall down as soon as you need other methods than GET as the available HATEOAS descriptors in HAL support that less well. JSON API however is much friendlier in that respect.

I also agree with Matteu, or Wim, well both, that JSON API is going to be a great 90% solution and that REST will still have a place for its agnosticism in format output, which will allow us to grow as new formats come into prominence.

dawehner’s picture

One really huge point of us getting JSON API in is that it would allow us to dogfood our own APIs easier. When a module, let's say views_ui2, adds a dependency to jsonapi, it can rely on the resources being available and feature complete, as outlined in the issue summary. This should really enable us building different admin UIs on the longrun.

Fabianx’s picture

+1 for JSON API in core and yes for new front-end we need something like that.

fgm’s picture

It's a pity the current GraphQL module is not yet at a stage where it would be considered for the role : beyond the current state of the D8 implementation, it seems to have /lots/ more traction in the general world, and focusing on JsonAPI feels a bit like entering yet another not-so-popular solution just because we happen to have it right now. And it focuses on the same issue, but tackles it more in-depth (schema introspection, non-entities).

More generally, both modules have to expose the entity/fields system (actually the complete typeddata system for GraphQL) : IMHO the proper solution would be to have core provide the general lower-level ability to expose the entity/fields graph at the PHP level, and let contrib provide the serializations. I even seem to remember this being discussed at DDD Milan ?

andypost’s picture

@fgm There's typed data & entity query so introspection is here) but neither views nor other "serialization" modules able to map this knowledge somehow

IMO jsonapi query syntax is more human-readable so +1 here

e0ipso’s picture

@fgm I don't see this as GraphQL vs JSON API. As stated in #2757967: API-first initiative GraphQL is still under consideration at the same level of JSON API.

I am not sure about the adoption rates you state, since JSON API is huge in the RoR and Ember communities. I have personally only worked with JSON API + React, so I may be wrong. Comparing both adoption rates would require to define a metric (do we favor number of requests, number of apis, number of 3rd party libraries, do we include generic REST libraries or not, …), and then perform a measure. That sounds like a lot of work to measure something that will change very quickly. I would be curious to see the results, but I don't think that this should go in detriment of any particular option. I really don't want this to become the focus of the discussion here, because it may distract from the other points under consideration.

We believe that there are many benefits in having this as an experimental module (as stated in the issue summary and comments), beyond the popularity the spec versus other options.

My goal is that I'd like to see decoupled Drupal thrive like never before, because I really believe that it's where the future of Drupal is. For that GraphQL plays an immense role.

dawehner’s picture

More generally, both modules have to expose the entity/fields system (actually the complete typeddata system for GraphQL) : IMHO the proper solution would be to have core provide the general lower-level ability to expose the entity/fields graph at the PHP level, and let contrib provide the serializations. I even seem to remember this being discussed at DDD Milan ?

We have been there and we kinda agreed that you basically implement a system which is super generic and suiteable for GQL. Especially the complexity to achieve something like that is insane compared to solve concrete problem with jsonapi.

tkoleary’s picture

Re #2 and #3. Now that you're done patting each other on the back, let me pat you both on the back. :)

Great stuff.

redbrickone’s picture

We're using JSON API on a project right now and it's great. Planning on using it on all of our future headless D8 projects. Definitely add to core :)

postporncig’s picture

I love this module , would be nice if gets added to experimental core modules , i'm sure so many people going to adopt it

e0ipso’s picture

Completely unscientific poll: Check out @e0ipso's Tweet: https://twitter.com/e0ipso/status/815107293335404544?s=09

Wim Leers’s picture

In case the link in #14 stops working at some point:

  • the question was: If we add @JSONAPI in #Drupal core, will it increase the chances to choose it for your application's backend?
  • 70% voted yes, 30% voted no
  • 127 votes
Wim Leers’s picture

Issue summary: View changes

Adding a small clarification to the IS that I forgot when I initially wrote it.

Wim Leers’s picture

Issue summary: View changes

At the request of Josh Koenig, embedded a video showing JSON API's cleaner output.

This unfortunately requires me to use the Full HTML text format, meaning that only documentation maintainers are able to edit the issue summary now.

Wim Leers’s picture

Assigned: Unassigned » Wim Leers

The first remaining task is: Build consensus that adding JSON API to core is a good idea.

I think that the many supportive comments here combined with the Twitter poll at #14/#15 and its results are pretty strong signals that there is consensus that adding this to core is a good idea. 90 people saying that adding JSON API to core causes them to be more likely to pick Drupal 8 to power the back-end, that's pretty significant out of 127 respondents!

So, marking that task as done.

Next up: the final remaining task, Formulate detailed plan. — working on that now.

Wim Leers’s picture

Issue summary: View changes

Now actually marked Build consensus that adding JSON API to core is a good idea. as done.

Wim Leers’s picture

Issue summary: View changes
Wim Leers’s picture

Wim Leers’s picture

Status: Active » Reviewed & tested by the community

Apparently I need to move this to RTBC if we have community consensus. Per #18, that's done. And since we now also have a plan, I think it's totally fair to move this to RTBC.

Wim Leers’s picture

e0ipso opened the corresponding core issue that contains the patch that is adding the JSON API module to Drupal core: #2843147: Add JSON:API to core as a stable module.

Wim Leers’s picture

Quoting #2843147-36: Add JSON:API to core as a stable module:

Per https://twitter.com/webchick/status/839240958105432065, since March 7, the decision was communicated by core committers (sadly not in a very clear way, sadly not here, sadly not in the "ideas" issue queue…) that no additional experimental modules would be added in 8.4.x.

Consequently, the JSON API module must remain a contrib module for now. Which also has a bright side: we can iterate much faster in contrib. And we're already seeing a massive uptick in JSON API usage: from 240 on March 5, to 982 on April. In between those two times, the first beta release of JSON API was published, and this seems to ad adoption significantly. So, we're getting lots of contrib validation before moving it into core, which is better anyway.

The mere existence of this issue, and the publicity around it, have already helped a lot with awareness. That's in the end the key reason for wanting to add JSON API to Drupal core: to make more people aware, to lower the bar to find this module, which we believe is a better fit/offers a better DX than the REST module for many (most?) use cases.

For now, closing this issue — I will reopen this once it's possible again to add an experimental module to Drupal core.

tim.plunkett’s picture

Issue summary: View changes

Removed Full Html filter (and iframe) as that prevents anyone from commenting/editing.

Wim Leers’s picture

@tim.plunkett++

Wim Leers’s picture

Status: Reviewed & tested by the community » Postponed
smndey’s picture

I am about to build a headless drupal application and I was bit confused between Drupal 8 core REST API and JSON API. Now choice is clear to use JSON API as an API first approach. Well explained. Thanks.

webchick’s picture

Back in ~August, this issue was tentatively added to the Drupal 8.5 roadmap by the API-First team: https://www.drupal.org/core/roadmap#api-first

We're about mid-way through the development cycle (feature freeze is the week of Jan 17); does this still feel like a reachable goal for 8.5, or should we move to 8.6?

Wim Leers’s picture

I was supposed to work a lot on JSON API this past week, but lots of core API-First blockers landed, so I ended up working on REST/Serialization issues all week. I will be working a lot on it. #2842148: The path for JSON API to "super stability" describes what the key things to do are to get JSON API to "core-worthy stable". @gabesullice has landed the most significant step forward in the past few months.

e0ipso’s picture

I'd like to have a conversation about this this week. While my instinct is on caution, I'd like to push forward as soon as possible. JSON API can perfectly live in contrib for sites to build upon. Many have done so in the last years.

However if we want to experiment on building better UIs for Drupal core on React (or else) we should do it based on JSON API. Doing so will help with the eventual inclusion of JSON API in core and with better DX with the implementation of these UIs. There is also some confusion (like smndey is showing in #28) that we want to clarify sooner.

I'll argue that many of the remaining in #2842148: The path for JSON API to "super stability" are nice-to-haves and not a blockers now.

However, I see the other side of the coin too. That's why I'd like to discuss this before coming to a conclusion.

e0ipso’s picture

We met on Monday to talk about this: https://youtu.be/lbq7qmdi0SY

TL;DR

Agreements:

  • We still think that there is value of having it into core, not just contrib. We advertise Drupal and we say that you can build decoupled apps with just core. While that is true, many agree that only REST core is not enough for a modern decoupled app.
  • If it goes in 8.5 we'd like to see it as experimental. If it slips further the goal is to include it as stable. I both cases we shoot for 8.6 as JSON API as stable in core.
  • We agreed that we will keep the current API surface to the minimum: Drupal services + the HTTP API. JSON API Extras will be the module exercising those integration points for extensibility, we will discover missing public APIs with it.

Daniel raised the concern of core committer availability to deal with this in 8.5, although he also prefers to shoot for 8.5 if possible.

dawehner’s picture

Functionality wise I think JSON API totally makes sense. People are having a good time using it in contrib, but as #28 points out, there is confusion about what is the recommended way. There are days I wish our help in core would include not just help for modules, but actual help for users, like general recommendations / documentation what Drupal is, how you build sites, and for example how to get data out of Drupal.

Making it clear that JSON API will be the primary option to build decoupled sites would be huge for all the possible reasons listed here. Personally I still think though, that we need to find a usecase inside core for it. Otherwise we don't have the safety whether it actually works as expected. One huge mistake which was done with the REST module was the missing real world experience. For example quickedit module could use jsonapi, when its available.

Timewise I think we should "simply" work on it, things are done, when they are done. Promising something adds unnecessary stress for a potential slightly earlier release.

Before doing though it would be nice though to find a framework maintainer, who is willing to follow the process from start to end.

I'm sorry for being less involved that I thought I could do it. Well you know, work and other things are sometimes simply more important.

e0ipso’s picture

This part of #33 caught my attention:

Personally I still think though, that we need to find a usecase inside core for it.

I totally agree. My opinion is that we don't need JSON API for anything at the moment. As the needs for a web service arise we can keep adding hard coded controllers (for the lack of a better name). However given that the React in Core Initiative will likely need to make Ajax calls, I think it would be a fantastic way to dogfood our API. Advanced admin UIs will likely need to interact with entities via Ajax, and JSON API is one of the best solutions for that. In this scenario, JSON API benefits because its APIs are exercised, and fancy admin UIs benefit because they don't need to write a myriad of hard coded controllers. In fact, JSON API is the generic implementation of a myriad of hardcoded controllers for entities.

Note that these initiatives could totally be independent, but I think there is a good change of mutual benefit if we think that's the best approach.

Before doing though it would be nice though to find a framework maintainer, who is willing to follow the process from start to end.

@webchick do you think you can help us with that?

Timewise I think we should "simply" work on it, things are done, when they are done.

While I agree with that, I assume we need to allocate resources (framework maintainer, volunteer planification, …). I think planning can help with that. It's important to notice that volunteers make soft commitments (that can vary unexpectedly), we cannot make promises. That applies at all times.

Making it clear that JSON API will be the primary option to build decoupled sites would be huge for all the possible reasons listed here.

That's a very good call. @Wim Leers (and everyone) would you agree with that? If there is consensus, we should create documentation issues to reflect that.

Wim Leers’s picture

Status: Postponed » Needs review
Related issues: +#2931785: The path for JSON API to core

In my opinion, the JSON API module is close but not yet quite ready to be moved into Drupal core. I think everything in #2931785: The path for JSON API to core needs to be resolved first. In #30, I said #2842148: The path for JSON API to "super stability" contained that list of things, but that also contained nice-to-haves. #2931785 contains only must-haves.

At least, based on my experience A) as somebody who added new modules to core, B) as a REST/Serialization module maintainer, C) a core component maintainer in general, I came up with that list of must-haves. JSON API maintainers @e0ipso and @gabesullice agree with that list, at least mostly: @e0ipso disagrees about one issue, see #2931785-15: The path for JSON API to core.

Marking needs review to get feedback from a core framework manager.

e0ipso’s picture

It's been a while since we've had real movement in this issue. However many things have happened in the JSON API module.

This has been in the roadmap for Drupal core for a while now. See #29, so there is implicit agreement from the product managers that we want JSON API in core (and we are close).

I understand that when an issue in the Ideas queue comes to agreement to move forward, then it should be fixed. If that's the case, can we fix this one? We can track further progress with code in: #2843147: Add JSON:API to core as a stable module and #2931785: The path for JSON API to core (which is almost complete).

Sorry for not staying on top of the process.

webchick’s picture

Just like any other queue, if it feels like this issue has community consensus, feel free to mark it "RTBC" and we'll review on our weekly product management meetings. :)

e0ipso’s picture

Status: Needs review » Reviewed & tested by the community

Ah! Yes. Thanks for the clarification @webchick. I'll reach out to see if others think it's RTBC as well.

gabesullice’s picture

+1 🎉

dawehner’s picture

+1!

justafish’s picture

+1 ⬅️✌️➡️

prestonso’s picture

+1!!

Wim Leers’s picture

#2931785: The path for JSON API to core is indeed getting very close. The TL;DR of that meta issue is:

  1. 100% done: Explicit JSON API spec compliance testing.
  2. 95% done: JSON API's test coverage now rivals core REST's test coverage (see https://wimleers.com/blog/api-first-drupal-really), so we can be very confident. (That we'd know about regressions before shipping them and that there's no security vulnerabilities.)
  3. 100% done: The most significant (and also only!) aspect in which JSON API's DX was inferior to core REST, was the reason for a 403. Now it provides that too.
  4. 50% done: API-First ecosystem: ensure that JSON API maximally reuses normalizers written by contrib modules, and respects core's metadata (e.g. isInternal()) and limitations (e.g. modifying config entities is very risky).
  5. 100% done: maintainability: very very clearly delineate the API surface, limit it as much as possible, and document the hell out of (the rationale behind) the architecture.

So: +1!

skyredwang’s picture

Status: Reviewed & tested by the community » Needs review
dawehner’s picture

Maybe this is just me, but to be honest I think software has bugs, and they can be fixed. The idea that everything is perfect always seems weird :)

gabesullice’s picture

Status: Needs review » Reviewed & tested by the community

I don't think this issue needs to be blocked because a bug was discovered.

This issue is really about whether we all agree on the "idea" of moving JSON API into core and we reached consensus on that.

So, I think now the proper place to voice concerns about the stability of the module for core inclusion is: #2843147: Add JSON:API to core as a stable module

However, even that issue is really supposed to be high-level as @dawehner pointed out.

effulgentsia’s picture

This is an Idea issue. +1 to the Idea.

it would be nice though to find a framework maintainer, who is willing to follow the process from start to end

I am.

Once this Idea issue is approved (marked Fixed), I think it would be helpful to create a new "Proposed Plan" issue. I think that could link to #2931785: The path for JSON API to core, but also include thoughts about what within core should be the first (and also second?) consumer of it (QuickEdit? something else?). If the first and/or second logical consumer is something that requires editing config entities, then we should also include links to issues that need to be fixed to enable that (e.g., config entity validation, etc.). And finally, thoughts about what the transition might look like from users of the stable contrib module to the experimental core module might be good to include in the plan as well.

webchick’s picture

Status: Reviewed & tested by the community » Fixed
Issue tags: -Needs product manager review

Product management also has no concerns, and is in fact very excited about adopting well-known standards. Onward! :)

e0ipso’s picture

🎊

gabesullice’s picture

@effulgentsia, I think the issue you're talking about is: #2843147: Add JSON:API to core as a stable module

I'll update the issue summary :)

Also...

🎉🎉🎉🎉🎉🎉🎉🎉🎉

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.