Jump to:


Background

Making Drupal API-first means making the data stored and managed by Drupal available for other software. It allows for:

  1. new types of integration (those not defined in Drupal/PHP code)
  2. full decoupling
  3. progressive decoupling (e.g. the overall page is still served by Drupal, but certain parts of the page provide highly interactive, JS-driven/enhanced interactions, which need to get certain data, which they can via REST)

Therefore it's very important that we advance Drupal's web services.


Drupal 8.0 shipped with a REST API. Unfortunately, it's in a state where it's very difficult to set it up (configuration is hard) and use (DX is hard). There are many subtle bugs (test coverage is weak). However, in many cases, it is "just" a matter of polish.

At the time, Drupal 8 selected HAL+JSON as the default format. However, since then, the JSON API spec has been finalized and matured. Roughly speaking, it has two things going for it that HAL+JSON does not:

  1. it is highly opinionated/prescriptive, which means better interopability and less bikeshedding about details (many things not yet supported in D8's REST have standardized mechanisms/conventions in JSON API)
  2. an extension negotiation system

Drupal 8's REST API's "Entity" REST resource does not support revisions, and it uses IDs, not UUIDs. RELAXed Web Services adds support for both of those, which makes it possible to use REST for complex workflows. Or for replicating content in their entirety (thanks to CouchDB compatibility): for deployment purposes or for offline-first apps/sites (using PouchDB).

What's also problematic, is that Drupal 8 only supports two authentication mechanisms out of the box: Basic Auth and Cookie (which only works very well for browsers, not apps — so only really suitable for progressively decoupled use cases). The default expectation for front-end developers and native app developers is OAuth2 support.

A non-RESTful data exchange API has also since emerged: GraphQL. It's not very well suited for use cases like syncing data, but it's extremely well suited for the use case of fetching data to be used in UI components. In the long run, GraphQL might become part of how Drupal 9 renders itself, but long before then it'd allow front-end developers to already start using it.

Addressing all of the above means making Drupal 8 an unparalleled API-first platform:

All data available:

  • RESTfully
    • in any format (JSON, HAL+JSON …)
    • in multiple flavors ("HTTP purist", jsonapi.org …)
  • non-RESTfully
    • via GraphQL
    • via CouchDB protocol, for replication & syncing

… using any authentication mechanism (Cookie, Basic Auth, OAuth2)

Goals

Must have
Should have
Could have

Process and where to find it

Monthly meeting at https://plus.google.com/hangouts/_/acquia.com/drupal-8-2-web. Notes: https://docs.google.com/document/d/1ZNfUjBg2O0gUxslt99zhh33zX5n3dkuilhGP....

REST
Work is being done at https://drupal.org/project/issues/drupal?component=rest.module, plus the associated issue queues: https://drupal.org/project/issues/drupal?component=serialization.module and https://drupal.org/project/issues/drupal?component=hal.module
(Module already lives in core, arguably should have been experimental.)
JSON API
Work will be done at https://www.drupal.org/project/jsonapi to first get a stable contrib module during the 8.2+8.3 cycles.
Weekly meetings on Wednesday at https://plus.google.com/u/0/events/cho7spo7r4tlp7kb4dvor0p4ggc
During the 8.3 cycle, it will be moved to an experimental core module. Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core.
UUID & revision support
TBD
GraphQL
Work will be done at https://www.drupal.org/project/graphql to first get a stable contrib module during the 8.2 cycle, possibly longer when it happens (since there are currently no active maintainers).
During the 8.3 cycle (at earliest), it will be moved to an experimental core module. TBD
OAuth 2.0 Bearer Token
Work will be done at https://www.drupal.org/project/simple_oauth to first get a stable contrib module during the 8.2+8.3 cycles.
During the 8.3 cycle, it will be moved to an experimental core module. Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core.
CouchDB support
TBD

Proposed roadmap

Much work can happen in parallel!

Each of the following bullets lists parallel tracks, and within the bullet a sequence is defined:

  • R1 -> R2A + R2B
  • J1 -> J2
  • G1 -> G2
  • JG
  • O1 -> O2
Phase R1: REST usable (8.2) DONE!
Address all top priorities (#2721489: REST: top priorities for Drupal 8.2.x), and improve test coverage significantly (#2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method) so we can have very strong guarantees that it works as expected. (Test confidence level should be similar to that of the BigPipe module.)
Phase R2A: REST best-in-class (8.3+8.4)
See #2794263: REST: top priorities for Drupal 8.3.x + #2852860: REST: top priorities for Drupal 8.4.x.
Details TBD. Things being considered: collections, pagination, link relations, entity metadata (fields required vs optional, cardinality, validation constraints — see #2665602: Implement an API for REST clients to know which fields are editable), "playground" UI, auto-generated REST API docs, configuration UI.
Phase R2B: REST UUID + revision support (8.4)
This will be addressed in part or in whole by the Workflow Initiative — see #2721129: Workflow Initiative. Details TBD.
Phase J1: JSON API stable in contrib (8.2, 8.3)
Address all concerns in #2829327: Review JSON API module for core-readiness. DONE!
Currently in RC!
#2842148: [PP-2] Next steps: the path to stability for the JSON API module
Phase J2: move JSON API into core (when: ?)
Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core. Details TBD.
Phase G1: GraphQL stable in contrib (no timeframe, because lack of maintainers)
Details TBD.
Phase G2: move GraphQL into core (>=8.3)
Details TBD.
Phase JG: factor a shared component out J[1,2] & G[1,2] (8.2)
Already happening to some extent at https://github.com/drupal-http-apis/type-graph
Phase O1: OAuth 2.0 Bearer Token stable in contrib (8.3)
Phase O2: move OAuth 2.0 Bearer Token stable into core (>=8.4)
#2834718: New experimental module: OAuth2
Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core.

Not in scope

TBD

Existing core features

The REST module.

Contributed projects

Team and resources

Initiative coordinator: Wim Leers

REST (usable + best-in-class)
tedbow, dawehner, Wim Leers
More people welcome — we provide mentoring :)
Subcoordinator: Wim Leers
JSON API
e0ipso, gabesullice, dawehner, Wim Leers
Subcoordinator: e0ipso
GraphQL
fubhy (contrib module maintainer), others TBD
OAuth 2.0 Bearer token
e0ipso (contrib module maintainer), others TBD
REST UUID + revision support
TBD, hopefully the Workflow Initiative does most of the heavy lifting, and we just need to help out with reviews.

Sign-offs

[This section still needs work]

  • REST: Crell, dawehner, klausi, Wim Leers
  • JSON API: e0ipso, Ed Faulkner, gabesullice, klausi, Wim Leers

Alex Pott has stepped up and is showing interest, so we probably want his sign-off as well.

Signoffs needed

TBD

Signoffs given

TBD

Comments

Wim Leers created an issue. See original summary.

neclimdul’s picture

Just for clarity, Oauth actually has disclosure concerns over HTTP as well(the module mentions this) so we can't really get away from that reality. It is definitely something we should be supporting though. Documentation in the OAuth RFC http://tools.ietf.org/html/rfc6750#section-5

That said 100x yes to every bit of this. :-D

dawehner’s picture

+1 for not mentioning the type graph idea. Its a really abstract concept with a lot of effort before it provides actual value to people, unlike all the other efforts which happen at the same time.

Wim Leers’s picture

Category: Bug report » Plan

Oops :)

webflo’s picture

Issue summary: View changes
Wim Leers’s picture

Title: Web services initiative » API-first initiative
timmillwood’s picture

I hope we (the workflow initiative) can help make things work well with revisions, as a contributor to the RELAXed Web Services contrib module I'm keen to make sure that everything still works there.

dawehner’s picture

Phase 8.3:

I have one general question . Do we want to expose multiple collection capabilities by default? One could say, when we expose a collection resource, we introduce some custom standard for filtering/sorting, so you could also just use another standard like JSONAPI in the first place. Providing collection of bundles is a non trivial problems, as for example, documenting the available properties is no longer clear, as they change on a per bundle level. Given that collection/pager feature could be something we expose just via jsonapi/graphql, but I agree this could be a more radical approach.

e0ipso’s picture

Some extra context about what @dawehner is asking can be found in https://youtu.be/mveQFnm8S1E?t=38m19s.

kalinchernev’s picture

Issue summary: View changes

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

e0ipso’s picture

JSON API update

The jsonapi module is progressing steadily. There are several contributors that are currently using the module reporting (and fixing!) bugs.

This is good news for the goal of «get a stable contrib module during the 8.2 cycle».

Wim Leers’s picture

@e0ipso: Nice! :)

rest.module update

I've also got news from the rest.module side. I closed #2721489: REST: top priorities for Drupal 8.2.x and opened #2794263: REST: top priorities for Drupal 8.3.x, which is the new issue to look at for progress in rest.module as part of this initiative in the next six months.

I'm proud of the progress we've made. We've tackled lots of tricky problems. rest.module is in a much better place now! We fixed the majority of things listed here.

Note that of the 34 issues are tagged with 8.2.0 release notes, 9 are for REST module!, that's more than 25%!

Wim Leers’s picture

Issue summary: View changes
fgm’s picture

Note that there is a week-long GraphQL sprint at Drupalcon Dublin, from today to next Sunday : you can register on the Drupalcon Dublin sprint attendance worksheet.

e0ipso’s picture

The Simple OAuth 8.x-2.x module has been upgraded to use the http://oauth2.thephpleague.com PHP library.

It has been architected in a way that the authorization grants are plugins, therefore they can be altered easily. Additionally that gives us the benefit of being able to add the:

  • Password grant.
  • Client credentials grant.
  • Refresh grant.

to core, while maintaining:

  • Authorization code grant.
  • Implicit grant.

in contrib land. According to http://oauth2.thephpleague.com/authorization-server/which-grant I think that's the separation that makes more sense.

I'm not sure how we want to move forward with the inclusion of this into core. Thoughts?

e0ipso’s picture

I forgot to add a link to the module: https://www.drupal.org/project/simple_oauth

It's also worth mentioning that I recorded a video tutorial series on how to use this new version: https://www.youtube.com/playlist?list=PLZOQ_ZMpYrZtqy5-o7KoDhM3n6M0duBjX

Wim Leers’s picture

Awesome news, e0ipso :)

Wim Leers’s picture

Issue summary: View changes

OAuth2 is now officially proposed as an experimental module for Drupal 8.3: #2834718: New experimental module: OAuth2.

Wim Leers’s picture

Issue summary: View changes

JSON API is now officially proposed as an experimental module for Drupal 8.3: #2836165: New experimental module: JSON API.

Wim Leers’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: rest.module » Idea
Issue tags: +Needs framework manager review, +Needs product manager review
Wim Leers’s picture

Status: Active » Needs review
Wim Leers’s picture

Status: Needs review » Reviewed & tested by the community

Gábor tells me an initiative *team* marks their issue RTBC once they believe its complete for review. Given the feedback from neclimdul, dawehner and e0ipso here, I think it's fair to do this.

Also, this initiative has been moving forward for the past six months already anyway. #2721489: REST: top priorities for Drupal 8.2.x + #2794263: REST: top priorities for Drupal 8.3.x + #2836165: New experimental module: JSON API + #2834718: New experimental module: OAuth2.

e0ipso’s picture

Issue summary: View changes
Wim Leers’s picture

Issue summary: View changes

I closed #2794263: REST: top priorities for Drupal 8.3.x and opened #2852860: REST: top priorities for Drupal 8.4.x, which is the new issue to look at for progress in rest.module as part of this initiative in the next six months.

I'm proud of the progress we've made. We've tackled lots of tricky problems. rest.module is in a much better place now! We fixed the majority of things listed here.

Note that of the 53 issues are tagged with 8.3.0 release notes, 5 are for REST module!, that's almost 10%!


Also updating the timing of JSON API, GraphQL and OAuth 2

yoroy’s picture

Component: Idea » Active Initiative
Status: Reviewed & tested by the community » Active

Marking this "fixed" would eventually close out this issue. We don't want that so setting back to active and reclassifying as an active initiative.

Maybe update the roadmap to focus on the work for 8.4?

Wim Leers’s picture

Does that mean this is being added to MAINTAINERS.txt as an active initiative?

Marking Needs issue summary update to update for 8.4 (although there are already several 8.4 mentions in there, it definitely needs an update).

yoroy’s picture

Looks like it should be added to maintainers.txt indeed.

Wim Leers’s picture

Completely overhauled the issue summary. Matches what @e0ipso and I will present at the API-first initiative core conversation at DrupalCon Baltimore: https://events.drupal.org/baltimore2017/sessions/api-first-initiative.

Wim Leers’s picture

Also postponed the two new experimental modules that were already being proposed as part of this initiative: #2836165-27: New experimental module: JSON API and #2834718-33: New experimental module: OAuth2.

Wim Leers’s picture

Wim Leers’s picture

Issue summary: View changes

Adding "jump to" navigation at the top. Specifically to make it easier to find out who to ping for what.

dawehner’s picture

I'm wondering whether we can open up a discussion about dogfooding. Can we leverage our RESTAPI inside core? This could reveal a lot of potential bugs / DX issues, but at the same time move us to a better architecture as well.

Simon Georges’s picture

Since there are talks about being "best in class" regarding APIs, I wanted to share what the Plone team is currently doing. There is a Rest API like in Drupal, but what's good about it is the depth of what you can get, especially regarding other components of the page like Breadcrumb or Menus (see Components doc), allowing to build a full page without much work directly in JS.

I see that @davidwbarratt is working on that (see REST Menu Tree or REST Block Layout modules), but I can help but wondering if core shouldn't lead the effort this way (or to what extent we should go).

Wim Leers’s picture

I'd never heard of https://www.drupal.org/project/rest_menu_tree, so updated #2732283: Menus: ability to retrieve all menu links for a given menu, regardless of whether a menu link is defined in content/config/code for that. That's the core issue we have for that.

Drupal core offers only two @RestResource plugins: EntityResource and DbLogResource. The latter is mostly meant as an example. The former is the one that is very useful.

Everything you mention is related to getting data for particular types of front-end components. You can support that in Drupal 8's REST API by providing/writing additional @RestResource plugins. But providing those in Drupal core would be a mistake IMHO, because we A) need to know whether there is sufficient demand for this to be provided by Drupal core, B) need to agree on how these should work exactly (need different sites have different data needs for these components). The logical conclusion is therefore to do this work in contrib first, to let it mature there, and then given sufficient maturity & demand (which its usage will prove or disprove), we can move it into core then.

Simon Georges’s picture

Ok, thanks for the feedback!

dawehner’s picture

Issue tags: +API-First Initiative