Jump to:


The sunset of the API-first initiative

TL;DR: we are putting an official end to the API-first initiative although we don’t consider our work done. The initiative leads have struggled to find availability or energy to keep moving it forward. The upcoming JS components initiative will likely light this flame back up, and we’ll be there to assist and guide in their API endeavors.

See #3170020: The sunset of the API-first initiative for more information.

Even though the initiative is not active anymore, this page is left intact for archival reasons.

Background

Making Drupal API-first means making the data stored and managed by Drupal available for other software. It allows for:

  1. new types of integration (those not defined in Drupal/PHP code)
  2. full decoupling
  3. progressive decoupling (e.g. the overall page is still served by Drupal, but certain parts of the page provide highly interactive, JS-driven/enhanced interactions, which need to get certain data, which they can via REST)

Therefore it's very important that we advance Drupal's web services.


Drupal 8.0 shipped with a REST API. Unfortunately, it's in a state where it's very difficult to set it up (configuration is hard) and use (DX is hard). There are many subtle bugs (test coverage is weak). However, in many cases, it is "just" a matter of polish.

At the time, Drupal 8 selected HAL+JSON as the default format. However, since then, the JSON API spec has been finalized and matured. Roughly speaking, it has two things going for it that HAL+JSON does not:

  1. it is highly opinionated/prescriptive, which means better interopability and less bikeshedding about details (many things not yet supported in D8's REST have standardized mechanisms/conventions in JSON API)
  2. an extension negotiation system

Drupal 8's REST API's "Entity" REST resource does not support revisions, and it uses IDs, not UUIDs. RELAXed Web Services adds support for both of those, which makes it possible to use REST for complex workflows. Or for replicating content in their entirety (thanks to CouchDB compatibility): for deployment purposes or for offline-first apps/sites (using PouchDB).

What's also problematic, is that Drupal 8 only supports two authentication mechanisms out of the box: Basic Auth and Cookie (which only works very well for browsers, not apps — so only really suitable for progressively decoupled use cases). The default expectation for front-end developers and native app developers is OAuth2 support.

A non-RESTful data exchange API has also since emerged: GraphQL. It's not very well suited for use cases like syncing data, but it's extremely well suited for the use case of fetching data to be used in UI components. In the long run, GraphQL might become part of how Drupal 9 renders itself, but long before then it'd allow front-end developers to already start using it.

Addressing all of the above means making Drupal 8 an unparalleled API-first platform:

All data available:

  • RESTfully
    • in any format (JSON, HAL+JSON …)
    • in multiple flavors ("HTTP purist", jsonapi.org …)
  • non-RESTfully
    • via GraphQL
    • via CouchDB protocol, for replication & syncing

… using any authentication mechanism (Cookie, Basic Auth, OAuth2)

Goals

Must have
Should have
Could have

Process and where to find it

Drupal API-First Weekly meetings happen every Monday at 4 p.m. CET. An announcement with the hangouts on-air link happens in #drupal-contribute and in multiple channels in Slack right before start. You can find the log of previous meetings in http://youtube.com/c/Mateu-e0ipso.

Meetings happen in a friendly environment and are open to everyone. Listen-only attendance is more than welcome.

REST: DONE!
Work is being done at https://drupal.org/project/issues/drupal?component=rest.module, plus the associated issue queues: https://drupal.org/project/issues/drupal?component=serialization.module and https://drupal.org/project/issues/drupal?component=hal.module
(Module already lives in core, arguably should have been experimental.)
JSON API
Work will be done at https://www.drupal.org/project/jsonapi to first get a stable contrib module during the 8.2+8.3 cycles.
During the 8.3 cycle, it will be moved to an experimental core module. Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core. Being added to core in #2843147: Add JSON:API to core as a stable module.
UUID & revision support DONE, see #53.
TBD
GraphQL Not doing in core, see #50.
Work will be done at https://www.drupal.org/project/graphql to first get a stable contrib module when it happens (since there are currently no active maintainers).
During the 8.3 cycle (at earliest), it will be moved to an experimental core module. TBD
OAuth 2.0 Bearer Token
Work will be done at https://www.drupal.org/project/simple_oauth to first get a stable contrib module during the 8.2+8.3 cycles.
During the 8.3 cycle, it will be moved to an experimental core module. Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core.
CouchDB support Not doing in core, see #49.
TBD

Proposed roadmap

Much work can happen in parallel!

Each of the following bullets lists parallel tracks, and within the bullet a sequence is defined:

  • R1 -> R2A + R2B
  • J1 -> J2
  • G1 -> G2
  • JG
  • O1 -> O2
Phase R1: REST usable (8.2) DONE!
Address all top priorities (#2721489: REST: top priorities for Drupal 8.2.x), and improve test coverage significantly (#2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method) so we can have very strong guarantees that it works as expected. (Test confidence level should be similar to that of the BigPipe module.)
Phase R2A: REST best-in-class (8.3+8.4+8.5+8.6) DONE!
See #2794263: REST: top priorities for Drupal 8.3.x + #2852860: REST: top priorities for Drupal 8.4.x + #2905563: REST: top priorities for Drupal 8.5.x + #2941316: REST: top priorities for Drupal 8.6.x.
Details TBD. Things being considered: collections, pagination, link relations, entity metadata (fields required vs optional, cardinality, validation constraints — see #2665602: Implement an API for REST clients to know which fields are editable), "playground" UI, auto-generated REST API docs, configuration UI.
Phase R2B: REST UUID + revision support (8.4) DONE! (In JSON:API, not REST.)
This will be addressed in part or in whole by the Workflow Initiative — see #2721129: Workflow Initiative. Details TBD.
Phase J1: JSON API stable in contrib (8.2, 8.3)
Address all concerns in #2829327: Review JSON API module for core-readiness. DONE!
1.0 released! And 2.0 even by now.
#2842148: The path for JSON API to "super stability"
Phase J2: move JSON API into core (when: ?)
Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core. Details TBD.
Phase G1: GraphQL stable in contrib (no timeframe, because lack of maintainers) Not doing in core, see #50
Details TBD.
Phase G2: move GraphQL into core (>=8.3) Not doing in core, see #50
Details TBD.
Phase JG: factor a shared component out J[1,2] & G[1,2] (8.2)
Already happening to some extent at https://github.com/drupal-http-apis/type-graph Has turned out to be not relevant, this was first proposed in the very very early days of both the GraphQL and JSON:API modules.
Phase O1: OAuth 2.0 Bearer Token stable in contrib (8.3)
Phase O2: move OAuth 2.0 Bearer Token stable into core (>=8.4)
#2834718: New experimental module: OAuth2
Delayed per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core.

Not in scope

TBD

Existing core features

The REST module.

Contributed projects

Team and resources

Initiative coordinator: Wim Leers

REST (usable + best-in-class)
tedbow, dawehner, Wim Leers
More people welcome — we provide mentoring :)
Subcoordinator: Wim Leers
JSON API
e0ipso, gabesullice, dawehner, Wim Leers
Subcoordinator: e0ipso
GraphQL
fubhy (contrib module maintainer), others TBD
OAuth 2.0 Bearer token
e0ipso (contrib module maintainer), others TBD
REST UUID + revision support
TBD, hopefully the Workflow Initiative does most of the heavy lifting, and we just need to help out with reviews.

Sign-offs

[This section still needs work]

  • REST: Crell, dawehner, klausi, Wim Leers
  • JSON API: e0ipso, Ed Faulkner, gabesullice, klausi, Wim Leers

Alex Pott has stepped up and is showing interest, so we probably want his sign-off as well.

Signoffs needed

TBD

Signoffs given

TBD

Comments

Wim Leers created an issue. See original summary.

neclimdul’s picture

Just for clarity, Oauth actually has disclosure concerns over HTTP as well(the module mentions this) so we can't really get away from that reality. It is definitely something we should be supporting though. Documentation in the OAuth RFC http://tools.ietf.org/html/rfc6750#section-5

That said 100x yes to every bit of this. :-D

dawehner’s picture

+1 for not mentioning the type graph idea. Its a really abstract concept with a lot of effort before it provides actual value to people, unlike all the other efforts which happen at the same time.

Wim Leers’s picture

Category: Bug report » Plan

Oops :)

webflo’s picture

Issue summary: View changes
Wim Leers’s picture

Title: Web services initiative » API-first initiative
timmillwood’s picture

I hope we (the workflow initiative) can help make things work well with revisions, as a contributor to the RELAXed Web Services contrib module I'm keen to make sure that everything still works there.

dawehner’s picture

Phase 8.3:

I have one general question . Do we want to expose multiple collection capabilities by default? One could say, when we expose a collection resource, we introduce some custom standard for filtering/sorting, so you could also just use another standard like JSONAPI in the first place. Providing collection of bundles is a non trivial problems, as for example, documenting the available properties is no longer clear, as they change on a per bundle level. Given that collection/pager feature could be something we expose just via jsonapi/graphql, but I agree this could be a more radical approach.

e0ipso’s picture

Some extra context about what @dawehner is asking can be found in https://youtu.be/mveQFnm8S1E?t=38m19s.

kalinchernev’s picture

Issue summary: View changes

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

e0ipso’s picture

JSON API update

The jsonapi module is progressing steadily. There are several contributors that are currently using the module reporting (and fixing!) bugs.

This is good news for the goal of «get a stable contrib module during the 8.2 cycle».

Wim Leers’s picture

@e0ipso: Nice! :)

rest.module update

I've also got news from the rest.module side. I closed #2721489: REST: top priorities for Drupal 8.2.x and opened #2794263: REST: top priorities for Drupal 8.3.x, which is the new issue to look at for progress in rest.module as part of this initiative in the next six months.

I'm proud of the progress we've made. We've tackled lots of tricky problems. rest.module is in a much better place now! We fixed the majority of things listed here.

Note that of the 34 issues are tagged with 8.2.0 release notes, 9 are for REST module!, that's more than 25%!

Wim Leers’s picture

Issue summary: View changes
fgm’s picture

Note that there is a week-long GraphQL sprint at Drupalcon Dublin, from today to next Sunday : you can register on the Drupalcon Dublin sprint attendance worksheet.

e0ipso’s picture

The Simple OAuth 8.x-2.x module has been upgraded to use the http://oauth2.thephpleague.com PHP library.

It has been architected in a way that the authorization grants are plugins, therefore they can be altered easily. Additionally that gives us the benefit of being able to add the:

  • Password grant.
  • Client credentials grant.
  • Refresh grant.

to core, while maintaining:

  • Authorization code grant.
  • Implicit grant.

in contrib land. According to http://oauth2.thephpleague.com/authorization-server/which-grant I think that's the separation that makes more sense.

I'm not sure how we want to move forward with the inclusion of this into core. Thoughts?

e0ipso’s picture

I forgot to add a link to the module: https://www.drupal.org/project/simple_oauth

It's also worth mentioning that I recorded a video tutorial series on how to use this new version: https://www.youtube.com/playlist?list=PLZOQ_ZMpYrZtqy5-o7KoDhM3n6M0duBjX

Wim Leers’s picture

Awesome news, e0ipso :)

Wim Leers’s picture

Issue summary: View changes

OAuth2 is now officially proposed as an experimental module for Drupal 8.3: #2834718: New experimental module: OAuth2.

Wim Leers’s picture

Issue summary: View changes

JSON API is now officially proposed as an experimental module for Drupal 8.3: #2836165: New experimental module: JSON API.

Wim Leers’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: rest.module » Idea
Issue tags: +Needs framework manager review, +Needs product manager review
Wim Leers’s picture

Status: Active » Needs review
Wim Leers’s picture

Status: Needs review » Reviewed & tested by the community

Gábor tells me an initiative *team* marks their issue RTBC once they believe its complete for review. Given the feedback from neclimdul, dawehner and e0ipso here, I think it's fair to do this.

Also, this initiative has been moving forward for the past six months already anyway. #2721489: REST: top priorities for Drupal 8.2.x + #2794263: REST: top priorities for Drupal 8.3.x + #2836165: New experimental module: JSON API + #2834718: New experimental module: OAuth2.

e0ipso’s picture

Issue summary: View changes
Wim Leers’s picture

Issue summary: View changes

I closed #2794263: REST: top priorities for Drupal 8.3.x and opened #2852860: REST: top priorities for Drupal 8.4.x, which is the new issue to look at for progress in rest.module as part of this initiative in the next six months.

I'm proud of the progress we've made. We've tackled lots of tricky problems. rest.module is in a much better place now! We fixed the majority of things listed here.

Note that of the 53 issues are tagged with 8.3.0 release notes, 5 are for REST module!, that's almost 10%!


Also updating the timing of JSON API, GraphQL and OAuth 2

yoroy’s picture

Component: Idea » Active Initiative
Status: Reviewed & tested by the community » Active

Marking this "fixed" would eventually close out this issue. We don't want that so setting back to active and reclassifying as an active initiative.

Maybe update the roadmap to focus on the work for 8.4?

Wim Leers’s picture

Does that mean this is being added to MAINTAINERS.txt as an active initiative?

Marking Needs issue summary update to update for 8.4 (although there are already several 8.4 mentions in there, it definitely needs an update).

yoroy’s picture

Looks like it should be added to maintainers.txt indeed.

Wim Leers’s picture

Completely overhauled the issue summary. Matches what @e0ipso and I will present at the API-first initiative core conversation at DrupalCon Baltimore: https://events.drupal.org/baltimore2017/sessions/api-first-initiative.

Wim Leers’s picture

Also postponed the two new experimental modules that were already being proposed as part of this initiative: #2836165-27: New experimental module: JSON API and #2834718-33: New experimental module: OAuth2.

Wim Leers’s picture

Wim Leers’s picture

Issue summary: View changes

Adding "jump to" navigation at the top. Specifically to make it easier to find out who to ping for what.

dawehner’s picture

I'm wondering whether we can open up a discussion about dogfooding. Can we leverage our RESTAPI inside core? This could reveal a lot of potential bugs / DX issues, but at the same time move us to a better architecture as well.

Simon Georges’s picture

Since there are talks about being "best in class" regarding APIs, I wanted to share what the Plone team is currently doing. There is a Rest API like in Drupal, but what's good about it is the depth of what you can get, especially regarding other components of the page like Breadcrumb or Menus (see Components doc), allowing to build a full page without much work directly in JS.

I see that @davidwbarratt is working on that (see REST Menu Tree or REST Block Layout modules), but I can help but wondering if core shouldn't lead the effort this way (or to what extent we should go).

Wim Leers’s picture

I'd never heard of https://www.drupal.org/project/rest_menu_tree, so updated #2732283: Menus: ability to retrieve all menu links for a given menu, regardless of whether a menu link is defined in content/config/code for that. That's the core issue we have for that.

Drupal core offers only two @RestResource plugins: EntityResource and DbLogResource. The latter is mostly meant as an example. The former is the one that is very useful.

Everything you mention is related to getting data for particular types of front-end components. You can support that in Drupal 8's REST API by providing/writing additional @RestResource plugins. But providing those in Drupal core would be a mistake IMHO, because we A) need to know whether there is sufficient demand for this to be provided by Drupal core, B) need to agree on how these should work exactly (need different sites have different data needs for these components). The logical conclusion is therefore to do this work in contrib first, to let it mature there, and then given sufficient maturity & demand (which its usage will prove or disprove), we can move it into core then.

Simon Georges’s picture

Ok, thanks for the feedback!

dawehner’s picture

Issue tags: +API-First Initiative
Wim Leers’s picture

I closed #2852860: REST: top priorities for Drupal 8.4.x and opened #2905563: REST: top priorities for Drupal 8.5.x, which is the new issue to look at for progress in rest.module as part of this initiative in the next six months.

We made solid progress. We've tackled lots of tricky problems. rest.module is in a much better place now! We fixed about half of the issues we wanted to tackle, including adding comprehensive REST test coverage for more than 30 additional entity types! (See #2824572: Write EntityResourceTestBase subclasses for every other entity type..)

Note that of the 69 issues that are tagged with 8.4.0 release notes, of which 10 are for the API-First Initiative — or 14% of all release notes issues!

For more details about the progress made, see #2852860-41: REST: top priorities for Drupal 8.4.x.

Wim Leers’s picture

Issue summary: View changes
webchick’s picture

The issue summary lists monthly meetings for this initaitive, but not when they happen. The hangout linked has "8.2" in the title, and the Google Doc of notes was last updated in 2016. Have these moved to a new schedule/format, or have they tapered off since then?

tormi’s picture

> have they tapered off since then?

Not to my knowledge, https://www.youtube.com/watch?v=wbFM1cn2trc

e0ipso’s picture

You are both correct.

@webchick as @tormi is mentioning we have kept the weekly meetings. There is a video log of (almost) all the meetings.

The monthly meetings are not happening anymore. They have definitely tapered off since then. I will update the IS.

e0ipso’s picture

Issue summary: View changes
e0ipso’s picture

I also notice that some of the goals listed here are already achieved 🎇❗️

I'll talk to Wim today in our sync about how to keep issues up to date with minimal effort. Thanks for the patience.

Wim Leers’s picture

I closed #2905563: REST: top priorities for Drupal 8.5.x and opened #2941316: REST: top priorities for Drupal 8.6.x (but forgot to report that back here) and opened #3002188: REST: top priorities for Drupal 8.7.x just now, which is the new issue to look at for progress in rest.module as part of this initiative in the next six months.

For 8.5, this was my high-level summary:

We made more progress than I expected based on progress in previous minors. We reached 100% integration test coverage at last! We achieved roughly half of the top priorities!

Note that of the 11 issues that are tagged with 8.5.0 release notes, of which 3 are for the API-First Initiative, and 22 issues are tagged with 8.5.0 highlights, of which 6 are for the API-First Initiative — or 27% of all release notes issues and 27% of highlights!

See #2905563-48: REST: top priorities for Drupal 8.5.x for more details.

For 8.6, which just shipped, this is my high-level summary:

We made less progress than I hoped in some areas. But some foundational pieces finally clicked into place. Most importantly, core's rest.module is now in a maintainable state. For the past 4 months, the number of open issues has been below 50, about half of which are feature requests, and a fifth are blocked on other subsystems. And as of Drupal 8.6, each core module that has an HTTP API surface is now responsible for its own test coverage. See https://wimleers.com/blog/api-first-drupal-two-big-maintainability-miles....

Note that of the 14 issues that are tagged with 8.6.0 release notes, of which 2 are for the API-First Initiative, and 22 issues are tagged with 8.6.0 highlights, of which 4 are for the API-First Initiative — or 14% of all release notes issues and 18% of highlights!

Now it's time to move on to 8.7.x. I've opened #3002188: REST: top priorities for Drupal 8.7.x for that, which is the continuation of this issue.

See #2941316-27: REST: top priorities for Drupal 8.6.x for details.

Wim Leers’s picture

For most of 2018, my primary focus has been on JSON API, to make that module ready to ship with Drupal core. See #2843147-62: Add JSON:API to core as a stable module for progress made in the first half of 2018, #2843147-71: Add JSON:API to core as a stable module why it didn't ship with Drupal 8.6, and https://www.drupal.org/project/jsonapi/releases/8.x-2.0-beta1 for the first JSON API 2.x release, which is the version of JSON API that will ship with Drupal core.

Wim Leers’s picture

Issue summary: View changes

As of https://wimleers.com/blog/api-first-drupal-8.6, we've considered phase R2A done. Updating IS.

Wim Leers’s picture

Issue summary: View changes

JSON:API 1.x reached 1.0 in May 2017, a long time ago. It was only rock solid/mature many months later though. Still: we can definitely consider that done.

Marking phase J1 as done.

Wim Leers’s picture

Issue summary: View changes

RELAXED WS/CouchDB in core: originally this was our intent, but it does not make sense. Current maintainer Tim Millwood confirmed this.

Wim Leers’s picture

Issue summary: View changes

As of https://dri.es/headless-cms-rest-vs-jsonapi-vs-graphql, GraphQL is no longer being proposed for core inclusion. To be fair: this was always just an eager ambition from the start, the maintainers of https://www.drupal.org/project/graphql were never actively advocating for it to get into Drupal core.

Wim Leers’s picture

Issue summary: View changes

Missed a few bits for #47 and #49.

Wim Leers’s picture

Issue summary: View changes

And a few missed bits for #48.

Wim Leers’s picture

Issue summary: View changes

Revision and UUID support has been blocked in part by #2721129: Workflow Initiative, but even more so by the REST module's architectural choices.

Those choices made it nearly impossible to add revision support to the existing @RestResource=entity plugin, especially without breaking BC. The JSON:API module did gain support for it, thanks to it being data-centric rather than a very generic framework like the REST module.

Similarly, adding UUID support to the existing REST API for entities would have been a nightmare. But we made the JSON:API module UUID-centric early in the development cycle.

Based on that, I'm marking this "done".

Wim Leers’s picture

Continuing where #46 left off: for the remainder of 2018, our attention went to stabilizing JSON:API and dealing with https://www.drupal.org/sa-contrib-2018-081, which was caused by a major feature gap in Entity/Field Query API that JSON:API then had to work around. #2843147-144: Add JSON:API to core as a stable module is now RTBC.

In Drupal 8.7, a fair amount of REST work still happened: #3002188: REST: top priorities for Drupal 8.7.x, with a focus on those things that would also help JSON:API, i.e. that would allow REST and JSON:API to use the same code.

That should mean that in the next two weeks, we'll be able to mark JSON:API as "done" too and we'll be left with only OAuth2 support here!

e0ipso’s picture

Issue summary: View changes
e0ipso’s picture

Issue summary: View changes
nod_’s picture

Status: Active » Fixed

Since the initiative is closed, this should be fixed no?

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.