Problem/Motivation

For core-readiness, there are IMHO 3 overarching questions:

  1. What is the API?
  2. Are we confident our tests will catch BC breaks?
  3. Are we confident it is secure?

In other words: API surface (1), maintainability (2), security (3).

Based on these questions, I've identified the following areas (and issues within those areas) as being essential for getting JSON API into Drupal core.

JSON API spec compliance

Ensure that the JSON API module's HTTP API effectively matches the JSON API spec. (Also means fixes to comply with the spec are not BC breaks.)

Issues:

  1. DONE: #2917260: Validate against specific JSON Schemas if Schemata is present (validate using schemata module + OpenAPI docs are in sync)
  2. DONE: #2932030: Indicate supported JSON API specification version Indicate supported JSON API specification version (1.0)

Test coverage

Ensure that:

  • BC breaks are detected before they are committed.
  • Security vulnerabilities are proven to be absent.
  • All relevant test coverage lives in the jsonapi module, not jsonapi_extras

Issues:

  1. DONE: #2930028: Comprehensive JSON API integration test coverage phase 1: for every entity type, individual resources only Comprehensive testing of individual resources, for the most essential entity types
  2. DONE: #2932031: Comprehensive JSON API integration test coverage phase 2: for every entity type, complex use-cases Expand this to comprehensive testing of ALL entity types
  3. DONE: #2945093: Comprehensive JSON API integration test coverage phase 3: test JSON API-specific use cases: related/relationship routes, includes and sparse field sets
  4. DONE: #2953318: Comprehensive JSON API integration test coverage phase 4: collections, filtering and sorting
  5. DONE: #2953321: Comprehensive JSON API integration test coverage phase 5: nested includes and sparse field sets
  6. DONE: #2972808: Comprehensive JSON API integration test coverage phase 6: POST/PATCH/DELETE of relationships Comprehensive testing of POST/PATCH/DELETE of relationships
  7. DONE: #2950486: Test coverage: custom modules' format-agnostic @FieldType and @DataType-level normalizers behavior in JSON API (test that custom modules' format-agnostic field type normalizers ARE NOT picked up)
  8. DONE: #2950486: Test coverage: custom modules' format-agnostic @FieldType and @DataType-level normalizers behavior in JSON API (test that custom modules' format-agnostic data type normalizers ARE picked up)
  9. DONE: #2950744: Move relevant test coverage from jsonapi_extras to jsonapi Move relevant test coverage from jsonapi_extras to jsonapi
  10. DONE: #2936673: Add explicit JSON API test coverage for security issues already uncovered in REST core (prove security is on-par with core REST module)

DX

  1. DONE: #2930231: JSON API 403 errors don't tell the user *why* access is not granted: requires deep Drupal understanding to figure out (show reason from AccessResult::getReason() in 403 responses -> #2930028 will follow-up and test this.

API-First ecosystem

Ensuring that

  • A) the same data is exposed by core REST + JSON API + GraphQL + RELAXed Web Services
  • B) the pieces of data that should be normalized in the same way across all of the above, are normalized in the same way

Issues:

  1. DONE: #2860350: Document why JSON API only supports @DataType-level normalizers
  2. DONE: #2921257: On Drupal 8.5, JSON API should respect Typed Data's new ::isInternal(), and JSON API Extras should use ::setInternal() (support non-internal computed fields)
  3. DONE: #2929932: Work around core's ill-designed @FieldType-level TimestampItemNormalizer normalization until #2926508 lands (work-around for timestamp field type normalizer: don't wait for it to be converted to a data type normalizer, but duplicate that instead for now)
  4. DONE: #2929935: Remove JSON API's ScalarNormalizer: it's no longer necessary thanks to #2751325's PrimitiveDataNormalizer Assigned to: e0ipso (remove scalar normalizer in favor of core's)
  5. DONE: #2926463: [>=8.5] Remove JSON API's "file URL" field work-around now that Drupal core 8.5 fixed it (have a plan for removing JSON API's "file download URL")
  6. DONE: #2887313: JSON API indicates it supports POST/PATCH/DELETE of config entity types, but that's impossible (stop pretending JSON API supports config entities)
  7. DONE: #2937279: [META] Introduce concept of internal resource types Provide mechanism to exclude "internal" entities until a more generic option is supported by core
  8. DONE: #2937797: Entity UUID Converter performance

Maintainability

  1. DONE: #2923779: Normalizers violate NormalizerInterface (JSON API normalizers violate Symfony's NormalizerInterface)
  2. DONE: #2935370: Mark the JSON API serializer, normalizer and encoder services as private (mark all normalizer services as internal and private, cfr #2920536: Force all serializer encoders + normalizers services to be private)
  3. DONE: #2935947: Comply with code standards (comply with the core coding standards)
  4. DONE: #2933343: Define, document and mark scope of PHP and HTTP API

Comments

Wim Leers created an issue. See original summary.

wim leers’s picture

Issue summary: View changes
gabesullice’s picture

Issue summary: View changes
e0ipso’s picture

TODO (e.g. content_moderation_rest_resource_alter() opts out entity types from APIs; until there's a generic API for this, mirror those)

The ResourceTypeRepository service will allow to do that. One can decorate the service and filter the resources out for that purpose. If that DX is too convoluted we could add a hook_alter (or similar), but I think it makes sense to keep that pattern we are already using in JSON API Extras.

e0ipso’s picture

TODO (re-analyze code base to make as much as possible @internal)

This is a check, I don't expect much work to come out of this. Good caution nevertheless!

e0ipso’s picture

Issue summary: View changes
e0ipso’s picture

TODO (also test collections, related, include, sparse fieldsets etc -> don't test every include, just one, etc)

We'll need to come up with a reasonably complex use case for the test. We need to make sure to add that to the issue summary when one is created.

gabesullice’s picture

One can decorate the service and filter the resources out for that purpose. If that DX is too convoluted we could add a hook_alter (or similar)

Given that this is a temporary solution, I think the alter option is the better DX. We don't want to make developers spend a ton of time on something which they can just delete later. Let's make sure the hook is immediately marked as deprecated.

wim leers’s picture

#4: hah, I made the same observation :) But it does mean this will need to become a non-internal API. So we should be very very careful with that, because we'll have to support it forever.
Which is why I'd +1 what @gabesullice proposed: Let's make sure the hook is immediately marked as deprecated.

#5: Indeed, should be very little work!

#7: Indeed, details TBD. That's why I kept it very high-level.

e0ipso’s picture

But it does mean this will need to become a non-internal API. So we should be very very careful with that, because we'll have to support it forever.

That's a good point. I think we have consensus on hook_alter + deprecation.

gabesullice’s picture

Issue summary: View changes
wim leers’s picture

wim leers’s picture

gabesullice’s picture

Issue summary: View changes
e0ipso’s picture

I don't think we need #2923779: Normalizers violate NormalizerInterface to be added here. Any strong feelings about it?

wim leers’s picture

#15: I do. Core committers will never allow a new module to be added if it violates existing interfaces. We either need to stop violating that interface or we need to thoroughly document/justify A) why we're doing it, B) why it's safe to do so.

wim leers’s picture

xjm’s picture

@gabesullice and @Wim Leers pointed me at this issue. Exciting! A few initial questions and comments:

  1. A) the same data is exposed by core REST + JSON API + GraphQL + RELAXed Web Services
    B) the pieces of data that should be normalized in the same way across all of the above, are normalized in the same way

    This sounds like the most important piece to me for core inclusion.

  2. BC breaks are detected before they are committed.

    Does this mean BC for the module API between the contrib module and core? Test coverage preventing BC breaks for the exposed API (i.e. the JSON API API itself, not the PHP API)? Something else?

  3. Security vulnerabilities are proven to be absent.

    This is impossible to prove. :) What I'd say instead is:

    • No known security issues with the JSON API module in the public or private trackers.
    • Verifying that JSON API is also hardened against the access bypass and unserialization injection issues that were fixed in core REST last year.
      (If issues are identified they should be reported privately as security issues, not discussed here, since the contrib module has stable releases.)

     

  4. It looks like there's some question as to whether #2923779: Normalizers violate NormalizerInterface is a required part of the roadmap or not. This would probably fall under the architectural review for the module and I expect we'd want to answer the question (whether it's "yeah we need to fix this" or "wontfix" or something in between) before the module were marked stable for core, but it doesn't need to block adding the module as an alpha module.

    We can ask the framework managers about the issue when the time comes, but I think that can come after some of the other priorities and I think it's okay if it's an open question when the module is initially added to core.

wim leers’s picture

  1. Yes, the exposed API: the HTTP API.
  2. Yes, there's no test coverage that JSON API is hardened to the same extent as core's REST API.
  3. it doesn't need to block adding the module as an alpha module.

    + (in chat):

    We don’t need to worry about core BC until it’s marked beta, unless this is another case like the Media one where we’re concerned with BC to contrib

    That’s exactly what the concern is, and why the bar needs to be higher for the JSON API to be added to core than, say, the Settings Tray module.

    All JSON API contrib module users should be able to update to JSON API in core.

    It makes no sense to add an alpha-experimental module to core in which we don’t care about BC, because then we’d be screwing over the many thousands of existing JSON API users. The contrib module already cares about BC, so adding it as alpha-experimental would effectively be a step backwards. Plus, even as alpha-experimental, the iteration pace in core is much, much slower than in core.

    In other words: the JSON API contrib module needs to be added to Drupal core with at least beta-level stability: https://www.drupal.org/core/experimental#beta.

e0ipso’s picture

From #19

the JSON API contrib module needs to be added to Drupal core with at least beta-level stability

I agree with this statement. +1.


From #18

Verifying that JSON API is also hardened against the access bypass and unserialization injection issues that were fixed in core REST last year.
(If issues are identified they should be reported privately as security issues, not discussed here, since the contrib module has stable releases.)

That's a great suggestion. I'll make sure that's added to the current priorities.

It looks like there's some question as to whether #2923779: Normalizers violate NormalizerInterface is a required part of the roadmap or not.

Yeah, I'm feeling that we'll have to do a videochat meeting for that one. I think that the suggestion is well intentioned but I feel there are several non-obvious problems that may make this impossible (or too big of a task to be worth it). And moving forward with it will definitely break BC. That's why I'm leaning towards wontfix or do some more preliminary investigation when all priorities are knocked off.

wim leers’s picture

wim leers’s picture

Issue summary: View changes

Oh and #2917260: Validate against specific JSON Schemas if Schemata is present landed 6 days ago!

This means all critical JSON API spec compliance issues are DONE! 🎉

wim leers’s picture

Issue summary: View changes

@gabesullice has opened and is working on #2935947: Comply with code standards.

e0ipso’s picture

e0ipso’s picture

@xjm we met and debated about #2923779: Normalizers violate NormalizerInterface and we decided on the course of action. This should address #18.4.

PS: Meeting notes available.

e0ipso’s picture

@xjm we also have #2936673: Add explicit JSON API test coverage for security issues already uncovered in REST core to make sure we explicitly cover the security vulnerabilities uncovered in REST core.

Is there anything lingering from your questions above that you'll like to see addressed?

gabesullice’s picture

Issue summary: View changes

We got a ton of commits in this past week! 🎉

wim leers’s picture

Issue summary: View changes

Updating IS for #26.

#27: 🎉

wim leers’s picture

wim leers’s picture

Issue summary: View changes

All child issues of #2937279: [META] Introduce concept of internal resource types have been completed, so marking that done too! Go @gabesullice especially, but also @e0ipso for arriving at this solution!

wim leers’s picture

wim leers’s picture

Issue summary: View changes

Created #2950486: Test coverage: custom modules' format-agnostic @FieldType and @DataType-level normalizers behavior in JSON API for

  1. TODO (test that custom modules' format-agnostic field type normalizers ARE NOT picked up)
  2. TODO (test that custom modules' format-agnostic data type normalizers ARE picked up)
wim leers’s picture

Issue summary: View changes
  1. TODO (move relevant test coverage from jsonapi_extras to jsonapi — see #2932625-7: Fix query count after query builder refactor)

Issue created: #2950744: Move relevant test coverage from jsonapi_extras to jsonapi.

e0ipso’s picture

e0ipso’s picture

Issue summary: View changes
e0ipso’s picture

Issue summary: View changes
wim leers’s picture

Issue summary: View changes

Updating IS for #38.

#40: I don't understand why that's a must-have for this to go into core. It doesn't require architectural changes. Can you explain that a bit? :)

wim leers’s picture

e0ipso’s picture

Can you explain that a bit? :)

There is a requirement for performance for inclusion of a contrib into core. This feels like a necessary fix in that regard. Regardless, this was merged and fixed already.

e0ipso’s picture

Now #2950744: Move relevant test coverage from jsonapi_extras to jsonapi is fixed. We can move all of our attention to the two remaining issues.

wim leers’s picture

Issue summary: View changes

That will leave POST/PATCH/DELETE of individual resources and relationships

POST/PATCH/DELETE of individual resources is already tested. It's indeed not yet tested for relationships. So, can you clarify?


#46: marked it as done in the IS.


#2937797: Entity UUID Converter performance also landed, marked that as done too!

wim leers’s picture

wim leers’s picture

Issue summary: View changes

That will leave POST/PATCH/DELETE of individual resources and relationships

This is already done for individual resources, not yet for relationships. Checked this with @gabesullice (per my question in #47). Clarifying IS.

wim leers’s picture

Issue summary: View changes

#49 still needed an issue. Now there is one.

wim leers’s picture

Issue summary: View changes

Yay, @gabesullice got #2953318: Comprehensive JSON API integration test coverage phase 4: collections, filtering and sorting to an RTBC state; I just committed it! Only 6 left!

wim leers’s picture

#2929932: Work around core's ill-designed @FieldType-level TimestampItemNormalizer normalization until #2926508 lands is RTBC as of yesterday! It'll get committed once we wrap up the work for 1.x — to minimize divergence on patches that need to go into both branches.

wim leers’s picture

wim leers’s picture

wim leers’s picture

Issue summary: View changes
wim leers’s picture

Issue summary: View changes

Removed #2955020: Spec Compliance: JSON API's profile/extention (Fancy Filters, Drupal sorting, Drupal pagination, relationship arity) needs to be explicitly communicated per #2955020-50: Spec Compliance: JSON API's profile/extention (Fancy Filters, Drupal sorting, Drupal pagination, relationship arity) needs to be explicitly communicated:

@gabesullice and I discussed this in detail last week, together with core committers @xjm, @webchick, @effulgentsia and @lauriii.

Until https://github.com/json-api/json-api/pull/1268 lands and JSON API spec version 1.1 is released, the JSON API module for Drupal would not possibly be able to implement this according to the spec, since the spec hasn't been finalized yet. This might be okay for a contrib module, but imagine what the consequences are for when JSON API is added to Drupal core: we'd end up with potentially our own fork of the spec. This is too big a risk.

It's better for now to let JSON API to work as it does: its primary addition is "fancy filters", and the spec already gives us the leeway to implement this as we see fit: http://jsonapi.org/format/#fetching-filtering.

Therefore, postponing this. Also removing this from #2931785: The path for JSON API to core.

wim leers’s picture

Version: 8.x-1.x-dev » 8.x-2.x-dev

With only one remaining open (and RTBC!) issue against the 1.x branch left (#2977653: Spec Compliance: Return 204 or 200, not 201 for relationship POST requests.), we've now reached the point where the remaining work needs to happen in the 2.x branch.

Those remaining issues are:

  1. #2929932: Work around core's ill-designed @FieldType-level TimestampItemNormalizer normalization until #2926508 lands
  2. #2926463: [>=8.5] Remove JSON API's "file URL" field work-around now that Drupal core 8.5 fixed it
  3. #2887313: JSON API indicates it supports POST/PATCH/DELETE of config entity types, but that's impossible

All 3 of those can only ever happen in the 2.x branch.

Therefore moving this issue to the 2.x branch too.

wim leers’s picture

wim leers’s picture

wim leers’s picture

Issue summary: View changes
Status: Active » Fixed
Evgeny_Yudkin’s picture

JSON API is hardened to the same extent as core's REST API.

But why JSONAPI duplicates rest functionality? Why JSONAPI can not work through REST plugins system?

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.