This is how I'm using JSON API and JSON API Extras

I believe this is a major contribution :)

Thanks @jibran!

To figure out how to filter out data based on fields, sorting and paging was a lengthy task. I had to look at the tests and put some breakpoints in to figure out the real format. Let's discuss how can we improve this.

I would love to help with this. Will you open an issue to do so?

We ended up using JSON API Extras to leverage the count variable in the response. The idea was to get the count on the first request and then send subsequent parallel requests to fetch all the remaining nodes.

Is this the only reason you used JSON API Extras, or were there other reasons?

As I said before, there were ~250 nodes on the page and on a cold cache, the page takes 30-40 sec to load all the nodes.... Loading ~250 nodes from the DB is a pain but I don't think JSON API can do anything about it.

😱

This is the second time in about a month that we've heard a similar report. The other one did not open a new issue and simply appended to an existing one. Would you consider making a new issue about this? I'd love to figure out where the bottlenecks are. It may become a [META] issue.

All in all client is happy, front-end team is happy and I'm happy too that we were able to deliver such a complex solution and there are a lot of improvements in the pipline for the app and for backend stuff which will be fun to jump in.

Thank you for creating and maintaining JSON API and JSON API Extras.

❤️ And thank you! For this fantastic experience report!

@jibran thanks for the report, this is great. I feel your pains as well. You articulate very well what the caching solutions look like and what are the pain points with them. Maybe an interesting way to shave some milliseconds could happen with #1869338: Support ETag + If-Match for PATCH requests to prevent concurrent updates by the way of 304 responses.

Other than that I think you are right, we know that our query builder cannot be flexible and performant at the same time, or at least it's very hard.

Thank you for creating and maintaining JSON API and JSON API Extras.

:_)

Thank you so much, @jibran, for filing this issue! 🙏 This is exactly the constructive yet criticizing feedback we need! 👍

The OP describes my situation almost perfectly, including the follow up comments. I've literally started working on a computed field. Following related tickets with interest.

My issues are with nested paragraphs and includes, but I would say that it's not unreasonable the amount of raw data being requested from the FED, but it has backend implications, and I find it hard to explain to FEDs how to use includes and sparse fieldsets - because with nested paragraphs it's actually a bit of a nightmare negotiating the URL arguments to make it work.

I don't have the same issue as OP technically. We get lists from a solr endpoint (a very good decision in hindsight). Our problem is more for single node loads that can result in 250+ entities being loaded, because paragraphs, media, etc. Awkward combination of factors.

Having done some analysis I feel like it's the sheer workload that is biting us - I counted 3000 urls on one response, and they all take time! I then noticed the references like "uid" are being returned in our response, and they all get a few links.

Overall the things we learned

• be careful about the includes being used, avoid "just in case I need it"
• A nosql api for search/lists is great (that's not just a jsonapi thing)
• Lots of nested things, paragraphs is the big one, consider a custom RestResource endpoint
• Don't output anything you don't need - jsonapi extras to tweak fields, or some code like below

This snippet refers to the last point above.

function MYMODULE_entity_field_access($operation, $field_definition, $account, $items = NULL) {

  if (SOME LOGIC)) {
      $name = $field_definition->getName();
      switch ($name) {
        case 'uid':
        case 'revision_uid':
        case 'moderation_state':
        case 'unpublish':
        case 'review_date':
        case 'vid':
        case 'type':
        case 'field_admintags':
        case 'revision_translation_affected':
        case 'field_something_else_irrelevant':
        case 'default_langcode':
        case 'revision_id':
          return AccessResult::forbidden();
      }
  }

  return AccessResult::neutral();
}

@sime thanks for the report!

There is a HUGE point with regards to performance that we are not communicating well enough: Make sure to remove the fields your front-ends will never care about (revision_log, sticky, …) from the API using JSON API Extras.

Another good performance recommendation: turn off assertions in production.

@sime, a small suggestion: your code in #9 can set those fields as internal rather than making them forbidden. That would be even better for your performance since even your access checks won't need to be called. Perhaps SOME_LOGIC makes that a no-go though.

@gabe that sounds awesome :D

@e0ipso i've not heard of that. not sure what acquia does in this respect, but i'll keep an eye out for it.

@gabesullice i'm not quite sure where `internal` sits - is it a drupal field mechanism or a jsonapi mechanism. Is it setting fields to "DISABLED" in jsonapi_extras?

@sime, it's a Drupal core thing (https://www.drupal.org/node/2916592).

You can implement alter hooks and call the setInternal() method to mark fields and entity types as internal.

Setting a field as "internal" means it shouldn't be exposed to outside systems. So core REST, JSON API and GraphQL won't include them in any normalizations.

It has the same effect as setting fields as "disabled" using JSON API Extras but should work across all the main API modules. It also works on the entity type level.

JSON API will not generate any routes for internal entity types or any internal entity reference fields either.

@sime @gabesullice If you want to save yourself the trouble of implementing weird Entity API metada alter hooks that point to customized classes, JSON API Extras will do that for you, with one click.

Rather than having each site do a potentially buggy implementation of this, it's recommended to use a solid contributed implementation that anyone can help improve.

#6++ ❤️
#9++ ❤️ One remark though: you shouldn't forbid access to those fields (that will have consequences outside of REST/JSON API too), you should mark those fields as "internal". See https://www.drupal.org/node/2916592. EDIT: d'oh, @gabesullice said the same in #12 :P
#16: we need to update JSON API Extras to set this "weird metadata" instead of doing its own "weird thing" though ;)

Rather than having each site do a potentially buggy implementation of this, it's recommended to use a solid contributed implementation that anyone can help improve.

I completely agree. I volunteer to write and maintain this module. I've been saying and thinking it for a while, I'll just get it done. @e0ipso, @sime, @gabesullice: module name suggestions? internal_fields, disabled_fields, omitted_fields, internals, internalizer, internist … any other suggestions? I dislike the ones that mention "fields", because entity types can be marked internal too (to omit those from JSON API too). I quite like internist 😃

'intern' ? 😉

I kind of like something with the word "exposure" in it.

OTOH, I don't know if it deserves its own module at all.

I agree that it should be easier. But I'm not convinced that "one moduel to rule them all" is really desirable.

Perhaps a common core API (the "internal" concept) is sufficient and every API provider (e.g. REST, JSON API and GraphQL) having it's own UI for tweaking is acceptable. I don't think many people are using all these modules in conjunction with one another.

IOW, maybe the naming difficulty is a sign of a bad abstraction.

I've been letting my mind flirt more and more with a submodule in a JSON API for customizations.

The "special relationship" between Extras and JSON API seems to create more headaches than it's worth. Having everything marked @internal is great for speed and maintenance. But it also creates an increased maintenance burden when we don't iterate them in lock-step and that's unfair to all involved... maintainers and users alike.

/me ends his late night ramblings.

But I'm not convinced that "one moduel to rule them all" is really desirable.

Why not?

Perhaps a common core API (the "internal" concept) is sufficient and every API provider (e.g. REST, JSON API and GraphQL) having it's own UI for tweaking is acceptable.

This is EXACTLY why it is desirable! Each "API provider" having its own UI is hugely problematic:

• duplication of implementation effort
• duplication of maintenance effort
• possibility of each doing things slightly differently or get tempted to layer on their own mechanisms, which then results in inconsistencies between the different APIs

I'm going with internist unless somebody convinces me otherwise in the next 24 hours.

I believe that JSON API Extras is the place where this functionality should live. It is where it always lived well before core had a flag. Also, remember that I have gone through the effort of teaching the community how to use this. All in all, I think it's quite weird to propose this duplication of effort.

I'm sad that this happens against what we discussed (this exact topic) on this very Monday.

Also, remember that I have gone through the effort of teaching the community how to use this. All in all, I think it's quite weird to propose this duplication of effort.

… Huh?

I'm not proposing a duplication of effort. I'm proposing to remove this from the list of things that JSON API Extras does.

I've been asking for about a year that JSON API Extras make the switch from its own custom configuration to "disable" or "omit" or "hide" fields from API normalizations to use core's internal flag. #2871591: Allow ComplexData in TypedData to specify computed properties that should be exposed in normalization and other contexts was committed on November 6, 2017 (CR: https://www.drupal.org/node/2916592). It made sense to not do it right away because it only shipped with Drupal core 8.5. But we're now on Drupal 8.6. JSON API 2.x requires Drupal >=8.5. The GraphQL module has just entered the RC stage. Finally all things are aligned where making the switch from something JSON API-specific to something ecosystem-wide makes sense. As Drupal 8 grows older, and a particular Drupal 8 site is more likely to power multiple clients, through perhaps different APIs, this consistency in data exposure becomes increasingly important.

Continuing to do this in JSON API Extras is bad for the API-First ecosystem in Drupal. Especially if JSON API Extras only affects JSON API. Then you'd have to repeat the same configuration in every module.

I'm sad that this happens against what we discussed (this exact topic) on this very Monday.

I've been waiting for this internist module to happen for over a year (#2871591: Allow ComplexData in TypedData to specify computed properties that should be exposed in normalization and other contexts was created in April 2017, and I'd been discussing it for months prior to that). This is completely unrelated to any recent discussions! Please don't make more of this than it is: it's mere enthusiasm on my side to finally build something I envisioned ~1.5 year ago…!

We did discuss a few days ago the need to make all JSON API-related modules compatible with JSON API 2.x. We didn't discuss this disabling/hiding/… of fields at all. I still intend to help do that. You haven't seen that happen yet simply because I've been having to deal with very complex Drupal security issues.

We discussed this in IRC. We have bigger priorities right now than this internist thing — we'll just stick with jsonapi_extras for now.

Especially because @e0ipso said that he's been working on making it use setInternal(). So it'll use the right API, which means that it'll affect core REST and GraphQL too. The integrated DX of JSON API Extras rather than the focused-but-very-very-limited nature of internist (only disabling fields!) means that JSON API Extras would remain simpler to use.

#6 opened issues for the things that this experience report surfaced.

Ever since #17, this was about this proposed internist module. Let's just make JSON API Extras use setInternal(), that's good enough for now. Perhaps in a distant future, renaming of fields etc could also happen in a standardized way.

Nothing left to be addressed here, so marking Fixed!