Support mixed-bundle collections (e.g. `/jsonapi/node`)

+1! Great idea :)

+++ b/src/Controller/EntryPoint.php
@@ -101,4 +105,28 @@ class EntryPoint extends ControllerBase {
+    throw new HttpException(300, 'The JSON:API module for Drupal does not support heterogeneous collections (it only exposes "per-bundle" collections in Drupal terminology). The reason is that each resource type (each "bundle") can have a different set of fields, which makes it impossible to evaluate some sorts and filters.', NULL, ['Link' => $link_headers]);

Maybe we could also mention that due to having different fields it would be impossible to generate a schema for them.

First, let me say that I really like this idea. If we're not going to support the bundle-agnostic route, having a route there is a win :)

Ubernit: "JSON API" is always the correct human-readable form (not JSONAPI, JSON:API, JsonApi, JSON-API, etc). See the spec introduction.

I would actually change the language to something similar to jsonapi.api.php to be more accurate:

The JSON API module only supports single type collections ("per-bundle" collections in Drupal terminology). Since the JSON API specification does not have a concept of resource type inheritance or composition, the JSON API module implements different bundles of the same entity type as *distinct* resource types.

That's because the comment and the remark in #6 are not technically accurate. We certainly could sort by a missing field and filter by it as well. For sorts, you just follow the rules "NULL is < any value" and "filter by X implies X must exist". As for schema, it's entirely feasible to have a schema for a heterogenous collection (more below, but only if you want to get into the weeds).

The specification makes no prescription about single type routes. In fact, it makes no prescriptions about URLs at all.

We know that a route can support heterogenous types because of any of the related routes permit this. This is even implied by the spec because every resource object must carry a type and id member (instead of just id), even on the individual route.

To make a long story short, the spec is "loudly" silent about URLs as they relate to resource types, I believe specifically to not preclude this use-case (it is simply left as a recommendation that URLs are formed from types).

Finally, JSON Schema is entirely sufficient to support it too (search for "oneOf" here: http://json-schema.org/example2.html).

I like the idea to have special handling. However I'd like to respond with a 404 with the aforementioned links. I think that it is an error to construct a URL manually (use links!), and there is no content under that route hence a 404 seems more adequate. On top of that throwing a 4xx error conveys manual URL construction is wrong, dont's do it..

I would even be on board of adding manual URL construction is wrong, dont's do it. to the error message in the details section. Hehe.

As a side note I'd like to stop merging code until we have a candidate for the core patch. That's blocked by the test coverage and the 2.x branch. Therefore setting this to go against 2.x.

I would even be on board of adding manual URL construction is wrong, dont's do it. to the error message in the details section. Hehe.

👏👏 yes.

As part of the javascript modernisation initiative we realised that /admin/content in its current form kinda needs a way for you to fetch nodes across bundles.

@dawehner is there a strong UX reason to mix nodes of different bundles together but not mix other entity types? It has always been very confusing for Drupal newbies in my experience.

Could this be our chance to break away from that (anti?)pattern?

@Wim Leers I am working adding this ability to the support module for React decoupled admin ui https://github.com/jsdrupal/drupal-admin-ui/pull/96

This is support building the /admin/content page in the React app which of course needs to get a collection of node across bundles and needs filtering, sorting and pagination.

Basically it creates collection-only GET resource at /jsonapi/[entity-type]
The items under data in the collection would still point to /jsonapi/[entity-type]/[bundle]/[uuid]
So you could not POST to /jsonapi/[entity-type] and there is no route at /jsonapi/[entity-type], so no PATCH, DELETE, or GET there.

Right now it has to replace the jsonapi.resource_type.repository service so that it provide additional resources that use a resource type that extends \Drupal\jsonapi\ResourceType\ResourceType this is only to provide a different path.

It also has to replace the jsonapi.request_handler service to override resourceFactory() to return a different resource type CrossBundlesResourceType. Right now CrossBundlesResourceType only overrides getCollectionQuery() to add the bundle condition.

If a ResourceType could specify a controller class then would not need to override the jsonapi.request_handler service.

The current logic make incompatible with jsonapi_extras because it also needs to replace the jsonapi.resource_type.repository service. And even if that wasn't the case the fact that it uses different resource type means the new resources would not be configurable if the both were enabled.

@gabesullice had mentioned making a service_collector that would collect repository services. That would at least mean I would not need to replace the main repository service.

@gabesullice had mentioned making a service_collector that would collect repository services. That would at least mean I would not need to replace the main repository service.

Very small correction: I didn't mention alternative repositories. I think the resource repository should remain untouched. I proposed an alternative collection provider based on the concept introduced here: #2956353: [Experiment] Move fancy filtering to submodule.

This would mean that there would be no conflict w/ JSON API Extras.

Re #15

I do think there is UX to mix entities across bundles of the same entity type more that entities from different entity types

I am saying it would never be useful to mix entities from different entity types but for entities of the same entity types:

1. share some of the same fields by nature of base fields
2. Because of the way drupal is architected we have a bunch of hook_ENTITY_TYPE_[something] hooks which tends to encourage entities of the same type behaving the more similarly than other entity types. This has effect on user perceive them.
3. The different entity types are often used separate out things the end user can easily perceive as different, such as commerce products from site users.
4. Bundles in a entity type are often used for things the end user can easily see as similar, i.e, different product bundles all have prices and can be bought.

I do think there are different entity types which the users might have a difficult time distinguishing such as:

1. nodes which could have image field and media which also could have an image
2. Users and nodes that user to denote "people" such as authors who aren't actually users on the site

I think I disagree WRT the UX implications but I just don't have the credentials to say so. 😂

I asked @ckrina to chime in from the UX perspective. She is under incredible demand 😛, but let's see if she can find the time at some point.

Sorry @e0ipso I totally forgot to comment here :P
I guess the key question is who is this page for and what knowledge, experience or UI options are available. My first answer for a site builder experience would be to have a list with all of them, but in a content author perspective separating them makes sense if we're talking about really different entities like user profiles vs commerce products.
Do you think it might be worth talking about this in the next UX meeting (Thursdays 19:30h UTC)?

I am strongly opposed to this in the base JSON API module. I'm OK with it in a sub-module.

I am not opposed to it. What do others think?

I would love to see this happen.

I am strongly opposed to this in the base JSON API module. I'm OK with it in a sub-module.

Okay, so I think that this means we have maintainer consensus that drupal/jsonapi should ship with code that supports mixed-bundle collections.

We're not sure how, or in what form yet though... but that's okay :)

Let's open the feature request issue for it and have that discussion!

Actually, nothing got committed to this. Let's just change the title+scope.

I am strongly opposed to this in the base JSON API module. I'm OK with it in a sub-module.

I think I know why you feel this way, but I'm certain most don't. @e0ipso, can you elaborate and lay out your thoughts, fears, concerns, hopes, dreams (maybe deepest, darkest secrets) about this. It seems your position is softening a bit, but I know you've given it a lot of thought and we shouldn't miss out on all that brain-time.

I'll try to find more time to paste those (I think there's even a video of it).

When I said sub module I meant a different module. Sorry for misspeaking. I was thinking about the jsonapi_extras namespace or something of its own.

he'd prefer to see this added by a contrib module.

I'm completely fine with this if the API is not so constricted. I have yet to reivew #2960766-5: Support ResourceType which may have a null bundle defined.

most projects can live without this

Any site using Search API or Drupal Commerce will need this.

Sorry for not seeing this issue before creating the other one.
I think all sites at some point might need this request, not only those who are using Search API or Drupal Commerce.
In my case, I want to list all the nodes and after that to sort them or to add some filters. On the way it is now, I cannot do it.
For the user entity, there is the functionality that list all the users. Why cannot be made like this for the other entities?

I think all sites at some point might need this request, not only those who are using Search API or Drupal Commerce.

None of the projects I have been involved with needed this feature. Everyone I talked about this, outside of Drupal, think it's an anti-pattern. However, I think we can implement some sort of heterogeneous collections with reduced JSON API features in a separate module.

On my listings, I use only the base fields for filters.
Aware that user entity is a little different from the other entities, but using only the base fields for nodes, it will be the same thing. I could get only the nodes(from all content types) that are published, for example.

But what about an integration with views?

Let's move this to JSON:API Extras per #27, #30 and #31.

Agree?

So what you're saying there is that it's already the case that the jsonapi module supports mixed-bundle collections. For example, on the Article bundle, if you create a mutlivalued ER field (e.g., "Related Content") that can target both Article and Page nodes, and then create an Article that actually does so, then /jsonapi/node/article/UUID-OF-THAT-ARTICLE/field_related_content returns a collection that contains the full resource objects of each node (regardless of bundle) that is referenced by that ER field.

I'm unclear, however, on why that requires us to add a /jsonapi/node collection to the core module rather than to JSON:API Extras. Unless there's an overwhelmingly compelling reason to do that, I'd rather for example, keep that out of the scope of what's being proposed for Drupal 8.7, and perhaps revisit adding it into Drupal 8.8 if it proves to be robust in JSON:API Extras first.

WFM

#42 what exactly do you mean there :) ? https://www.urbandictionary.com/define.php?term=wfm
I suspect Works for me.

Is the patch in #4 valid to any extent?
Is it compliant with the latest state of the module?

I think a pros+const can help alleviate the decision. From what I see

Have a mixed nodes collection:
Pros:
- More developer and end user flexibility.
- I think I will need this one at some near future... :D

Cons:
- Bigger change to have performance and technical issues (complexity).
- Goes against REST principles - 1 endpoint 1 resource type.
- In context with #2145751: Introduce ENTITY_TYPE_list:BUNDLE cache tag and add it to single bundle listing will still use the node_list tag, but display much more content.
- In context with #3039730: Include paths are resolved for every resource in a resource collection, instead of once per unique resource type will need to compute and validate includes against all types listed in the collection.

That's all I can come up with at the moment...

Pros: Drupal Commerce can properly use JSON API without custom modules for catalogs, carts,and product pages over GraphQL which supports this.

How much of those pros would be solvable with parallel requests to the resources for the different bundles?

It is solvable, but inconvenient. In case of cold cache this are X bootstraps and custom merging of the results.

In case of partial browser cache hits you may have a performance boost instead.

If multiple bootstraps are the big concern your front end client should transparently interface with Subrequests.

The use case that isn't covered by using parallel requests is when you have a list containing content from multiple bundles with sorting and pagination. This is something we need in the JavaScript & Admin UI modernization initiative for building the /admin/content route.

That was exactly what I was thinking @lauriii

The pros and cons of this issue should be restricted to that case. Everything else is doable (and potentially more performant with parallel requests).

If the JS & UI Modernization initiative is the main / only use case of this I'd like to retitle this issue for that specific use case. However I suspect that @mglaman will have a need for this too.

In the mean time you can achieve this with two serial levels of requests. Is that not good enough for your needs?

+1 to #52. Would be nice to hear from @mglaman as well if they have some uncovered problems that aren't solved by parallel requests for different bundles.

Parallel requests are not a solution for Commerce.

A product catalog has different bundles, just like /admin/content does.
An order is made of order items that have different bundles.

parallel requests for different bundles.

So, if I wanted to build a product catalog which had products of a mixed bundle, limited to 15 products, sorted by title... how would this even work? I need to make two requests and hope all goes well? It's unfortunate since it's viable out of the box with GraphQL.

I will agree that it makes sense for the entry point /jsonapi would still link to specific resource type links, while the generic endpoints are available and "undocumented."

Yes, I think we are on the same page. However remember that this URL hierarchy with node/article is not semantic, just pure un enforced convention.

Shoot, too bad we couldn't pick something friendlier, but we would need to avoid naming clashes. Can bundles start with an underscore? I think something like /jsonapi/entity/_all could be more palatable and also hints that it is more of a "not to be used" endpoint. Prefixing things with an underscore is across the board a pretty understood pseudo way of saying "private"

If I needed support for that today this is the way I'd implement it:

1. Create a custom controller (or better a JSON-RPC method) that executes the Entity Query based on input query string parameters. This would return an ordered list of resource_type:uuid
2. Use subrequests to make one call to each resource passing all the uuids for that at once.
3. Piece back together the responses in the order returned by 1.

I've been trying to build a personal blog for myself and to use JSON:API for it. I have a few different content types: posts, external posts, conference talks, and albums. I'd like to list them chronologically on my homepage and to be able to filter them by tags like "Drupal" or "Aviation".

Getting those things in a "feed" of sorts is not a very confusing idea, really, even though the listing will have resources of different types.

Let's imagine that the URL for /jsonapi/node was /jsonapi/content, conceptually, I think that URL would make perfect sense, even if fields varied drastically between resources. IOW, the main "Drupalism" in /jsonapi/node is the word node, not the idea of a collection with multiple resource types. Perhaps the confusing bit is that the types are hierarchical? That's why @Wim Leers suggested /jsonapi/node/all I think, to discourage that line of thought. We can totally ignore the fact that that bundles are an "extension" of the node entity type and still have a coherent API. Take this other hypothetical URL: /jsonapi/posts?filter[category]=photo_album. That'd actually be a very WordPress-y URL. And conceptually, /jsonapi/posts/photo_albums as an alias for that URL would also be handy. That's how I'd conceive of our URLs if we were to support collections entities of a single type.

All of the previous paragraph is essentially arguing against the idea that we'd confuse non-Drupal developers by having a /jsonapi/node route and what I'm getting at is that I think it's a little patronizing for us to imagine that this would somehow be too difficult or confusing for them. IOW, on its own, this argument doesn't stand up for me. Perhaps I'm just debating a straw man though. Maybe that's not a primary motivation for not having these routes (See below about "first principles" so that we can all get on the same page).

Let's do away with the suggestion that this would somehow be contrary to RESTful principles. (AFAIK, @e0ipso has never said that either). A REST resource certainly doesn't have to be of a single type.

Let's also dispell the notion that JSON:API can't support mixed-bundle collections because of deeply technical reasons. Every related endpoint of cardinality n>1 is already potentially a mixed bundle collection.

#63 sounds extraordinarily complex and I don't understand the great complexity we're avoiding in JSON:API to create such a high bar for the use cases so far mentioned.

@e0ipso, I know that this is probably very frustrating to argue this once again. But I've never really understood your reasoning completely. It would be very useful for me to understand your position from first principles.

#64 +1.

I'm very confused by the how the conversation is going in this issue. AFAIK I have already given my +1 more than once to implement this. However it seems like other people think I'm opposed to the idea 🤔

#63 is meant to be read as: I don't have availability to work on this now, if this is super pressing here's a possible implementation with the tools you have today.

I'm very confused by the how the conversation is going in this issue. AFAIK I have already given my +1 more than once to implement this. However it seems like other people think I'm opposed to the idea 🤔

Hah! This would indeed be very confusing! I think Wim and I must have misunderstood/missed/forgotten this +1. We have both been under the impression that we needed to convince you and that you were vehemently opposed to this being in JSON:API.

It seems I missed that this issue was moved to JSON:API Extras...

I once again am confused about why this can't be in JSON:API itself.

Ah, OK. I see where you are coming now.

You are arguing in #64 about the feasibility of the feature, not where we should implement it. I do feel strongly against implementing this in JSON:API directly. Maybe we can leave this issue for the discussion on how to implement this, and then have a discussion where to put this?

Maybe we can leave this issue for the discussion on how to implement this, and then have a discussion where to put this?

WFM!

A quick note about #4.

+++ b/src/Controller/EntryPoint.php
@@ -101,4 +105,28 @@ class EntryPoint extends ControllerBase {
+      if (strpos($resource_type_name, $entity_type_id . '--') !== 0) {

We can't rely on this because resource type names can be aliased. However it should be possible to get the entity type ID from the resource type.

Is the discussion about where to implement this happening in another thread? We're deploying a lot of decoupled Drupal these days and are eyeing the JSON API as a great alternative to the somewhat hacky views/rest exports and custom modules we've done in the past. However, being able to do entity wide filters is keeping us from fully adopting it.

So from my company's perspective its high enough value we'd be willing to help put some engineering resources behind making this happen, but I guess the next question is – where should we start contributing? Based on the fact that @e0ipso is highly opinionated on this not being in JSON:API directly, should patches for JSON:API Extras be created?

I just gave this a go against Drupal 8.7.4 based on the Admin UI implementation. I got close until it came on filtering field configuration fields.

This is due to \Drupal\jsonapi\Context\FieldResolver::getFieldItemDefinitions receiving a null bundle.

Yes. The fact that it only comes out so rarely feels like a validation of my feeling: it is important to a niche of projects and worth implementing, but not in jsonapi core due.

Let's have an implementation patch first. We can discuss again where to put it once we have a clear vision of what the code looks like.

We discussed this in Decoupled Days 2019. We came to the following conclussions:

• We will put this under a different contrib as per my request ☺️ jsonapi_cross_bundles
• There will only be support for GET
• The edge cases in filtering will be solved using whatever stance Entity Query API uses.
• In parallel we'll create a JSON:API issue to add a payload to the 404 routes like /jsonapi/node that will contain: (a) a list of links to the resource types for that entity type, and (b) link to a documentation page explaining the cross bundle situation (maybe even mentioning the experimental jsonapi_cross_bundles module).

@mglaman is already working on a jsonapi_cross_bundles module that will allow experimentation on this area.

I belive this can be closed and continue the discussions about particular approaches there.

FTR: https://www.drupal.org/project/jsonapi_cross_bundles

;-)