Fetching entities of an entity type regardless of their bundle (f.e. all nodes)

I think its quite a big design limitation.

I wholeheartedly disagree. And so do some people having to deal with the uncertainty of What fields does a node contain?. I don't see many arguing that Views has a severe limitation because you cannot list users and nodes at the same time. I feel that this is the same.

If I gave you two lists of fields one for a content type and one for a custom entity type, would you be able to tell them apart? If not, why does it make sense to list one along with other content types but not the other?

For your particular use case, you can request as many listings as you need (even in one request thanks to https://www.drupal.org/project/subrequests) and build any admin interface you need. The only limitation here is pagination for joint sets of results, which does not seem a reasonable trade off for all the problems the other approach brings in.

I'm curious about some of the arguments.

When you say:

I cannot agree more. But my point is listing all entities of the same type but regardless of their bundle, not mixing entity types. This is exactly what views allows, btw.

I totally get what you want to list. What I don't get is why not listing across entity types is not a problem. This is exactly what I was trying to get with my comment before:

If I gave you two lists of fields: one for a content type and one for a custom entity type, would you be able to tell them apart? If not, why does it make sense to list one along with other content types but not the other?

I'll be curious to see how the view experience goes. It seems like an interesting approach.

The concept of bundles is a Drupal thing, in the wide REST world there is no such thing every resource type is based on a REST entity type. The concept of bundles is just a trick to do config/code reuse, it only makes sense within Drupal. When you take your resources off Drupal there is no such concept. Therefore from a consumer perspective page is a different resource/rest entity type from contest and from location. This is regardless of how they were originated in the back-end server (Drupal). The whole point about decoupling is that consumers know nothing about the server implementations.

I think the feature can make sense in a consumer that is implementing a feature that re-couples a de-coupled site. However, that is a fair feature just not worth a re-architecture of the module.

I leaving the status as active to encourage other people to continue the conversation.

Hi guys, thanks for raising this issue. In my experience I already had a case where we needed to provide a mixed output of "blog" and "post" nodes mixed within the same paginated output, and it was really tricky workaround to get it done (spoiler: had to use Message) module. So as a frontend developer, I was really disapointed to meet this limitation. I assume that we're not going to compete with Views, because it's designed for another purpose. However, I'm not very clear on this architectual desicion as well. Mateu, could you please shine the light on this issue?

I would also really like to see this feature. My use case is querying all nodes for the value of a shared field, which contains information to create a URL in a front-end consumer.

e.g. I query the API with the slug "morning-joe/watch/joe-rural-health-care-would-be-savaged-by-this-bill-976931907840", and I want it to return the node where "field_slug" matches this value

While having some discussion with @justafish I found https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/... which has support for oneOf, so at least from the schema point of view it could be possible.

Client side parsing would of course be needed here, but I think they could deal with that. At least elm can :P

Note: I am not 100% sure whether this belongs into jsonapi itself, maybe jsonapi extras or a submodule there, is the better place for such a feature.

An option for my use case is to use https://www.drupal.org/project/subrequests and query across all node types

Note: I am not 100% sure whether this belongs into jsonapi itself, maybe jsonapi extras or a submodule there, is the better place for such a feature.

Tbh I'm not sure if this is a feature or not. Still wonder what is the architectual decision behind this. If it's a feature, then JSON API Extras should be a good place to make this feature configurable.

This was discussed intensely back in the day when the decision was made. It was not decided lightly. There is no turn back at the moment without a huge effort. The gain does not justify the amount of time invested on this.

I'm unsure this can be done in JSON API Extras, I think not. However the only thing that I will ask to anyone working on this is that we don't limit it to bundles. If someone is to get lists with mixed stuff it should work seamlessly across bundles and different entity types.

This was discussed intensely back in the day when the decision was made. It was not decided lightly. There is no turn back at the moment without a huge effort. The gain does not justify the amount of time invested on this.

I truly believe that there is a good reason and there was a big discussion behind this. However, unfortunately this answer still doesn't shine the light on why this decision had been made. Is there a reference to the discussion or any other historical notes? Very curious!

However the only thing that I will ask to anyone working on this is that we don't limit it to bundles. If someone is to get lists with mixed stuff it should work seamlessly across bundles and different entity types.

I'd propose to move step by step here. It feels like fairly simple job to return entities of the same entity type as the list, filter them by fields and so on. Essentially, it all comes down to a single entity field query.

However, when we get to mix entities of ANY type - that brings a lot of questions without good answers, and it will take a lot of time to get those issues sorted: how to filter, how to sort, how to merge entities into a single list, how to work with pages, how to prevent unwanted entities to appear in the list, etc. As well as that, it goes way beyond simple entity field query. My personal recommendadion would be to ignore mixing of entity types until we have at least mixed entities of the same entity type.

As the next steps I suggest to draft a patch where we get rid of entity bundles limitation and see what underlying issues will be met. Feels like this conversation will be hard, so it has to include at least technical concepts.

Is there a reference to the discussion or any other historical notes? Very curious!

I'm sure it findable in the issue queue. There is also links to the video where the discussion happened.

I'd propose to move step by step here.

I'm good with that, but I'll want to hold off to merge the whole ladder to make sure that we don't miss any steps.

As the next steps I suggest to draft a patch where we get rid of entity bundles limitation and see what underlying issues will be met.

I suggest you to contact @tedbow to understand the hurdles to support the bundle-less endpoints.

Yesterday we had a debates in Slack regarding this issue. Pasting the formatted coversation here for reference.

spleshka [7:30 PM]
hi @tedbow, @e0ipso recommended me to get in touch with you regarding this https://www.drupal.org/node/2886540 (see last comment). Let me know when you have a bit of time to discuss the issue. Thanks

e0ipso [7:31 PM]
@spleshka the first awkward question is. If you can have entities of multiple bundles

spleshka [7:32 PM]
not sure I follow. You mean entities like nodes?

e0ipso [7:32 PM]
yeah
how do you deal with sort
based on field_tags
with all the bundles that don't have that field
?

spleshka [7:34 PM]
I'd expect the same behavior like if you join field table to the node and sort by value of this field. They'll be filtered out
jsonapi basically wraps the entity field query, and that's what it is in the nutshell
so my preference would always be to keep the sql behavior
or actually entities without field_tags will be listed below entities with field_tags. As I said, happy with whatever approach is offered by sql
just go and create a new view in Views for this case. It looks like it keeps the nodes without field_tags below other nodes
and that is fine
in that view it's more important that we can list all nodes regardless of their type

e0ipso [7:49 PM]
sql would ignore them I believe

spleshka [7:51 PM]
I also thought so, but then quickly set up a test view using Views - it puts them at the end of the list

dawehner [7:53 PM]
@e0ipso I would really just fallback to the entity query behaviour. I guess you want to be able to define the semantics?

e0ipso [8:02 PM]
> in that view it's more important that we can list all nodes regardless of their type
I do not agree. Bundles are a drupal only thing
but more importantly not being able to say _this resource has this output_
(because you don't know the list of fields of "a node")
it a complete confusion for anyone

spleshka [8:04 PM]
We can define in docs that entity endpoint is a mix of different resources
If they need a fixed list of fields we can push them towards entity bundle resource
I totally understand where you are coming from
Though I'm wondering what is worse between two fires: not having the necessary functionality in place or having mixed resources on the entity resource page

e0ipso [8:19 PM]
@spleshka that's the exact solution that was in the codebase at some point
and we removed it because it made sense to no one but people that already knew what was going on
@spleshka we also won't be able to get schemas for those resources
@spleshka besides multiple maintainers of the REST module in core
told me that they would remove the support for that if they could
marginal gain
for gigantic gain

spleshka [8:21 PM]
What if schema would include only shared fields across an entity

e0ipso [8:21 PM]
> What if schematic would include only shared fields across an entity?
Then it would be incorrect, right?

spleshka [8:21 PM]
Yeah, true
Sort of a workaround..

e0ipso [8:22 PM]
like entities disappearing because of a sort in a non-base field
like …

dawehner [8:26 PM]
@e0ipso at least for me this schema problem seems solvable with openapi 3.x

spleshka [8:49 PM]
trying to summarize:
1. schema issue. As @dawehner mentioned, it seems solvable with openapi 3.x. Very good news, thanks for this note.
2.
> like entities disappearing because of a sort in a non-base field
it's not an issue at all. Both options (filtering out or being at the end of the list) are expected when you sort by a field which doesn't not exist for certain entity types (in my experience I've seen both). That's how entity query works, not sure why it should be considered as an issue. I believe that FE devs are smart enough to understand why either of those options is used to resolve this issue.
3.
> multiple maintainers of the REST module in core told me that they would remove the support for that if they could
I'd also prefer to have this feature and complain that it's not perfect, rather than not having this feature :slightly_smiling_face:
4.
> like …
I know it might seem like a lot of theoretical issues, but let's try not to make them open-ended. Let's keep having a healthy debate and see where we end up eventually. You've introduced me to the term "cow path", and I love it. Isn't it the case?)

dawehner [8:56 PM]
It should be definelty an extra module

spleshka [8:58 PM]
why?

e0ipso [10:01 PM]
@dawehner @spleshka I don't think openapi can do anything about it.
there is no schema of a `node`. OpenAPI won't fix that.

> I believe that FE devs are smart enough to understand why either of those options is used to resolve this issue.

I do not agree. Not because of FE-devs lack of understanding. It's because bundles are not a thing outside of Drupal
and this module is to expose content outside of Drupal

> I'd also prefer to have this feature and complain that it's not perfect, rather than not having this feature :slightly_smiling_face:

It's not about the feature perfection. It's about the gigantic complexity and hurdles it brings with it that are not obvious until you have to fix them, and then maintain it.

And let's not forget that the benefit is: _a list of unrelated resource types listed under some endpoint that is not a resource with pagination_

> Let's keep having a healthy debate and see where we end up eventually.
It is highly improbable that this is going to make its way back into the `jsonapi` module. I implemented it, then happily removed the feature, later on (several times) I made the comment of how immensely complex this would be with that feature in.

It's not that I'm speaking from a theoretical point of view or from ignorance.

> It should be definelty an extra module
@spleshka that's probably why @dawehner was suggesting a separate module.

I'm just warning you because I know _there be dragons_, but you may be able to tame them…

@spleshka would you be so kind to paste this conversation in the issue queue?

spleshka [10:19 PM]
Thanks Mateu. I'll certainly update the issue as soon as I get home.

Reading #16, it looks like what remains is here is: document why this is not supported, to pre-emptively answer people's future questions, and to also document the current architecture.

Is that right?

I'm sure it findable in the issue queue. There is also links to the video where the discussion happened.

This is the referenced video: https://www.youtube.com/watch?v=mveQFnm8S1E

I haven't taken the time to go back and watch it to summarize. There might be other videos that I am not aware of.

Assuming that #17 is correct, here is a patch that documents the current architecture and its limitations.

#2829398-20: Clean up JsonApiResource: the annotation, plugin type, plugin manager, plugin implementations, the dynamic routes generator and the request handler that again confirms what has already been said several times above.

Comments/issues informing the docs:

1. #2829398-36: Clean up JsonApiResource: the annotation, plugin type, plugin manager, plugin implementations, the dynamic routes generator and the request handler
2. #2841056: Remove use of plugins

If it turns out that we do want to keep this feature request open, I'll move this patch to a new issue.

1. +++ b/jsonapi.api.php
   @@ -8,6 +8,21 @@
   + * The unit of data in the JSON API spec is a "resource". The Drupal module that
   + * implements JSON API exposes every entity as a resource.
   

   every bundle

2. +++ b/jsonapi.api.php
   @@ -24,5 +39,29 @@
   + * The JSON API module does not provide a PHP API. It is designed to be
   + * "zero-configuration".
   

   There is a service nowadays that allows generating a JSON API representation of an entity programmatically.

   I can't recall its name from memory.

3. +++ b/jsonapi.api.php
   @@ -24,5 +39,29 @@
   + * - Adding new resources/resource types is unsupported: all entities/entity
   + *   types are exposed automatically. If you want to expose more data via JSON
   + *   API, make sure they're entities. See the "Resources" section.
   + * - Customizing the normalization of fields is not supported: only normalizers
   + *   for "DataType" plugins are supported (a level below fields).
   

   This is a fantastic summary.

4. +++ b/jsonapi.api.php
   @@ -24,5 +39,29 @@
   + * @section bc Backwards Compatibility
   

   This is an important section. I like your stance, let's be conservative.

   Should we mention how normalizers are considered internal despite them being services (that's an artifact of the Serialization integration, not an explicit decision to provide a service).

#22

1. No, resources are entities, but yes, resource types are entity bundles. I expanded this paragraph. Better now?
   :)
2. As #23 said, that's the jsonapi.entity.to_jsonapi service. This is indeed a PHP API technically, but it's a read-only API. So added it, but in a separate paragraph.
3. 😀
4. Yes, I like setting clear expectations!

   Great point about normalizers! We have #2920536: Force all serializer encoders + normalizers services to be private to fix that generically. But added that here, to be super clear and explicit!

LGTM. I'm sure we'll want to iterate further on this, but let's get this in. We can continue this in a broader issue.

I'd love to see this replicated or linked in the d.o documentation. @Wim Leers do you think you can take that on?

Yay progress!

I'd love to see this replicated or linked in the d.o documentation. @Wim Leers do you think you can take that on?

• Replicated = nightmare. If we replicate now, we need to keep them in sync forever. That's not going to work out very well. Especially not at this stage, when things are moving very fast (I intend to expand these docs further in upcoming patches). How about we replicate every time we tag a new release?
• Linked = happy to do so, but that'll mean linking to this *.api.php file without it being rendered in a nice way (like entity.api.php for example) , because this is contrib, not core. I see this *.api.php file also as a stepping stone towards core, where modules are required to document their API this way.

So… would you prefer a link, or a once-per-release replication?

I'm bad at recurring tasks. I'd rather the less nice link that is a one-time task. People will have to live with it, I don't think it's too much to ask.

Done: it's now linked from https://www.drupal.org/docs/8/modules/json-api (change: https://www.drupal.org/node/2849185/revisions/view/10484056/10761932).