Problem/Motivation

When listing content for a given entity type, some times we want to have all the content (including the bundle-specific fields). Examples of these are advanced content lists and data migration resources.

For some more context see: https://www.drupal.org/node/2751527#comment-11310159

Requirements

  • We would like to have addresses for entities at an entity level.
    • GET|POST /api/{entity_type}
    • GET|PATCH|DELETE /api/{entity_type}/{entity_id}
  • We would like to have addresses for entities at the bundle level.
    • GET|POST /api/{entity_type}/{bundle_type}
    • GET|PATCH|DELETE /api/{entity_type}/{bundle_type}/{entity_id}
  • We would like to be able to access bundle-level attributes at both the entity level and bundle level
  • We would like to be able to provide the attributes in common with all entities of a given type, regardless of bundle
  • We would like the developer experience to be intuitive and helpful
  • We would like the mechanics to be reasonable and efficient
  • We should attempt to avoid exposing implementation details

Proposed Resolutions

1. We expose a relationship to the bundle-level resource from the entity-level resource so we can do an include and get all the data we need in a single request. Additionally we want to add a custom link to the entity level document pointing to the bundle-level resource. We can label this link as entity-bundle.
2. Expose all attributes at either the entity or bundle resource; provide metadata about which attributes belong to which level.

Conclusion

This issue has been closed for the foreseeable future. The JSON API team has concluded that support for entity level resources is not tenable while adhering to the JSON API specification and RESTful principles and providing useful features and a straightforward implementation. The end result would probably lead to some confusion and frustration for users of the module, while not providing enough benefits to outweigh that pain.

The entity type to bundle relationship in Drupal can be a bit bizarre. Technically, no Drupal resource will every be purely an instance of an entity type. instead, every resource must be an instance of a bundle type. I.E., one cannot simply create just a 'node', it must either be an 'article' or a 'page.' This fact gave us reason to believe that purely providing bundle level URLs makes the most sense.

in order to provide entity level resources, we would need to support a separate entity level schema in addition to bundle level schemas. Awkwardly, a single resource would then be accessible by two different types and the same ID.

Since all resources under a URL like /node/{id} need to share the same schema, we would need to strip away any bundle level fields, rendering those resources much less useful. We would also be forced to prevent common REST operations like POST and PATCH to entity-level resources, or only allow these operations for just the set of fields that belong to the entity level schema. This would most likely lead to frustration and confusion when trying to execute these operations at the entity level. While we believe those errors could be handled gracefully and an attempt would be made to provide useful errors, we believe it is simply better to only provided the URLs for bundle level types.

This decision means a few use cases would become impossible with only the JSON API, namely administrative listings, migration endpoints, and 'most recent' type feeds. All of these use cases, however, are not necessarily client-driven. That is, those listings can be aggregated on the Drupal side and exported as JSON if necessary. For this server driven JSON output, we believe Drupal Core sufficiently fill those gaps with tools like Views REST exports.

While this issue is closed for the foreseeable future, this is not necessarily the final decision. If an elegant solution and/or a significant number of valid use cases demand the feature, the JSON API team is certainly willing the reopen the issue.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

e0ipso created an issue. See original summary.

e0ipso’s picture

This one proved to be very tricky. Let me know if you like the direction I ended up doing.

I need to put up some tests to check:

  1. That the relationship is added to the host
  2. That the relationship contains the correct links
  3. That the relationship can be included

Note that since this bundle-resource is not a field trying to traverse it with the query builder to add filters will throw an error. We probably want to catch that case and throw an appropriate exception.

e0ipso’s picture

Status: Active » Needs review

Triggering tests.

e0ipso’s picture

The current patch adds testing and support for configuration entities as well.

e0ipso’s picture

FileSize
9.95 KB

Added interdiff.

dawehner’s picture

Just some starting feedback. I would need more focus to review it more in depth.

  1. +++ /dev/null
    @@ -1,57 +0,0 @@
    -
    -  /**
    -   * Validates the JSONAPI parameter names.
    -   *
    -   * @param \Symfony\Component\HttpFoundation\Request $request
    -   *   The request.
    -   *
    -   * @return \Drupal\Core\Access\AccessResult
    -   *   The access result.
    -   */
    -  public function access(Request $request) {
    

    You like killing stuff, aren't you :)

  2. +++ b/src/Normalizer/ContentEntityNormalizer.php
    @@ -147,23 +154,18 @@ class ContentEntityNormalizer extends NormalizerBase implements DenormalizerInte
    -    return $fields;
    +    return $bundle_id ?
    +      $fields :
    +      $this->getResourceBundleRelationshipFields($entity, $fields);
    

    Nice idea to extract the method and not do another if, like many other people would do

  3. +++ b/src/Relationship.php
    @@ -0,0 +1,135 @@
    + * Class Relationship.
    + *
    + * Use this class to create a relationship in your normalizer without having an entity reference field.
    + *
    + * @package Drupal\jsonapi
    + */
    +class Relationship implements RelationshipInterface {
    

    Nice, a proper value object

  4. +++ b/src/Relationship.php
    @@ -0,0 +1,135 @@
    +  public function __construct(ResourceConfigInterface $target_resource_config, $field_name, $cardinality = FieldStorageDefinitionInterface::CARDINALITY_UNLIMITED, array $values = []) {
    ...
    +
    +  /**
    +   * {@inheritdoc}
    +   */
    +  public function setValues(array $values = []) {
    +    $this->values = $values;
    +  }
    +
    

    Are there usecases for this object to be muteable or would it be usually enough to just have the constructor and that's it?

  5. +++ b/tests/src/Kernel/Normalizer/DocumentRootNormalizerTest.php
    @@ -277,6 +292,61 @@ class DocumentRootNormalizerTest extends KernelTestBase {
    +    $route = $this->prophesize(Route::class);
    +    $route->getPath()->willReturn('/node_type/{node_type}');
    

    One thing I try to avoid is to mock value objects. They don't have any kind of behaviour, but are rather just a data store, in which case mocking that is more effort than it has to be.

e0ipso’s picture

You like killing stuff, aren't you :)

F*** me! Sorry I did this again. I'm not used to care about this stuff, I'm more of a PR dev. I'll fix it ASAP.

gabesullice’s picture

I like where this is going. I think it's certainly a requirement (or a "very nice to have") to provide generic entity level resources in addition to a bundle specific resource.

Regarding this patch specifically, I'm a bit uncomfortable with creating a resource-bundle "field" internally. This doesn't feel like the right abstraction. Already, the field idea is leading to some errors for me. When I follow the generated link to /api/node/1/resource-bundle?_format=api_json, I get a PHP Fatal error coming out of core about not being able to find the 'resource-bundle' field. I know we could code around this to prevent the error, but that feels a bit brittle.

Generally, it seems more appropriate to frame this in terms of whether to include additional attributes than it does to frame it in terms of a different resource altogether, as is implied by a relationship.

Perhaps a more semantically appropriate approach would be to use a custom query parameter and/or custom JSONAPI member field. We might provide a custom parameter like _bundleInclude=true and a custom top-level member like _bundleAttributes for all the fields that are bundle specific.

This will also sidestep the possibly-awkward HTTP semantics surrounding something like PATCH /node/1/resource-bundle and GET /node/1/resource-bundle not returning a collection.

Edit: fixed custom attribute name per the JSON API specification

Edit 2: If we do decide to stick with 'resource-bundle' relationships, I'd like to tie it in with this: https://www.drupal.org/node/2753803#comment-11330169

e0ipso’s picture

@gabesullice I understand that your concerns are around GET /node/1/resource-bundle and PATCH /node/1/resource-bundle. For those I would say that we can detect those calls and return an error along the lines of Use /node/article/1 instead. It seems that you believe that that would be a bit brittle, can you elaborate on that?

In general I would like to step away of custom extensions, unless we have a very good reason. In this particular case, I like the relationship approach because there is certainly a relationship between /node/1 and /node/article/1. They are not the same resource but they are undoubtedly related. In my opinion this relationship should be expressed, so consumers don't need to know about our implementation details to construct related URLs.

One change that I think we need is having the related link in the resource-bundle relationship point to /node/article/1 instead of to /node/1/resource-bundle. From an HATEOAS point of view, if the URL is not in the links, then there is no explicit support for it (we should still have those errors in place).

I would like to keep the possibility to do the include in the regular way, since this I see this as a regular 1st class relationship. One relationship that is not backed by a MySQL field table.

e0ipso’s picture

One change that I think we need is having the related link in the resource-bundle relationship point to /node/article/1 instead of to /node/1/resource-bundle.

I want to refactor the relationships so all the relationships implement the RelationshipInterface. Including the EntityReference relationships. When that happens we'll have a consistent way to generate the links that we need.

e0ipso’s picture

I merged 8.x-1.x into my feature branch to not delete the stuff. I also addressed the setValues and the test mocks (great suggestion!) that Daniel pointed out.

I created 2 follow ups to this issue (to limit scope): #2755025: [CLEANUP] Refactor relationships to have a unified base class and #2755033: [BUGFIX] Fix resource-bundle links to point to the correct location.

gabesullice’s picture

@gabesullice I understand that your concerns are around GET /node/1/resource-bundle and PATCH /node/1/resource-bundle. For those I would say that we can detect those calls and return an error along the lines of Use /node/article/1 instead.

Fixing the links/endpoints makes certainly makes me more comfortable with this proposal, although I'm still not convinced that the semantics of a relationship are ideal. Perhaps we should look at this JSON API issue for some guidance/ideas: Multiple/Hierarchical Types. That issue seems to be addressing our needs pretty closely.

+++ b/src/Normalizer/ContentEntityNormalizer.php
@@ -193,4 +195,36 @@ class ContentEntityNormalizer extends NormalizerBase implements DenormalizerInte
+    $field_name = 'bundle-resource';
+    $fields[$field_name] = new Relationship($target_resource_config, $field_name, 1, [
+      'target_id' => $entity->id(),
+    ]);
+    $fields[$field_name]->setHostEntity($entity);

This is what I was thinking of when I said we might be making something a bit brittle. We're creating a somewhat "magical" field here since 'bundle-resource' is not a true field. I'm worried we're going to end up carving out lots of little exceptions because of this. I think you may have already noticed this as well:

Note that since this bundle-resource is not a field trying to traverse it with the query builder to add filters will throw an error. We probably want to catch that case and throw an appropriate exception.

---

I want to refactor the relationships so all the relationships implement the RelationshipInterface. Including the EntityReference relationships. When that happens we'll have a consistent way to generate the links that we need.

Yes please! :) See #2753803: [FEATURE] Relationships should be dynamically registerable.

gabesullice’s picture

After digesting the JSON API issue a bit more. I think the "meta" member may be perfect for this. We can expose meta information about which attributes are entity fields and what are bundle fields without exposing the entity-bundle implementation in Drupal. What do you think of this:

GET /api/node
{
  "data": [
    {
      "type": "article"
      "id": 1,
      "attributes": {
        "created": 1234678,
        "updated": 1234678,
        "body": "This is some body text."
      }
      "relationships": {
        "uid": {
          "data": { "type": "user", "id": 3 }
        },
        "author": {
          "data": { "type": "user", "id": 4 }
        }
      }
      "meta": {
        "labels": ["node", "article"]
        "labelKeys": {
          "node": {
            "attributes": ["created", "updated"],
            "relationships": ["uid"],
          },
          "article": {
            "attributes": ["created", "updated", "body"],
            "relationships": ["uid", "author"],
          }
        }
      }
    }, {
      "type": "universityApplication"
      "id": 2,
      "attributes": {
        "created": 1234678,
        "updated": 1234678,
        "essay": "This is an essay."
      }
      "relationships": {
        "uid": {
          "data": { "type": "user", "id": 5 }
        },
        "applicant": {
          "data": { "type": "user", "id": 6 }
        }
        "cv": {
          "data": { "type": "file", "id": 7 }
        }
      }
      "meta": {
        "labels": ["node", "universityApplication"]
        "labelKeys": {
          "node": {
            "attributes": ["created", "updated"],
            "relationships": ["uid"],
          },
          "article": {
            "attributes": ["created", "updated", "essay"],
            "relationships": ["uid", "applicant", "cv"],
          }
        }
      }
    }
  ]
}

By exposing our type information in metadata, we can sidestep any custom extensions, hide the entity-bundle implementation details, and avoid any confusion about a bundle-resource being a field or separate object in the system.

e0ipso’s picture

Perhaps we should look at this JSON API issue for some guidance/ideas: Multiple/Hierarchical Types. That issue seems to be addressing our needs pretty closely.

That issue seems to be to list different types in a collection. This issue is not only concerned with collections.

We're creating a somewhat "magical" field here since 'bundle-resource' is not a true field. I'm worried we're going to end up carving out lots of little exceptions because of this. I think you may have already noticed this as well

I would define them as computed public fields, rather than magical. We will have to address the existence of computed pubilc fields soon enough. This is a common need for APIs, and we don't want to add the computed field to the entity itself b/c that may impact other subsystems.

By exposing our type information in metadata, we can sidestep any custom extensions

Shoving a bunch of metadata in the "metadata" property that is needed to drive business logic is not ideal. Implementors will need to know that the JSON API server is a Drupal installation using this module to use this data. That is the definition of a custom extension, without the extension discovery / negotiation. If we decided to go this route, I would much rather have this metadata be added only if a custom extension is requested (but I'm still not crazy about the idea).

At the moment our schema is determined by the entity model. It all boils down to:

  1. We want to accept relationships and attributes in the payload that are not Drupal fields.
  2. Our API data model is tightly coupled to the Drupal model. Anything else is custom extensions.

If we go with #2 I believe that sites will start adding custom stuff without going through the hassle to define custom exceptions, which may end up being a mess.

Thoughts?

gabesullice’s picture

Issue summary: View changes

I really hate to keep being a stick in the mud, but the entity-bundle relationship is so prevalent in Drupal, our decision here could end up being a big source of developer pain. Better to get it right this time than to have to worry about BC later.

I've updated the issue summary a small bit. I think it would be best for us to discuss in a Google Hangout, since I'm afraid we're both being misunderstood in writing.

e0ipso’s picture

After syncing in a hangout this is the agreements and disagreements we are facing.

Background

In order to have admin interfaces and migration sources we want to be able to have listings of entities independently of their bundle. That resolved in #2751527: [FEATURE] Create entity level resources.

Agreements

We agree that /node/1 and /node/2 should have the same schema. That schema should be in general /node/X and it should not change unless more entity fields are added. It should not depend on the bundle of the entity. That implies that we cannot have bundle fields in the entity type level resources. We also agree that a collection request to the node resource (located/indentified by the path /node and /node/{entity}) should only return resource entities of type "type": "node".

Disagreements

Linking from /node/1 to /node/article/1 (and /node/2 to /node/page/2) is where we start to disagree.

I am proposing having a computed relatinoship from /node/1 to /node/article/1 that allows implementors to include the bundle-level resource. Computed relationships are fully working relationships except that:

  • We are not building support for filters and sorts in the included attributes. You cannot do ?filter[resource-bundle.category]=foo. Anything that includes the dot notation is unsupported for computed properties because EntityQuery doesn't know about those.
  • There is no backing field or typed data behind them. We could rethink this in the future and require that computed API relationships (and attributes) have to be computed fields in the entity. If you feel that way, please create a follow up and let's have that discussion there.

@gabesullice doesn't feel comfortable with that idea since he feels that it will be very confusing (even if we put together enough documentation for it). He does not have a proposal at the current moment, but his hesitation is strong enough to ask to block this issue.

Unblocking proposal

Let's gather more opinions on this and see if someone has an alternative/creative solution for this problem.

dawehner’s picture

We are not building support for filters and sorts in the included attributes. You cannot do ?filter[resource-bundle.category]=foo. Anything that includes the dot notation is unsupported for computed properties because EntityQuery doesn't know about those.

That is a bit sad, because on the pure SQL level, this is kinda possible. I guess we can iterate later here.

In general for me it seems to be okay that those computed relationships have a different behaviour by default, as long we make that clear as part of the documentation. For example can we include that something into the jsonschema one day? Maybe as a special label.

e0ipso’s picture

That is a bit sad, because on the pure SQL level, this is kinda possible. I guess we can iterate later here.

That is true for relationships, but not necessarily for computed attributes. If you have a computed attributes, for instance a random number, and you want to filter all the resources by that number. It may be hard to find an abstraction that allows us to cover all cases.

Coming back to relationships, if we add to the mix entities stored across different backends the task get a bit harder.

Nonetheless, it is, as you point out, worth investigating it if we find use cases where that feature is needed/attractive.

For example can we include that something into the jsonschema one day?

+1

IMO that would be the perfect place for info like that.

gabesullice’s picture

I've been mulling over the conversation that @e0ipso and I had a couple days ago. I had not considered some of the points that @e0ipso brought up and I'm glad we had an opportunity to discuss those concerns.

I would actually flip some of the agreement/disagreements around.

Disagreement

It should not depend on the bundle of the entity. That implies that we cannot have bundle fields in the entity type level resources.

I am of the opinion that at the /node/1 URL, we must be able to satisfy all the attribute properties of being a node, but we should not be precluded from provided attributes specific to the actual entity it represents, meaning I'm fine with including bundle level fields. This would be akin to NodeInterface.

I recognize, however, that the JSON Specification is remains silent on the issue of hierarchical and composed types (if it is not, I would much appreciate a link to the relevant documentation. This should say either say that we CAN or CANNOT provide a superset of attributes as long as we satisfy the node subset).

@e0ipso is concerned that my proposal in #13 approaches the level of a custom extension. I do not believe it does, since it does not break implementations or require additional business logic.

Explicit support for hierarchical or composed types seems to be a missing piece of the JSON API specification. Perhaps this is an opportunity for us to contribute back. A proposal for this issue was supposedly scheduled for December of 2015, but I have not seen any follow up to it.

While the JSON API specification is agnostic to the issue, the JSON Schema specification does have a mechanism to support hierarchical/composed types: via the "anyOf", "allOf" and "oneOf" schema validations. See http://json-schema.org/latest/json-schema-validation.html#anchor75

This is very similar in spirit to my proposal.

Agreement
It is useful to provide entity and bundle level URLs, especially for administrative/migration purposes.

Linking from /node/1 to /node/article/1 (and /node/2 to /node/page/2) is where we start to disagree.

Perhaps we agree more than we think. I have no issue with linking to a bundle level resource. This could validly be done under the "links" member.

"links": {
  "bundleResource": {
    "href": "http://example.com/api/node/article/1",
  }
}

Summary
The heart of my disagreement is simply with treating a bundle as a separate resource, be it through a relationship or a computed field. My fear is that this will be non-intuitive to Drupal developers and deceptive to those unfamiliar with Drupal's node-bundle concept.

The root problem we're addressing is the problem of hierarchical types. We can solve this cleanly in the abstract. I feel we can do so in a way that does not force the consumer to realize the node-bundle concept at all.

By addressing the bundle concept without treating it as an inherited type, we are creating many unsolved issues for ourselves. This thread already has some examples. See #2 (QueryBuilder will throw an error), #9 (need to throw errors for PATCH|DELETE /node/1/resource-bundle), and #16 (sorting/filtering). Some of the same problems with sorting/filtering also mean that sparse field sets become an issue as well.

e0ipso’s picture

/me is in rage right now. After almost an hour of writing a comment it was lost in the wires.

I think this has become a time hog for me. Between the time invested in the patch #11 and the multiple comments in a foreign language. Let's hop into a hangout, record it, and reach a resolution.

One of the points I had there is that one of the main maintainers of JSON API thinks that the hierarchical types is not a good idea: https://github.com/json-api/json-api/issues/862#issuecomment-138712580

These API design decisions, much like table design in a relational database, seem very application-specific. I worry about leading API designers into using a hierarchical approach when a normalized, compositional approach is often preferable.

I had a LOT of stuff there that I don't have the energy to re-type.

e0ipso’s picture

e0ipso’s picture

Status: Needs review » Closed (won't fix)

@gabesullice will update this issue with the summarized reasons exposed in the hangout above. The decision was to drop support for the entity level resources.

gabesullice’s picture

Issue summary: View changes

Issue summary updated

gabesullice’s picture

Issue summary: View changes
gabesullice’s picture

Issue summary: View changes