Several times now, people who were tangentially involved with the development of Drupal 8's rest module (and related modules: serialization + hal + basic_auth) have stated that the main purpose of the REST module was "content sync".
The keyword "sync" is mentioned precisely zero times in the rest, serialization and hal modules' code. It's not mentioned in the handbook pages either: https://www.drupal.org/docs/8/core/modules/rest/overview. In all communication about it Drupal 8's REST support, it mentions "REST", "API", not "content sync". I was not involved in building the REST module, but as far as I know, the driving force was to make Drupal 8 support "decoupled Drupal" use cases.
It is very important to clearly document the scope/intent/target use case/target audience of the REST module. Otherwise we keep having this discussion. Reaching consensus about this and documenting it is the goal of this issue.
The two potential use cases are:
hal_json format) + https://www.drupal.org/project/entity_pilot (uses the hal_json format) + https://www.drupal.org/project/entity_share (uses the https://www.drupal.org/project/jsonapi module).There is one absolutely essential difference, and potentially competing requirement between those two. For content sync, the complexity of the serialized data doesn't matter, because developers don't need to make sense of its structure. For Decoupled Drupal, the complexity greatly matters, because it could very well be that people who've never even heard of Drupal, let alone are familiar with its data modeling concepts, need to make sense of the data, both for reading and for manipulating!
(On that note: for Content Sync, validation and access control are less important than for Decoupled Drupal, because usually A) the data models are the same, B) a root-level user determines which content is synced.)
For historical reasons, the (integration) test coverage of the REST module has not been great. As of Q4 2017, that's solved.
Because of that originally weak test coverage, we've collectively not been paying enough attention to the Developer Experience/Usability of non-Drupal developers consuming our REST API (for entities). Getting the data in and out of Drupal, by just using Drupal core, already was the major achievement. And it is!
The simplest possible way to build a REST API is to expose our internal data structures 1:1. And that is exactly what we have historically been doing.
This is fine! This is a great first step! But it's not going to make non-Drupal developers want to use our REST APIs. And frankly, it's not even going to make Drupal developers want to use our REST APIs. Because how many Drupal developers understand all details of our Entity/Field API? The whole point is that it's abstracted away by the UI: forms ensure that we can only perform valid modifications (that leave the data in a consistent, sensible state).
The REST API's responses and serialized entity representations (e.g. the JSON for node 5) are to Decoupled Drupal developers like forms are for content authors. We think optimizing the usability of our forms and their error handling is important. Well, ensuring clear names, structure, semantics, validation constraints and error responses are the equivalent for Decoupled Drupal developers.
Essentially: REST module's responses in Drupal 8.0.0 are pretty much the equivalent of telling content authors to use phpMyAdmin rather than our nice forms.
The REST API allows to interact with all entities. It does this automatically, based on metadata in Typed Data.
Based on that same metadata, the https://www.drupal.org/project/jsonapi and https://www.drupal.org/project/graphql modules expose alternative APIs, with different conventions, capabilities and response structures. But they expose fundamentally the same data. So they're affected as well.
=
The simplest possible way to build a REST API is to expose our internal data structures 1:1. And that is exactly what we have historically been doing.
We have been changing this. In Drupal 8.5, we're exposing the File entity's HTTP(S) URL as a computed field (#2825487: Fix normalization of File entities: file entities should expose the file URL as a computed property on the 'uri' base field) and the processed text of rich text fields is exposed as a computed property on those fields (#2626924: Include processed text in normalizations: "text" field type's "processed" computed property should be non-internal and carry cacheability metadata). These are essential for Decoupled Drupal, but are utterly pointless and even a waste of storage and bandwidth for the Content Sync use case.
This is why the Default Content module has an issue to create its own normalization: #2933777: REST/Serializer improvements in core/contrib make it less suitable for default_content use case.
If we're serious about Decoupled Drupal, we need to recognize and document that the Serialization and REST modules are primarily for Decoupled Drupal.
For the Content Sync use case, modules can either use Entity::toArray(), or write their own normalizers, which omit all computed fields and properties, and don't perform any transformations to the stored data to use less ambiguous representations (for example #2824717: Add a format constraint to DateTimeItem to provide REST error message).
Discuss
TBD
None.
None.
Comments
Comment #2
wim leers@gabesullice pointed out in chat that the https://www.drupal.org/project/relaxed module was notably absent from this issue summary. He's right! I totally forgot about it 😊😅
It occupies this interesting hybrid space: it does reuse the REST module (it depends on it), but it also relies on the https://www.drupal.org/project/replication module, which provides its own set of normalizers, encoders and its own formats:
streamandbase64_stream. It also looks like it's actually overriding pretty much all of core's normalizers. (I tried installing it to verify, but trying to install the half dozen modules required results in fatal PHP errors wrt service dependencies.)It looks like the only reason they use
rest, is to expose their own@RestResourceplugins, they don't seem to be reusing the@RestResource=entityplugin that core ships with. This is also suggested by their optional default config.Would be great if @dixon_ could actually confirm my hypotheses though! EDIT: pinged him: https://twitter.com/wimleers/status/953658238309986307
Comment #3
skyredwangHere is one use case, which doesn't agree with statement above: Sharding the content/data from Drupal and sync with mobile apps; the client side would care a lot of the data structure from Drupal. Each mobile app would only sync the content belongs to a specific user signed on this device. The content can be article, user, comments, taxonomy, etc. and they are (nested) referenced each other.
Comment #4
dixon_Regarding the Relaxed and Replication module use cases; I’m not too concerned with what direction we want core REST module to take. We started by depending a lot on it, but through the iterations we have ended up overriding most of it anyway. Even to the extent that we are considering dropping the dependency entirely in v2 of the modules.
The same goes for Serialization module; we use the core pieces of those components. But we’ve found that we really need a separate instance of the serializer because we’re having to fight too much about the weight/priorities of other serializer. For our use case we have one specific serialization spec we are following that we don’t want anyone to override.
So I guess what I’m saying is... For our use case it doesn’t matter too much. So let’s make REST become something simple and attractive to use for developers of simpler applications.
REST module won’t be too useful for content synchronisation, because for that to work well you’ll need many more components.
Comment #5
andrewmacpherson commentedDecoupling and Content Sync are two principal use cases, but there are certainly other use cases, and you may use one without the other. For example, on a recent project I had to prepare a very large CSV export. The CSV Serialization contrib module was very straightforward to use, but the size of the data export meant it needed Batch API process (~10 minutes), which effectively ruled out writing a @RestResource plugin.
Comment #14
smustgrave commentedThank you for creating this issue to improve Drupal.
We are working to decide if this task is still relevant to a currently supported version of Drupal. There hasn't been any discussion here for over 8 years which suggests that this has either been implemented or is no longer relevant. Your thoughts on this will allow a decision to be made.
Since we need more information to move forward with this issue, the status is now Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.
Thanks!