REST: top priorities for Drupal 8.4.x

Forgot to omit one that was fixed in 8.3 :)

#2649646: Normalize path fields as part of the entity: allow REST clients to know the path alias was closed as a duplicate of #2846554: Make the PathItem field type actually computed and auto-load stored aliases, updating IS.

#2826407: PATCH + POST allowed format validation happens in RequestHandler::handle(), rather than in the routing system was actually discovered by #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method, and there's a @todo for it in EntityResourceTestBase, but I forgot to add it to #2794263: REST: top priorities for Drupal 8.3.x. So it fell off the radar. That won't happen again now.

Adding #2737751: Refactor REST routing to address its brittleness.

Renamed the Test coverage: velocity+maintainability+reliability+predictability section to Velocity+maintainability+reliability+predictability — sometimes mere untangling of broken architecture and fixing it can also be important for those 4 things. #2737751 is an example of that. It's been haunting us for years. Time to fix it.

Added #2853460: Simplify RequestHandler: make it no longer ContainerAware, split up ::handle() and #2775479: Try to remove the "map HEAD to GET" logic in \Drupal\rest\RequestHandler::handle().

Added #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't and #2854560: \Drupal\Core\Routing\RequestFormatRouteFilter::filter() is too HTML-centric as JSON API blockers.

#2775479: Try to remove the "map HEAD to GET" logic in \Drupal\rest\RequestHandler::handle() landed!

#2854560: \Drupal\Core\Routing\RequestFormatRouteFilter::filter() is too HTML-centric landed!

Added #2854830: Move rest/serialization module's "link manager" services to HAL module, which also just landed!

#2825347: 'Accept' header still is used for determining response format of error responses, ?_format= has no effect AND serialization module's exception subscriber does not support status code 500 error responses landed, yay!

Added #2853300: Standardize fatal error/exception handling: backtrace for all formats, not just HTML.

#13: because JSON API != REST. They're orthogonal. They do both rely on the serialization module though. So many of the fixes that we are landing for the REST module also help the JSON API module.

You should consider this issue on the same level as #2836165: New experimental module: JSON API: both this and that issue are plan issues in the "Drupal core ideas" queue. And #2843147: Add JSON:API to core as a stable module is the child issue of #2836165 against Drupal core, so that that we can have a patch that is tested against core.

I totally forgot to add #2824717: Add a format constraint to DateTimeItem to provide REST error message to #2794263: REST: top priorities for Drupal 8.3.x! Adding it here now.

Per #2100637-156: REST views: add special handling for collections and later comments, we decided to not pursue this in core. We collectively agreed that it makes more sense to defer this to JSON API for now. Since #2099281: [PP-1] REST views: pagination link relations depends on #2100637, that issue is also impacted.

Removed those two.

#2648268 was deprioritized at #2648268-35: REST views: row-level caching doesn't exist, unlike for other types of views two weeks ago. Also removing it from this list.

#2751325: All serialized values are strings, should be integers/booleans when appropriate landed! Inexplicably, this was not listed here. This has taken months to get fixed. It definitely has earned a place here.

Added #2856110: [PP-1] Expose entity validation errors in a machine readable REST API: .

Added #2847708: RPC endpoint to reset user password.

#2827084: EntityNormalizer::denormalize should not throw UnexpectedValueException, but \Symfony\Component\HttpKernel\Exception\UnprocessableEntityHttpException, so we get a 422 response instead of 400 is listed here, but it already landed at #2794263-63: REST: top priorities for Drupal 8.3.x. Removing it.

#2844046: REST Resource config entities do not respect the status (enabled/disabled) landed.

#2825487: Fix normalization of File entities: file entities should expose the file URL as a computed property on the 'uri' base field is already listed as a serialization gap, but it's also a JSON API blocker. #2353611: Make it possible to link to an entity by UUID is another JSON API blocker.

See #2842148-11: The path for JSON API to "super stability".

#2817745: Add test coverage to prove that REST resource's "auth" configuration is also not allowing global authentication providers like "cookie" when not listed landed!

#2826407: PATCH + POST allowed format validation happens in RequestHandler::handle(), rather than in the routing system just landed!

Finally, another serialization gap closed! #2827164: Entity reference field normalization has target_type and target_uuid, but not used in denormalization is in :)

#2850034: CORS allow-origin '*' not possible because of cached headers landed!

And #2853300: Standardize fatal error/exception handling: backtrace for all formats, not just HTML also landed!

That's in JSON API, not in Drupal core's REST. We already have issues for the normalization of TimestampItem and DateItem in Drupal core.

#2824717: Add a format constraint to DateTimeItem to provide REST error message landed, which solves one half of the serialization gap problem.

Added #2827797: ResourceResponse(Subscriber) + Dynamic Page Cache: making authenticated ResourceResponses significantly faster + #2765959: Make 4xx REST responses cacheable by (Dynamic) Page Cache + comprehensive cacheability test coverage. They're super important for maintainability/reliability, and long-term velocity.

#2768651: Let TimestampItem (de)normalize to/from RFC3339 timestamps, not UNIX timestamps, for better DX landed! Finally :)

Added #2882717: EntityResourceTestBase: assert that PATCHed and POSTed entities contain the intended values to Velocity+maintainability+reliability+predictability.

#2846554: Make the PathItem field type actually computed and auto-load stored aliases landed! This is the third serialization gap we close.

#2847708: RPC endpoint to reset user password landed!

#2882717: EntityResourceTestBase: assert that PATCHed and POSTed entities contain the intended values landed!

#2824851: EntityResource::patch() makes an incorrect assumption about entity keys, hence results in incorrect behavior just landed and I notice now that it wasn't listed here. Added to the Velocity+maintainability+reliability+predictability
section.

Thanks everybody for helping out here!

We didn't make as much progress as I'd hoped in some areas (serialization gaps), but we made far more progress in other areas (test coverage). But obviously we fixed tons of issues, many of them requiring fixes in other subsystems. We achieved roughly half of the top priorities!

Note that of the 69 issues that are tagged with 8.4.0 release notes, of which 10 are for the API-First Initiative — or 14% of all release notes issues!

Now it's time to move on to 8.5.x. I've opened #2905563: REST: top priorities for Drupal 8.5.x for that, which is the continuation of this issue.

Thanks, and see you around!

(Sister posts at #2794263-64: REST: top priorities for Drupal 8.3.x and #2721489-48: REST: top priorities for Drupal 8.2.x.)

Let's do a restrospective. We landed:

Any use case

4 issues

DX

6 issues

Fully decoupled

2 issues (all of them!)

Velocity+maintainability+reliability+predictability

5 issues (over half!)

And note that #2824572: Write EntityResourceTestBase subclasses for every other entity type. is down to 11 remaining issues, it was at 41 remaining issues when 8.4.x started! That's a lot of missing test coverage that was added!

Many of them were very difficult to land, requiring lots of consensus building, or requiring blockers to be solved that aren't listed here, often in other subsystems. Many of the ones that did not land are already well on their way, making it very likely that they will land in 8.5.