REST: top priorities for Drupal 8.6.x

The IS above is identical to #2905563: REST: top priorities for Drupal 8.5.x, just with all completed things removed.

#2543726: Make $term->parent behave like any other entity reference field, to fix REST and Migrate support and de-customize its Views integration was already committed to 8.6.x! One thing already done, hurray!

1. Moved #2856110: [PP-1] Expose entity validation errors in a machine readable REST API: into the "validation" bucket.
2. Added #2926508: Add DateTimeNormalizer+TimestampNormalizer, deprecate TimestampItemNormalizer: @DataType-level normalizers are reusable by JSON:API.

@gabesullice started the work on a module-agnostic way to mark entity types as "internal" to exclude them from HTTP APIs like the REST module, https://www.drupal.org/project/jsonapi and https://www.drupal.org/project/graphql: #2941622: Make REST's EntityResource deriver exclude internal entity types and mark ContentModerationState entity type as internal instead of the current hack, which was blocked on #2936714: Entity type definitions cannot be set as 'internal' (landed this morning!), which was blocked on #2936725: EntityDataDefinition::create() does not respect derived entity definitions (landed some time ago)!

#2941622: Make REST's EntityResource deriver exclude internal entity types and mark ContentModerationState entity type as internal instead of the current hack is the core REST issue. #2932035: ResourceTypes should be internal when EntityType::isInternal is TRUE is the JSON API issue (which we don't track in this issue).

This is very important for the entire API-First ecosystem, because it means consistency across different HTTP API modules!

Moving Use @DataType-level normalizers whenever possible + Module-agnostic way to mark entity types as "internal" to exclude them from HTTP APIs like the REST module, https://www.drupal.org/project/jsonapi and https://www.drupal.org/project/graphql to a new Ecosystem bucket.

That makes more sense. None of these things are hard blockers for the REST module, but they do allow REST technical debt to be reduced, and most importantly allow for a coherent API-First ecosystem of modules for Drupal 8.

#2941622: Make REST's EntityResource deriver exclude internal entity types and mark ContentModerationState entity type as internal instead of the current hack landed late yesterday!

Added #2946972: EntityResource: revisions support.

Adding #2938035: When PATCHing a field is disallowed, no reason is given for *why* this happens to the DX section.

Clarifying #9.

Adding #2955383: List available representations in 406 responses to the DX section.

#1818574: Support config entities in typed data EntityAdapter is a blocker for #2300677: JSON:API POST/PATCH support for fully validatable config entities. Let's make that more explicit. full config entity support

#1927648: Allow creation of file entities from binary data via REST requests landed! 🎉🎉🎉🎉🎉🎉

See #2300677-242: JSON:API POST/PATCH support for fully validatable config entities — we had a big meeting earlier today at DrupalCon Nashville to ensure that we had consensus between component maintainers and framework managers about how we'd make this a reality, and we did! So this is now in a place where we can iterate very quickly on it from now on :)

#2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON was fixed as part of #2905563: REST: top priorities for Drupal 8.5.x. But that fix revealed that apparently many, many sites were relying on the debug-DX-enhancing behavior of automatically picking the first format for REST export displays of views … meaning that updating to 8.5 (which included #2449143)
broke their site. The long-standing issue #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't fixes that, but never gained traction. Until this affected so many 8.5 sites.

So: adding #2854543!

#2910883: Move all entity type REST tests to the providing modules landed!

#2938035: When PATCHing a field is disallowed, no reason is given for *why* this happens landed! A big DX improvement that first existed in https://www.drupal.org/project/jsonapi and now also is available in rest.module 🎉

#2955383: List available representations in 406 responses's blocker (#2955685: Unrouted URLs cannot have have overridden query or fragments) just landed!

#2955383: List available representations in 406 responses landed during the weekend!

Added #2977882: Term entity type's "parent" field does not specify "target_bundles" setting: a term from *any* vocabulary can be chosen.

#1818574: Support config entities in typed data EntityAdapter landed! 🎉🎉🎉🎉

#2300677: JSON:API POST/PATCH support for fully validatable config entities is now unblocked!

#2957385: FieldItemNormalizer never calls @DataType-level normalizer service' ::denormalize() method is a blocker for #2926507: Discourage @FieldType-level normalizers, encourage @DataType-level normalizers, to strengthen the API-First ecosystem.

(A sister issue for that in JSON API has a green patch: #2955615: Field properties are not being denormalized.)

#2935738: Entity reference field subclasses (such as file and image fields) lose the non-default properties upon denormalization, in both hal_json and json is another major serialization gap bug.

#2935738: Entity reference field subclasses (such as file and image fields) lose the non-default properties upon denormalization, in both hal_json and json landed!

#2977882: Term entity type's "parent" field does not specify "target_bundles" setting: a term from *any* vocabulary can be chosen landed!

To my surprise, #2895532: @DataType=map cannot be normalized, affects @FieldType=link, @FieldType=map was not yet listed above! Nor was #2915705: TypedData 'Any' can't be normalized to array if it stores an object.

And #2895532: @DataType=map cannot be normalized, affects @FieldType=link, @FieldType=map landed earlier today! 🤘

Thanks everybody for helping out here!

We made less progress than I hoped in some areas. But some foundational pieces finally clicked into place. Most importantly, core's rest.module is now in a maintainable state. For the past 4 months, the number of open issues has been below 50, about half of which are feature requests, and a fifth are blocked on other subsystems. And as of Drupal 8.6, each core module that has an HTTP API surface is now responsible for its own test coverage. See https://wimleers.com/blog/api-first-drupal-two-big-maintainability-miles....

Note that of the 14 issues that are tagged with 8.6.0 release notes, of which 2 are for the API-First Initiative, and 22 issues are tagged with 8.6.0 highlights, of which 4 are for the API-First Initiative — or 14% of all release notes issues and 18% of highlights!

Now it's time to move on to 8.7.x. I've opened #3002188: REST: top priorities for Drupal 8.7.x for that, which is the continuation of this issue.

Thanks, and see you around!

(Sister posts at #2905563-48: REST: top priorities for Drupal 8.5.x, #2852860-41: REST: top priorities for Drupal 8.4.x, #2794263-64: REST: top priorities for Drupal 8.3.x and #2721489-48: REST: top priorities for Drupal 8.2.x.)

Let's do a restrospective. We landed:

Any use case

5 issues

DX

3 issues

Coherent API-First ecosystem

3 issues (over half!)

Velocity+maintainability+reliability+predictability

1 issue

Many of them were very difficult to land, requiring lots of consensus building, or requiring blockers to be solved that aren't listed here, often in other subsystems. One of them stood out in particular: file upload support, see https://wimleers.com/blog/api-first-drupal-file-uploads and https://wimleers.com/blog/api-first-drupal-file-uploads-572-comments-sum.... Many of the ones that did not land are already well on their way, making it very likely that they will land in 8.7. In fact, one of them already landed in 8.7: #2922487: Follow-up for #2910211: fix all deprecation warnings!

I now see I failed to link to https://wimleers.com/blog/api-first-drupal-8.6! 😱