Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
During the discussion to use extensions vs. accept header based routing on /api we came up with also trying to explore query parameter based routing, as it could potentially
make things much simpler when we generate URLs.
Would help with critical #2364011: [meta] External caches mix up response formats on URLs where content negotiation is in use.
Proposed resolution
Use query parameters, so a REST request getting json would look like the following: node/1?_format=json
.
Disadvantages of this approach:
- DX of extensions is slightly better and more what you would expect
- If you think in terms of rest resources, extensions are more clean than query parameters since extension is often used for formats already
- Feel free to add more entries
Advantages of this approach
- It doesn't need any adaption of the primary routing bit:
\Drupal\Core\Routing\RouteProvider
- Because of that, query parameters don't cause problems when you actually meant "/sitemap.xml" and not "/sitemap" in the format of "xml"
- Afaik main advantage: In contrast to extensions query formats are optional so node/1?_format=json might or might not have to have a similar entry in routing, in contrast
to extensions, which need an entry with a matching pattern - Given that the routing would specify the format in the extension approach, it is easier to get exception handling right for query parameters. You can determine the format as early as possible
- It's forward compatible with core or contrib making accept header format negotiation available again (and even both could be in operation at the same time)
Remaining tasks
Task | Novice task? | Contributor instructions | Complete? |
---|
User interface changes
API changes
More under https://www.drupal.org/node/2501221
Before
curl -i http://d8.dev/node/1 -H "Accept: application/hal_json"
After
curl -i http://d8.dev/node/1?_format=hal_json
Comment | File | Size | Author |
---|---|---|---|
#137 | interdiff.txt | 2.48 KB | dawehner |
#137 | 2481453-137.patch | 79.59 KB | dawehner |
#132 | interdiff.txt | 864 bytes | dawehner |
#132 | 2481453-132.patch | 80.27 KB | dawehner |
#128 | interdiff.txt | 4.88 KB | dawehner |
Comments
Comment #1
dawehnerNothing really to see yet.
Comment #2
dawehnerNothing really to see yet.
Comment #3
dawehnerThank you for @neclimdul for providing so many help with some test!
We now have a request format router filter which basically works like the accept header one, beside ignoring headers completly. Therefore we also had to drop
the negotiation which is part of
ContentNegotation
. For rendering the links we now have a outbound route processor, which, that is indeed kinda interesting, also partially work for the case of extensions.Comment #7
dawehnerApplied #2472323: Move modal / dialog to query parameters and fixed quite a bunch of tests.
Comment #8
Crell CreditAttribution: Crell at Palantir.net commentedIsn't the conditional at the start of filter() redundant with the applies() call? (Minor nitpick, I know.)
Comment #10
dawehnerComment #11
rteijeiro CreditAttribution: rteijeiro at Tieto commentedFixed a few nits. Sorry for jumping in the issue :)
Comment #14
dawehnerDon't say sorry, its cool that you do it!
Comment #15
dawehnerI replaced the general attaching of the query string towards special casing hal_json for example.
Comment #17
dawehner.
Comment #19
dawehnersome fixes.
Comment #21
dawehnerThis could be green in the meantime
Comment #23
dawehnerFixing the remaining failures ...
Comment #24
dawehnerGet up to speed with the recent changes in #2472323: Move modal / dialog to query parameters
Comment #25
dawehnerRerolled now that #2472323: Move modal / dialog to query parameters got in.
Comment #27
pwolanin CreditAttribution: pwolanin at Acquia commentedfix constants, and put back one header change that we'll fix in novice issues.
Comment #28
pwolanin CreditAttribution: pwolanin at Acquia commentedmissed one
Comment #29
pwolanin CreditAttribution: pwolanin at Acquia commentedcomment cleanup and rm class
Comment #32
pwolanin CreditAttribution: pwolanin at Acquia commentedrm old test also
Comment #56
isntall CreditAttribution: isntall at Drupal Association commentedI know most of these will probably fail, but i want to make sure that tests are getting thru.
Comment #62
pwolanin CreditAttribution: pwolanin at Acquia commentedComment #64
pwolanin CreditAttribution: pwolanin at Acquia commentedre-roll
Comment #65
pwolanin CreditAttribution: pwolanin at Acquia commentedTo enable header-based format, you'd need to change the container to use a different middleware that acts like \Drupal\Core\StackMiddleware\NegotationMiddleware and inspects the headers instead or in addition to the query string
Comment #66
dawehnerThere we go, added a couple of test coverage to ensure that we can provide some basic form of accept header based routing ...
Comment #68
dawehnerI hope this fixes it
Comment #70
dawehnerOh PHP!
Comment #72
dawehner... A module install doesn't force a router rebuild, so the router filter should check first that the service actually exists.
Comment #73
jibranNice work @dawehner
"its not" is not correct either "it's not" or "it is not" :)
This is an ugly check can we at least make it trinity condition.
Still @todo?
Can we add a quick test method for exception as well?
Comment #74
YesCT CreditAttribution: YesCT commentedsince this is a major task, can we get a beta evaluation... to make sure time spent on this is time well spent?
Comment #75
YesCT CreditAttribution: YesCT commentedComment #76
catchThis is probably going to end up being the main patch on #2481453: Implement query parameter based content negotiation as alternative to extensions but it depends on a final decision between query parameters vs. extensions where we end up doing this. The beta evaluation is the same as the parent issue though.
Comment #77
Fabianx CreditAttribution: Fabianx as a volunteer and at Tag1 Consulting commentedThe advantage of query parameters is IMHO that we can do:
- Remove all Accept headers (optional, contrib can override somehow)
- Take query argument
- Put into Accept header
- Done
Works like before.
Comment #78
dawehnerThank you for your review!
Good point, fixed that.
Fixed a bug in the actual code, but I did not converted to a ternary condition. ajax.js has really similar kind of code already.
Well, yes, we need to write up the proper test coverage and see whether this works. At the moment it won't because we have a regression from D7 to D8 in general how path aliases work.
Opened up a follow up for that, see #2490920: Add back query parameter support for path aliases.
Fixed it.
This patch includes a couple of bits (especially the test coverage for accept header based routing), which would be helpful in extension based example.
In general I'm pushing this variant because a) I think its easier in many cases and b) I want to get this longrunning issue fixed rather sooner than later.
Comment #79
dawehnerSome points which are sorta important here:
a) We change the generated URLs in hal_json. Are there other places which might need an adaption?
b) do the cases in ContentNegotiationRoutingTest look as expected?
Comment #80
Fabianx CreditAttribution: Fabianx as a volunteer and at Tag1 Consulting commentedRTBC +1, looks great to me.
But this needs maintainer sign-off.
Comment #81
Fabianx CreditAttribution: Fabianx as a volunteer and at Tag1 Consulting commentedAlso tagging for framework manager review.
I really like the simplicity of the approach used here.
Comment #82
Crell CreditAttribution: Crell at Palantir.net commentedI'll try to review this later tonight. However, I still want to be able to compare it against the extension-based approach as planned since this is a rather large decision. (Reviews from others in the mean time are most welcome!)
Comment #83
dawehnerLet me update the issue summary to make clear, why I think the extension based solution is easier.
Added some pros/cons of the two approaches.
Comment #84
webchickI can't really tell if they're captured by the pros/cons in the current issue summary, but the two biggest advantages of query parameters I remember were:
- Query parameters don't cause problems when you actually meant "/sitemap.xml" and not "/sitemap" in the format of "xml"
- Query parameters can safely be ignored if they don't apply, so if ?_format=hal_json would be nonsense for e.g. node/1/edit, you can just fallback to HTML without breaking any spec.
Comment #85
Fabianx CreditAttribution: Fabianx as a volunteer and at Tag1 Consulting commentedQuery parameters have the following advantage over extensions / accept headers:
They are not negotiated during routing, but whatever you set is taken as input value and that is also what the cache varies on.
Yes, we currently use a middleware to set the format from accept and we tried to do the same for extensions, but catch always pushed back on that as it was not exact and had to be negotiated during routing really ...
Comment #86
dawehnerGood point, made that explicit as part of the issue summary.
I hope this is documented good enough already in the issue summary?
Catch agreed that this issue should be critical.
Comment #87
catchYes on critical we need to make a decision on this or extensions to go much further forward with either, and this hasn't had much in the way of reviews.
The patch here looks pretty encouraging to me.
Comment #88
googletorp CreditAttribution: googletorp at Reveal IT commentedI found a few nitpicks when reading through the code, also the patch doesn't apply, so it needs to be rerolled.
It seems that documetation for this is missing in the RouteFilterInterface
Typo, remove one T in HTTTP.
Class methods should have () included.
Comment #89
webchickIs there any compelling reason to go with extensions? DX is listed as a point in favour, but honestly either extensions or query string paramaters are way better DX than the current "manage to type an obscure header properly by hand in cURL" way we have now. And we did a survey of major REST APIs in the issue summary of #2364011: [meta] External caches mix up response formats on URLs where content negotiation is in use. Of those, only Twitter's API uses extensions (though granted, none of them used query string params to specify format). But it seems like basically no matter what, as a developer, you need to learn the unique quirks of whatever system you're trying to interact with... there doesn't seem to be any real consistency out there in the wild.
Comment #92
googletorp CreditAttribution: googletorp at Reveal IT commentedAdding tag, now that we saw the patch failed to apply.
Comment #93
dawehnerWell, I think I have to disagree with that. Its not an obscure header, but rather a commonly known header everywhere. Typing less is not a better DX, if you ask me.
In general though this approach would continue to work even in case someone would enable accept header based routing again.
Comment #94
dawehnerThank you for your review @googletorp
git++ :)
I try to understand that, there is the following piece of documentation:
, see
\Drupal\Core\Routing\RouteFilterInterface
Fixed
Fixed as well.
Comment #95
Crell CreditAttribution: Crell at Palantir.net commentedThe comment is subtly wrong. If the route has no _format specification, we move it to the end. If it does, then no match means the route is removed entirely.
I'm unclear on why this change is needed.
The comment doesn't make sense. An entity that doesn't have a canonical link template... shouldn't be linked to, because there is nothing to link to. Also, the grammar doesn't work, and I'm not even sure what it's saying. :-)
The code, too, confuses me. If it has a canonical link, use it; else... use the canonical link?
Isn't this an unrelated functionality change? This adds html as a valid format for all Views REST Export routes.
Note: I'm not saying that's a bad thing to do; thinking about it it makes quite a bit of sense, although that may render the point of setting _format irrelevant anyway. But it seems off topic here.
Is that accurate? From the code above it doesn't feel like it. If _format is set on a route, a non-specified format is removed entirely. It's not considered, even if it's HTML.
(This comment is repeated multiple times; if changed, remember to change it in the other places, too.)
If you're going to change one array() -> [], change all of them on the same line. :-)
(This comment applies to most of this class.)
Whichever approach we end up taking, I like this method! Should there be a GET version, too?
In these comments, can you specify in human words what the expected behavior is? This looks like it's testing that extensions work, but this is the query parameter patch I thought... (The FALSE on the end tells me nothing.)
If we can't use PHPUnit's lovely data providers here because it's KernelTestBase (*cries!*), at least we can name the array parameters to avoid positional data.
<3
TestController? To follow convention...
Comment #96
dawehnerThank you a lot @crell for a review!!!!!
Agreed, fixed.
Let's remove the registering of formats entirely. I think that is not needed anymore or rather, it is not allowed to be needed anymore.
Well, there are tests with entity types without canonical links. If we don't do that here, we end up with failing tests.
Well, I remember that it was needed in addition to the changes in Serializer.php, but I don't understand anymore why. This is why reviews and quick reviews are so essential :)
Anyway for now I reverted the bits in Serialzier.php and RestExport
Let me quote yourself :P
We could, this is for sure. What about moving that to a follow up?
Well, that
simple.json
example is the usecase ofsitemap.xml
so a valid check. Having this kind of test coverage here, is not a bad thing IMHO.I'd be happy with some help on describing the various cases here.
Well, not long anymore :) #2304461: KernelTestBaseTNG™
Fixed that.
Comment #98
dawehnerLet's register the mimetypes again, for now just for hal_json because the other ones are not REST specific.
Comment #99
googletorp CreditAttribution: googletorp at Reveal IT commentedI have some documentation things that need work.
Missing the variable name. This error appears twice, but missed the second when I pasted the code in.
RequestFormatRouteFilter class should have the full namespace, when referenced in comments.
The method has no documentation.
No class or method documentation at all.
Comment #100
dawehnerThank you. Quickly fixed those.
Comment #103
dawehnerLet's see
Comment #105
dawehnerOne less.
Comment #107
dawehnerWell, that bit is tricky. What we want is to return JSON, even we request actually just HTML, as the user clicked on a link.
Comment #109
Crell CreditAttribution: Crell at Palantir.net commentedThen those entity types are wrong. A linkable entity MUST have a canonical link. This is why. :-)
Works for me. Please file.
Comment #110
dawehnerIn the meantime we have one
Comment #111
dawehnerJust general review / cleanup.
Comment #113
dawehnerUps.
Comment #114
dawehnerUpdated the issue summary
Comment #115
alexpottThis was discussed on the recent EU criticals call - both @catch and I agree that we should proceed with this solution since it requires the least amount of change and leaves the option of exploring accept headers in contrib.
Comment #116
catchPretty sure that won't work. See #118072: Allow query strings in URL aliases. However also don't think that matters. You can't alias an explicit format with accept header negotiation now either.
Nice to see test coverage proving this is possible.
I have mixed feelings on throwing the 406 when the query parameter doesn't match, but they are just mixed feelings - not sure any other alternative would be better and that's something we could discuss further in a follow-up.
Overall this is very encouraging.
Comment #117
dawehnerWe clarified on IRC, that it actually works as expected.
If you have setup a normal route and a json route, and want XML it will return a 200 with the normal route.
Comment #119
tim.plunkettThis needs a
$requirements = [];
before it now, this failed for me. Therefore tests too?Comment #120
dawehnerGood catch! Yeah I guess we don't have a dedicated test for that usecase.
Comment #126
dawehnerI think we no longer need an issue summary update? I also think @crell, @tim.plunkett are fine with the taken approach.
Comment #127
Crell CreditAttribution: Crell at Palantir.net commentedSome mostly minor commentary. Nothing fatal, I think. It sounds like we're all on board with this approach so this is just polish.
I still think this is working around a bug rather than fixing it. If fixing it is off topic here, at least let's add a @todo to fix those entities? If an entity can be linked to, it MUST provide a canonical link. That's the point of a canonical link.
Why the variable rename? Seems unnecessary...
dawehner explained this block to me in IRC as being mostly for debug support; ie, if you define a REST View in the UI and then click on it to see if you didn't screw it up, you want the browser to show the output. That's not clear from the comment, which leaves me going "WTF? Why would I want to always return HTML?"
Suggested alternative comment:
"Allow a REST Export View to be returned with an HTML-only accept format. That allows browsers or other non-compliant systems to access the view, as it is unlikely to have a conflicting HTML representation anyway."
Nitpick: The defaults in the method signature should use [], since the method body does so.
Same as previous.
OK, that works I guess for documentation. :-)
Comment #128
dawehnerWe seem to not longer need the this, the tests still pass ...
Let's fix it ...
Thank you!
Yeah, I promise to convert it to KTBNG, once its available for usage ...
Comment #132
dawehnerFixed the last failure ...
Comment #133
Crell CreditAttribution: Crell at Palantir.net commentedI think that covers it on the code front. Thanks, dawehner!
This technically needs a change record or record update or something. I'm not sure which. But RTBCing anyway to pressure dawehner into writing it before committers notice. :-)
Comment #134
dawehnerThere seemed to be one actual change record about REST: https://www.drupal.org/node/1975444
There are some minor updates, but nothing about accept header based routing directly.
I would suggest to a) add a new one b) update existing ones like https://www.drupal.org/node/2199185 and https://www.drupal.org/node/1989646
Will work on updating them, once the issue is done
On top of that, we should update the handbook: https://www.drupal.org/node/1972016
Comment #135
catchOverall this is great, some minor things in test coverage but also one @todo asks quite a big question that I think we need at least a follow-up to figure out and document.
Can we remove this and add it as an interdiff to the path alias query string issue?
This could do with a follow-up @todo, or we should decide in this issue what's going to happen with aliases. But this comment is a bit of a shrug as it is.
$test[3] (vary) is never used? Is it cruft or should we be asserting the header?
This doesn't have vary this time (and nor do the actual test arrays).
Comment #136
catchAlso general note this conflicts with #2381277: Make Views use render caching and remove Views' own "output caching" but it's a minimal conflict in either direction.
Comment #137
dawehnerPut the interdiff on the alias issue.
Well, for query parameters this actually works, doesn't it?
Yeah I killed most of them before.
Comment #138
catchYeah if you have /about and do /about?_format=foo then that should work.
What you can't do at the moment is create about-json and point it to node/1?_format=foo but that's a long-standing thing and has a feature request #118072: Allow query strings in URL aliases. Just removing the @todo is good, I don't think anything is broken there by this patch.
Comment #139
dawehner@catch
Do you have any other review points?
Comment #140
catch@dawehner, not really, I'm pretty happy with this.
Moving back to RTBC. I'll either give it another once over then commit tomorrow unless I spot something, or if someone else beats me to the commit that's fine too.
Comment #142
catchGave this another look over and nothing more to add. We should open a follow-up for the entity canonical route issue Crell brought up, and we still have #2364011: [meta] External caches mix up response formats on URLs where content negotiation is in use open for any additional follow-up to be tracked on.
I added neclimdul and znerol to commit credit due to patches on the other issue.
Committed/pushed to 8.0.x, thanks!
Comment #143
alexpottHere's my review - I guess this can all be handled in followups - no biggie :)
Nice - this looks a lot simpler :) And it's great that we're able to move
AcceptHeaderMatcher
to a test to ensure we maintain the ability to do it in contrib.Can we get a followup to remove the last remnants of drupal-ajax (used in tests alot) and it's drupal- friends. Or maybe it should be cleanup here? We still have things like:
and
Not the fault of this patch but at some point we should have a service name standard. Note: this is just @alexpott grumbling.
Looks like this accepts a
Url
object too.I think this should be:
Comment #144
dawehnerThank you alex!
Opened up #2502865: Remove all remaining usages of the drupal_ajax accept header
dawehner nods
Yes it does ...
Added a follow up: #2502867: Document all drupal(Post|Get)(*) methods $path parameter
Comment #145
mikeker CreditAttribution: mikeker as a volunteer commentedJust posted a follow-up in #2503643: Follow-up: #2481453 (content negotiation) causes Simpletest to fail on Windows.
Comment #146
Wim LeersWow, congrats!
Gentle nudge to @dawehner for the
tag.Comment #147
Jaesin CreditAttribution: Jaesin at Chapter Three commentedAnother followup. This one is about plugable type negotiation in contrib. It seems like it would make more since to have the type negotiation override in the ContentNegotiation class rather than having to replace the NegotiationMiddleware but there is no interface for ContentNegotiation and NegotiationMiddleware is expecting the ContentNegotiation class during instantiation.
#2506533: Remove ContentNegotiation and embed functionality in the middleware
Comment #149
fgmSorry not to have seen this one earlier : while moving from extension to query is as good a move as described, and switching from Accept header to query as a default makes /some/ sense, actively removing the ability to use that header in content negotiation within core, relegating its use to contrib strikes me as a significant loss, especially when REST best practice is a D8 sellling point.
And, yes, I know Fielding says content negotiation needn't be used constantly, but this is mostly in the context of published content to be consumed by browsers, not API access by smart clients like servers and apps.
Edit: quoting http://www.newmediacampaigns.com/blog/browser-rest-http-accept-headers :
Comment #150
Wim LeersUnfortunately it turns out this was incomplete: #2825347: 'Accept' header still is used for determining response format of error responses, ?_format= has no effect AND serialization module's exception subscriber does not support status code 500 error responses.
Comment #151
klausiThis issue has moved unit tests to a location where they never get executed, opened #2859538: system/tests/modules/accept_header_routing_test/tests are in the wrong place, never get executed.