REST views: regression in 8.5.x: view with display using 'csv' format now returns a 406 response, need to add ?_format=csv to URL to make it work

Let's see what changed in RestExport:

$ git log 8.4.0..8.5.0-alpha1 --oneline -- core/modules/rest/src/Plugin/views/display/RestExport.php
850f03b Issue #2825204 by dawehner, BR0kEN, xjm, pcambra, Wim Leers, tim.plunkett, tstoeckler, damiankloip, larowlan, effulgentsia, alexpott: REST views: authentication is broken
8f022a6 Revert "Issue #2825204 by dawehner, BR0kEN, pcambra, Wim Leers, tim.plunkett, tstoeckler, damiankloip: REST views: authentication is broken"
40b143f Issue #2825204 by dawehner, BR0kEN, pcambra, Wim Leers, tim.plunkett, tstoeckler, damiankloip: REST views: authentication is broken
4997192 Issue #2449143 by damiankloip, tedbow, Wim Leers, dawehner, Pavan B S, Tybor, effulgentsia, larowlan: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON
f659d6c Issue #2901726 by mfernea, gmario, finn.lewis, robertoperuzzo, rachel_norfolk: Fix 'Squiz.Functions.MultiLineFunctionDeclaration' coding standard
wim.leers at acquia in ~/Work/d8 on 8.5.x*

So that's:

1. #2825204: REST views: authentication is broken
2. #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON
3. #2901726: Fix 'Squiz.Functions.MultiLineFunctionDeclaration' coding standard

I think the latter can safely be ignored. My bet is on #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON.

Looking at your report more closely:

The views have URLs like /some/path/to/download/content.csv
They work if I change the URLs to /some/path/to/download/content.csv?_format=csv

Ah, yes, that is almost exactly the situation of the major bug that was being fixed there: if you had a taxonomy term html view and a REST export (e.g. json) at the same path, then it was impossible to access the HTML view!

For BC shouldn't we detect that there are no html displays and continue to display the other format (be it json or whatever) - or maybe even better - issue a redirect?

#3 + #4: quoting @dawehner from #2449143-188: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON in October:

Well, the question is, can we do something in some automated update path? Could we automatically select 'json' as one of the format, given it was the fallback previously as well? I guess we can't given that's the entire point of the issue.

To which I replied Exactly., and then no more discussion followed.

I'll ask @damiankloip to look at this.

For BC shouldn't we detect that there are no html displays and continue to display the other format (be it json or whatever) - or maybe even better - issue a redirect?

This is a bit tricky, given it could be even another route adding this.

RE #7 yes, a route could be added in via a later hook/event to views too. There is no real way to guarantee anything consistent around that, unfortunately...

I was initially hoping we could get around this by using some of the features Symfony bakes into its routing layer around the _format parameter (See https://symfony.com/doc/current/routing.html#advanced-routing-example and below). 1. being that you can use {_format} as a path slug that is then set as the request format, and 2. setting _format as a default on the route. Alas these are both a no go as we don't actually use Symfony for our routing. We, of course, use our own format handling (Looking at you RequestFormatRouteFilter) which lacks these features.

I tried playing around with the default format, but you cannot have that along with route requirements. The requirements are used first. RequestFormatRouteFilter also has some strange handling around what the default format should actually be. I.e. it doesn't care is you set a default _format or not! I'm not sure I want to get into making changes to that class here.

So, IMO, we have possibly one more option here. It might work if we add a setting that allows us to opt in to behaviour more like we had before. Which is basically, add html as an allowed _format to the requirements of the route. Maybe an option that can be used like 'Allow requests using HTML format to return default format' (or something). Then have a way to select which is the preferred default format to return. Otherwise, not sure what else we can do here.

Allow requests using HTML format to return default format' (or something)

I think I'd hide this detail but rather talk about the fact that you are able to download the data using a browser.

It might work if we add a setting that allows us to opt in to behaviour more like we had before.

But that'd still break BC, because we'd have no way of knowing whether a site was relying on it or not. We can't just enable this for all existing REST views, because then we'd not fix the bug.

Hi all,

I might not be in the exact same situation but it seems like the same root issue here: I was playing around with Views on Drupal 8.5.0-beta1 today, and I've been facing a very similar problem when creating a "REST export" display (using core module "rest"): Even though I was telling Views it could accept JSON and/or XML, it kept responding with an HTTP 406 error with the following message on screen (as an HTML web response with Drupal theme and all - not as a JSON/XML response as one would expect):

Client error
A client error happened

Tried several combinations of Headers with a REST client (ARC), most of my trials focused on several configurations of the "Accept" header. I was expecting the View to reply with a JSON response when setting "Accept" to "application/json".

When I found this thread, I tried adding query parameter "_format=json" to my request and everything suddenly started working like a charm.

Shouldn't the REST module take the HTTP header "Accept" into account by default?

@pacproduct: Drupal 8 chose not to support Accept header-based negotiation. See https://www.drupal.org/node/2501221 for details, but in short: browsers and (reverse) proxies have very poor support for it and can cause severe problems.

It sounds like what you're really reporting, is that the "REST export" views plugin should explain in its UI how to access the view in one of those formats.

@wim-leers: Thanks for the heads up.
Documentation could be a way forward, or at least it would make it clear how to access the resource.

However I am not convinced Views should react this way: we should be able to return JSON/XML on a given path even without "_format" parameter.
When building an endpoint for an existing external service, we shouldn't have to require some additional parameters like "_format".
But that's just a personal opinion...

Maybe we just need (or is there) a way to set a default format for the Views? As in if it does not find any "_format" parameter, it fallbacks to serving the default format (JSON for instance) configured in the Views' display?

I found this as well now.

The interesting thing is that this even applies to views_data_export-based views, including those that use #2789531: Support for batch operations, so you don't *really* want to access a csv format. You click on the CSV icon views attachment/icon, then you get to a batch page (HTML obviously), and then you are redirct back to the view, with a download link (obviously again HTML).

Since you are forced to select the request format, this now results in a Client error when clicking on the CSV icon. I guess a workaround is relatively easy by just explicitly adding the format to the generated link, but it's still a bit strange. But possibly views_data_export also shoudn't actually require you to select a non-html request format if you use it with batch.

I solved it with an addition in #2789531: Support for batch operations for the use case of attached displays. I was wondering if the RestExport should override the getUrlInfo() method and add the _format there, and initially started with that but then noticed that views_data_export overrides the query array anyway, so didn't bother to create a patch for that yet.

I think one thing that I've mentioned before is that if a route specifies a single format, then at some point during routing, we should just default to that format instead of hardcoding that it is HTML. But maybe that would cause regressions again, don't know.

@pacproduct: everybody agrees that that would be nice, but as explained above, this is not possible without breaking things. "Slightly less nice, but works reliably" vs "Nicer, but works unreliably" is the choice to be made here, and we went from the latter to the former.

@Berdir: thanks for posting your additional insight! To be clear: you also don't see a solution here, right?

@WimLeers: Well, I did propose something in my last paragraph? IMHO, if only a single route matches and that route only allows a single, specific format, then we should just use that and not require _format. If there are multiple routes with different formats, then that doen't work of course.

I'm curious: Why can https://www.drupal.org/project/views_data_export not define a default value for _format in the route by having an additional checkbox?

I've added the @Berdir workaround in Views Data Export #2951185: Download link returns 406 with Drupal ^8.5.0.

This seems to break all rest exports, not only csv? I have some feeds which have don't have any format defined, but they are being consumed as JSON by default. Now _format=json is required, _format=xml also works.

This is a regression because a link that worked in Drupal <= 8.4.x is broken now, with Drupal 8.5.x. For this reason I think it's at least Major if not Critical.

Is there any known workaround to fix this in a custom module? I don't want to install more contrib modules if this can be done in a alter hook or extending Serializer for example?

I think setting _format on the route defaults could do it :)

Using any hook_view_* will not work because they are never called. I tried to do something in a custom RouteSubscriber but that didn't work either because the route is system.4xx due to the 406 error. Do I have to override the RestExport.php views display plugin with a custom one?

After updating to 8.5.0 today, had all of my REST exports break because of this issue. Requiring ?_format= strings at the end of URLs is frankly bad design when better options exist. I understand that browser support for header negotiation may not be universal, but it is supported by most major browsers and worked fine until 8.5.0. Wouldn't it be a better idea to use ?_format= as a fallback rather than a requirement?

Frankly, I'm disappointed that Drupal 8.5.0 released without this being fixed, as this issue completely destroys applications that use Drupal as a backend for data requiring emergency updates to make them work at all.

After updating to D8.5.0, all my REST Exports (json) break. Log detail below :

Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException : No route found for the specified format html. in Drupal\Core\Routing\RequestFormatRouteFilter->filter() (line 55 of /Applications/MAMP/htdocs/adeupa-d8/web/core/lib/Drupal/Core/Routing/RequestFormatRouteFilter.php)

In the Format settings, json was unchecked. I checked it, but the issue remained.
I think it is (very) critical, not major.

I agree with the other sentiment about 8.5 being released without this being fixed. I had a lot of functionality break because of the URL requirement change which turned into emergency patches and unexpected downtime.

This was reported again by @mpdonadio at #2954286: REST views: Default format no longer works. Quoting his report verbatim:

[Setting this as a major, as it may cause breaks on live sites. Also not 100% sure if this is a proper bug, or just something that should not have worked before.]

I have a view over a content type with a REST Export display on it. This particular view exports data as an XLS download via the XLS Serialization module. The display has a path defined on it, /export/mydata.xls Since I need my user's to do this without my help, I also have a menu item for /export/mydata.xls so they can download these reports whenever they want (I have a somewhat related entry about REST Export displays and menu entries somewhere). Other than the master, this is the only display on this view.

This has been working happily for a long time (potentially as early as 8.0.x as this was my first real site in this branch).

When I updated to 8.5.0, the display broke,

Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException: No route found for the specified format html. in Drupal\Core\Routing\RequestFormatRouteFilter->filter() (line 55 of /var/www/mysite/web/core/lib/Drupal/Core/Routing/RequestFormatRouteFilter.php).

I had to update the menu item to /export/mydata.xls?_format=xls (can't do this on the display). Not a big deal, but a little confused why this is happening now. This could be related to #2858482: Simplify REST routing: disallow requesting POST/PATCH in any format, make consistent, but I am not 100% sure.

All existing rest export views are broken after upgrading to 8.5 on our site and this made a huge mess for us.

We have existing (desktop) software using these rest exports and without updating software (with new url's with _format parameter) we are currently not able to resolve this. Updating software will also resolve problem for latest update. All existing old builds will still have problem.

There should be a solution on Drupal side (using "default" format if format is not provided in url) for this and similar cases noted above.

For example, if only json format is set in Style options (Accepted request formats in Views, rest export Format settings) than format should be set to json even if it is omitted in url.

This would make website compatible with old clients with missing format parameter.

Here is a demo test.

Oops. Left in some debug.

Maybe something like this? Ignore the files/ view / display / class names, as I just don't feel like renaming them right now...

The idea is that if a view has multiple displays on it, and none support HTML output, then allow HTML output for anything that can have a path.

I suspect there are some edge cases to consider...

#15/#17:

I think one thing that I've mentioned before is that if a route specifies a single format, then at some point during routing, we should just default to that format instead of hardcoding that it is HTML. But maybe that would cause regressions again, don't know.

Actually, that is what #2854560: \Drupal\Core\Routing\RequestFormatRouteFilter::filter() is too HTML-centric implemented!

Sadly, #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON still gets in the way. If we'd fix that, that'd work.

This would be a reliable solution for any route that will only ever have one format. But routes that would gain support for a new format at a later time would then suddenly start failing. Because only as long as there's a single format, it can automatically guess. So the URL that worked fine would cease to work. Just as it did here. The solution would still be the same: explicitly specify the format.

#18:

I'm curious: Why can https://www.drupal.org/project/views_data_export not define a default value for _format in the route by having an additional checkbox?

#29:

For example, if only json format is set in Style options (Accepted request formats in Views, rest export Format settings) than format should be set to json even if it is omitted in url.

This would make website compatible with old clients with missing format parameter.

Agreed that that would be nice.

But … remember that if you then later add a new format, that it'd then start failing. Because as long as there's a single format, it can automatically guess. Once there's more, it can't. So the URL that worked fine would cease to work. Just as it did here. The solution would still be the same: explicitly specify the format.

We have existing (desktop) software using these rest exports and without updating software (with new url's with _format parameter) we are currently not able to resolve this. Updating software will also resolve problem for latest update. All existing old builds will still have problem.

You can work around this by doing either custom PHP code or a URL rewrite rule that redirects /foo to /foo?_format=something.

#25:

I understand that browser support for header negotiation may not be universal, but it is supported by most major browsers and worked fine until 8.5.0.

No, it did not work fine. Until 8.5, it just a format, without negotiating at all. That's the very bug that solved.

Wouldn't it be a better idea to use ?_format= as a fallback rather than a requirement?

No. See my reply to #15/#17 as well as #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON. The entire problem is caused by clients not being explicit about which formats they expect. And due to a poorly thought through decision before 8.0.0 shipped, that's what we ended up supporting, along with the bugs it caused. We now require you to explicitly specify the expected format. In doing so, we indeed do require some users to update some client URLs. But it's literally appending ?_format=SOMETHING, nothing more. It's painful, annoying, but also required. It was required to fix a major bug that had been open since before 8.0.0 was released, and which should've been fixed before 8.0.0 but alas wasn't.

Everyone: I'm very very very sorry for the pain this has caused. The reason this is causing friction is that the way it worked in 8.0.0 through 8.4.0 was fatally flawed (and was introduced in #1969870: REST export view should default to JSON). It was done for the sake of simplifying debugging… because people found it too hard to specify an Accept request header (back when Drupal still did Accept header based content negotiation). Well-intentioned, but with unintended/unforeseen consequences!

Technically speaking, all clients were actually doing the wrong thing: it always was required to explicitly specify a format for a request if you wanted to be sure it never would change. Not specifying a format (initially through Accept: …, then via ?_format=…) was only a debug nicety.

Of course, this doesn't help you.

This shipped in 8.0.0 as stable, but it really wasn't stable/well thought through. And in fact, #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON was already reported before 8.0.0 shipped!

At some point we're going to have to make this painful change, because only when clients specify the format they request we can guarantee that things won't break in the future.

In other words: there is no painless solution.

What is unquestionable a mistake of ours back when we worked on #2449143, is failing to write & publish a change record. Did so just now: https://www.drupal.org/node/2954953.

Bumped #2854543: #2854543-27: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't.

Finally: we sadly had no way to know that apparently so many people were relying on a debug feature :( Only #2916212: Fix rest tests. was reported at #2449143-185: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON, and that was a contrib module whose tests were easily updated to do "the right thing". If we'd known so many people were affected, we'd have prioritized #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't accordingly.

Fortunately, this is a one-time fix! Make your requests more specific once, and benefit forever from painless updates!

@WimLeers, thanks for pulling together all of the related, intertwined issues.

Taking a step back and thinking out loud here, but I think this could help explain the situation.

We have the RestExport display for REST. Because content-negotiation is horribly broken in places out of our control, we use the _format query parameter to force it. Since REST is API centric, this really isn't an issue, as it is supposed to be a machine consumable resource.

However, there are needs for non-HTML serialized output that are human consumable. CSV and XLS are two obvious ones, where reports / exports get downloaded by people via menu links, but I used to routinely trade XML files back and forth with clients (ie, XML as a file format and not an on-the-wire protocol) and I have been doing this lately with JSON. Typically, these get done with menu links (see also #2848971: REST views: change PathPluginBase so RestExport displays allow menu paths (for serialization formats such as CSV or XLS)). Specifying the _format or the need to is not exposed via the UI (ie, UX bug).

So, people in human consumable camp have been using the machine consumable solution, and it has been working accidentally because of a debug feature.

Where am I going with this? It seems that RestExport should

- be moved to SerializedExport, where the collectRoutes() method ignores the _format
- RestExport extends SerializedExport, there the collectRoutes() adds in the _format logic

Then we fix help text to explain all of this to help users select the proper plugin in Views, and figure out a way for site builders to update their views.

?

Interesting analysis and description, @mpdonadio, thank you! 👏

The two "human consumable" use cases you cite are CSV and XLS. (I'm sure there's more.) But there's no reason why you can't have both at the same time, at the same URL. For example, something like: https://example.com/data-for-last-24-hours?_format=csv + https://example.com/data-for-last-24-hours?_format=xls.
My point being: you can just as easily run into the same problems even with "human consumable" use cases.

Where am I going with this? It seems that RestExport should […]

Wouldn't that make the update path even more complicated? And wouldn't that make the site builder UX even more difficult? To support the use case of "I'll configure a separate path for each format", we could just as well fix #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't?

#43, yeah, that is a good (and likely) edge case, and yes that update path would be painful.

#2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't does seem like the best stopgap for the current issue w/ broken things in 8.5.x.

The last patch on #2848971: REST views: change PathPluginBase so RestExport displays allow menu paths (for serialization formats such as CSV or XLS) has parts that may help the general case, at least for URL generation, but that has a fail that I am starting to dig into.

The biggest issues with this seem to be:

A. common javascript libraries such as angularjs and jquery have helper functions that automatically specify accept headers, such as $.getJSON and $http.get. These worked beautifully at consuming JSON created by Drupal Views. Yes, the fix is relatively simple, but the change is also breaking meaning these functions now fail, and the error message returned is not clear as to why or how to fix it.

B. Drupal Views data in this format is being consumed by enterprise applications. These applications run on a strict release cycle, require 24/7 up-time, and power critical services. This update, with no change record to indicate such, broke all of these calls. This means people being woken up in the middle of the night to troubleshoot issues in their application ultimately caused by an update in a different application whose data they happen to consume. I work in such an enterprise, that handles sensitive topics such as child abuse issues. Perhaps you can understand my anger at undocumented breaking changes in a minor release.

the error message returned is not clear as to why or how to fix it.

It's a 406 response, isn't it?

[…] require 24/7 up-time […] power critical services. This means people being woken up in the middle of the night to troubleshoot issues in their application […]

I say this with the best of intentions, but I fear this will still anger you, yet it needs to be said.
If this broke those services, then there's a serious problem with your development process, because it means no/insufficient QA was done. Your QA needs to catch these problems; your QA is flawed if critical services were disrupted because of this. Please use this unfortunate incident to improve your QA for the future!

Software is imperfect, processes are imperfect, humans are imperfect.

#2449143 was committed in October, yet nobody reported this in >5 months. Run your QA tests against the next minor branch of Drupal. Report failures you encounter. This both helps your QA and helps Drupal's! We had no way of knowing so many sites were relying on a debugging feature.

Finally, it pains me to say, but the "REST export" functionality has quite a few problems, and nobody is working on them: https://www.drupal.org/project/issues/drupal?text=REST+views&component=r.... Drupal is open source software, and it's only with the help of everybody in the community that we can make every aspect of Drupal rock-solid, well-tested and a pleasure to use.

Again, sorry for the pain this caused. But believe me when I say that we don't commit/ship things lightly. Check the comments in #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON. It took 181 comments over 2.5 years to get it committed. Participate to help us be better: that means even more use cases are taken into account while reviewing!

In the mean time, I just spent an entire day working on #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't to make this better for you and others. Please help!

It is a 406, specifically, it says:

Error 406 not acceptable

Client error, A client error happened

From the Mozilla MDN:

If a server returns such an error status, the body of the message should contain the list of the available representations of the resources, allowing the user to manually choose among them.

You're absolutely right that Drupal's 406 responses should be better! This has frustrated me too. However, for most, a 406 is sufficient, and hence improving this was never prioritized.

AFAICT I first encountered this in #2737719-41: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method, from Q3 2016:

[…]
Worse, it's even possible to get the appropriate 406 response in case of an invalid format, but for that response to be sent with Content-Type: text/html! (In case of anonymous.)

More edge cases than I can count.

Seems like the authentication system and routing system are integrated in a brittle, confusing, backward manner. And the content negotiation system makes it that tiny bit extra unpredictable. This is no one's fault. This is complex software with (too) many layers, and without comprehensive integration tests, this sort of thing is to be expected.

So, the updated test coverage verifies this crazy behavior is at least consistent. And then when we fix it, in #2805279: Routing system + authentication system + format-specific routes (e.g. those in rest.module) = frustrating, unhelpful 403 responses instead of 406 responses, we can massively simplify this crazy test logic. It's better to embrace and accept the status quo, because that means we are better prepared to improve it :)
[…]

So, see #2805279: Routing system + authentication system + format-specific routes (e.g. those in rest.module) = frustrating, unhelpful 403 responses instead of 406 responses. I'm hoping to see you push the patch there forward!

Also note that https://tools.ietf.org/html/rfc7231#section-6.5.6 says

6.5.6.  406 Not Acceptable

   The 406 (Not Acceptable) status code indicates that the target
   resource does not have a current representation that would be
   acceptable to the user agent, according to the proactive negotiation
   header fields received in the request (Section 5.3), and the server
   is unwilling to supply a default representation.

   The server SHOULD generate a payload containing a list of available
   representation characteristics and corresponding resource identifiers
   from which the user or user agent can choose the one most
   appropriate.  A user agent MAY automatically select the most
   appropriate choice from that list.  However, this specification does
   not define any standard for such automatic selection, as described in
   Section 6.4.1.

… i.e. there's no standardized way to communicate the list of acceptable formats. Also see https://httpstatuses.com/300. See https://github.com/Respect/Rest/issues/39 for an example discussion of a project attempting to implement this … and getting sucked into an endless discussion. That being said, I think we should do something until it's standardized.
Opened #2955383: List available representations in 406 responses for that and created an initial patch. I hope you'll appreciate that — I stayed up late to do that just for you ;)

Prior to my upgrade to Drupal 8.5, I could do a GET request to my view's rest export and I would get a 200 response.

After upgrading to Drupal 8.5, in Postman, I am getting a 500 response when I make a GET request to my rest export view, even if I append ?_format=hal_json.

However, in my browser, I can see the json output if I append ?_format=hal_json to the same URL.

#49: so with the exact same request, you get different behavior in Postman vs your browser? That sounds impossible. Are you sure there isn't a subtle difference somewhere?

Now I am getting a 200 in Postman when I make a GET request to my views rest export, but am getting a 500 when I make a GET request to a jsonapi endpoint. The results reversed since my last comment. At the moment I am getting a good response in my browser for both rest and jsonapi.

Meanwhile, I have been messing with my cors setting, toggling that between false/true. I don't know if that's relevant here, but that is another variable besides the upgrade to 8.5 that could be a factor in these inconsistent results.

I can definitely confirm that I am getting a different result in Postman vs in my browser at times, though I'm not sure if that could be due to caching on some level. It's hard to account for all the possible factors.

ps -- A little later now, and I am now getting 500 responses for all endpoints in Postman. I am using my app to get auth tokens. I log the token to the console after logging into my app. I paste the token into the token field in Postman. Each time I make a change I want to test, I clear Drupal's cache, run cron, then generate a new token in my app and paste into Postman for testing. This was all working consistently before Drupal 8.5. No problems in the browser, though. These endpoints all work in my browser.

I don't know if this will be useful info or if I'm cluttering the thread, but I have noticed that if I use an older token for the Postman request -- say, one I generated a half hour ago -- it sometimes works in Postman, even if a newer token does not. I have noticed this a few times today. The latest token that I print to the console does not work when I paste it into Postman, but the one from a few attempts earlier sometimes does work. (My calls to /oauth/token always work.)

The problem I'm describing here could be outside the scope of this bug, but it wasn't happening before 8.5. Before 8.5, I could make POST requests and GET requests from my app to Drupal, and from Postman to Drupal, consistently.

I am building a React Native app with Drupal as the backend.

Weird. I went away from my computer for a few hours, came back, hit the send button on my posts in Postman and now I get 200 responses for everything. The status is 403, as it should be, because the tokens are now expired.

So I wonder what happened while I was away from my computer that made these requests start working? I'm not having any hosting issues today.

UPDATE: Part of my issues stem from uninstalling jsonapi_extras, I think. In any case, I changed too many things in one day to be sure about anything. I have rolled back to 8.4.5, rolled back the database accordingly, and now my app works.

+1

we should not have to add ?_format=hal_json or else ?_format=nnn to the urls for REST Export Views displays if i specify my desired serialization method in the View's configuration...

nor should we have to add ?_format=hal_json to the built in core rest endpoints for GET / POST / PATCH if we are specifying content settings in the http header, that should be sufficient.

having to add '_format' to everything;

a) makes it ugly
b) breaks previous working functionality (pre-8.5) which decoupled apps relied on
c) is confusing and currently lacks documentation examples in the Drupal UI

+ we should be careful to break backwards compat as we move forward, whether the feature was by design or mutated into the design...once it's in and people start building, we cannot / should not just take away...

( ....yes, upgrading to d8.5 broke my Alexa App because i was consuming the clean api url endpoint in there...)

#54: please read what has been said before. This makes no sense:

nor should we have to add ?_format=hal_json to the built in core rest endpoints for GET / POST / PATCH if we are specifying content settings in the http header, that should be sufficient.

See #12 + #38.

#55 i read and i disagree, it makes perfect sense, i am saying exactly what #13 is saying.

Also re: #38 flawed is a perception..this is how we released it and how it should of stayed.

It is perfectly valid to default to json as everyone is saying & requesting...#cleanAPIurls4theWin

Just to note, that if one uses exposed filters or something similar, you need to append "&_format=csv" or add "_format=csv&" just after the first "?" already in the URL.

For now I've managed to overcome the problem by kinda hack.

Define the service:

services:
  MODULENAME.request_format_route_filter:
    class: Drupal\MODULENAME\Routing\RequestFormatRouteFilter
    tags:
      - name: route_filter
        # Execute before core filter (see parent class).
        priority: 1

<?php

namespace Drupal\MODULENAME\Routing;

use Drupal\Core\Routing\RequestFormatRouteFilter as RequestFormatRouteFilterBase;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\RouteCollection;

/**
 * Fix the regression, introduced in 8.5.x.
 *
 * @todo Remove the implementation once the issue gets resolved.
 * @link https://www.drupal.org/project/drupal/issues/2937942
 */
class RequestFormatRouteFilter extends RequestFormatRouteFilterBase {

  /**
   * {@inheritdoc}
   */
  public function filter(RouteCollection $collection, Request $request) {
    $default_format = static::getDefaultFormat($collection);
    $request_format = $request->getRequestFormat($default_format);

    // The request format will be "html" by default if "?_format=csv" is
    // not passed explicitly. Imagine we have a route that has the only
    // single format and it's not the "html"? The request will be failed.
    // Let's try to overcome.
    if ($default_format !== $request_format) {
      foreach ($collection as $name => $route) {
        // A route is accessible only in a certain format(s).
        if ($route->hasRequirement('_format')) {
          // Get the list of format a route supports.
          $supported_formats = array_filter(explode('|', $route->getRequirement('_format')));

          // The list of supported formats for a route has only its default
          // value so we can forcibly set this format to the request.
          if ([$default_format] === $supported_formats) {
            $request->setRequestFormat($default_format);
          }
          else {
            // Assume the path is something like "list/export.csv" - the
            // format is specified as kind of extension.
            $path = explode('.', $route->getPath());

            if (count($path) > 1) {
              $route_format = end($path);

              if (in_array($route_format, $supported_formats, TRUE)) {
                $request->setRequestFormat($route_format);
              }
            }
          }
        }
      }
    }

    return $collection;
  }

}

This allows to not specify the ?_format=csv if it's selected in view's display configuration or to extract an "extension" from a path, like give/me/file/in.csv.

I wonder if we can take the route filter, add a BC flag to it, and maybe a post_update hook to set it, and use that as the long-term solution?

#58 worked like a charm. I didn't understand how the extension should work though, I tried with news.json but I get client error unless I choose json as accepted format in the view. But I don't need it in my case.

#58 effectively achieves the same as #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't, and will become obsolete once #2854543 lands, right, @BR0kEN?

Just applied #67 from #2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't and yeah, it does the same.

#62: thanks for confirming!

I am having the same problem as it is said in comment #11, but I am having exposed filters.. I have tried adding the _format=json, but still encounter the error " Client error". Does anyone know how should I solve this?

#2854543: NegotiationMiddleware calls $request->setRequestFormat('html') when there is no _format request parameter, but shouldn't landed! That also fixes this problem :)