Problem/Motivation
In Change Notice https://www.drupal.org/node/2199185 based on #2019123: Use the same canonical URI paths as for HTML routes we missed the change to POST request.
Some route like rest.entity.node.POST
have paths prepended by /entity
like /entity/node
which is not in line with the other methods GET, PATCH, DELETE.
Furthermore, for entity types that have multiple words in their name, it's often even worse: GET /taxonomy/term/42
, PATCH /taxonomy/term/42
or DELETE /taxonomy/term/42
, but POST /entity/taxonomy_term
(note the prefix and the underscore).
Proposed resolution
- POSTing entities via REST should happen at
/{entity_type}
(e.g./node
,/comment
…), not at/entity/{entity_type}
(e.g./entity/node
,/entity/comment
…). - But we shouldn't hard code this, we should use the same opt-in mechanism that we already use for
canonical
: entity types that specify acanonical
link template will be available at that same path via REST. We should do the same for<code>https://www.drupal.org/link-relations/create
. - Provide BC through a path processor.
In other words: just like the REST module in Drupal 8.0.x, 8.1.x and 8.2.x has already been using the canonical
link template of an entity type if it is specified in favor of its default (/entity/{entity_type}/{entity}
), starting in Drupal 8.3.x, it will do exactly the same for the create
link template (for the https://www.drupal.org/link-relations/create
link relation type): it will use that instead of the default (/entity/{entity_type}
).
Remaining tasks
Blocked on #2730497: REST Views override existing REST routesFully functional patch.Small test to prove that the BC layer works.
User interface changes
None.
API changes
None.
Data model changes
None.
Comment | File | Size | Author |
---|---|---|---|
#239 | 2293697-239.patch | 24.07 KB | Wim Leers |
#233 | 2293697-233.patch | 24.04 KB | Wim Leers |
#230 | 2293697-230.patch | 24.09 KB | dawehner |
#223 | 2293697-223.patch | 23.9 KB | jofitz |
#223 | interdiff-221-223.txt | 864 bytes | jofitz |
Comments
Comment #1
clemens.tolboomAdded SQL + result
Comment #2
clemens.tolboomAdded rest.node.POST
Comment #3
chx CreditAttribution: chx commentedHere is some supporting evidence:
Indeed it looks this is correct. However the patch missed two:
Comment #4
clemens.tolboom@chx thanks
Question now is what endpoints should not start with /entity?
In #2292707-10: GET on entity/taxonomy_vocabulary/{id} is not working I enabled all possible endpoint then checked for the non /entity endpoints.
Comment #5
clemens.tolboomRunning on my `enable all` install
gives 178 result (divide by ~5 GET (json, xml), PATCH, POST, DELETE) ~40 'entities'.
Comment #6
chx CreditAttribution: chx commentedI do not know the generic answer to that but at least user DELETE seems wrong since #4 found a rest route for user/{user} . I would say let's fix that one at least. I absolutely have no idea how or where rest routing is generated as I generally try to stay away from routing. (And in fact, I am out of this issue too.)
Comment #7
clemens.tolboomDigging around I found #2019123: Use the same canonical URI paths as for HTML routes which suggests Rest ResourceBase is in err.
So a completely different patch without test fixes yet.
SELECT * FROM router WHERE name LIKE "rest.entity.%";
gives a consistent pattern for node, user, comment and file. File is still on /entity/file which makes me puzzled again as I consider that a 'normal' exposed entity.Comment #9
Berdirfile has now routes in core, so it falls back to the default, that makes sense.
Only question is if the default should not include the leading entity/ now. file is an interesting example. because contrib adds file routes/links, so that would change the REST routes...
Comment #10
clemens.tolboomI don't get the failures of #7
Comment #11
R.Muilwijk CreditAttribution: R.Muilwijk commented#7 Fails because it only patches the setPattern and did not run your patch in https://www.drupal.org/files/issues/rest-post-canonical.patch. Provide a single patch?
Comment #12
clemens.tolboomPatch needed a combine (thanks @R.Muilwijk), reroll and additions from #3.
Comment #13
clemens.tolboomComment #15
clemens.tolboomRechecking this patch I ran into:
#2406439: Cleanup EntityDerivative and RouteBuilderInterface
#2281645: Make entity annotations use link templates instead of route names
which targets a higher goal.
Checking the current db routes on node, user, comment, file I find the odds
GET: /entity/file/{file}
PATCH: /entity/file/{file}
POST: ALL on /entity/{entity_type} which is better. This is caused by the default
DELETE: /entity/file/{file}
I think we can just reroll this patch and check for the odd /entity/file in the issue queue.
Comment #16
vedpareek CreditAttribution: vedpareek commentedRerolled
Comment #17
vedpareek CreditAttribution: vedpareek commentedRerolled
Comment #18
dawehnerTagging with API change, as the URLs are now some form of API.
Comment #20
vedpareek CreditAttribution: vedpareek commentedRerolled
Comment #22
dcam CreditAttribution: dcam commented#20 applies to HEAD. Removing the "Needs reroll" tag.
Comment #23
hchonovThis one should pass now.
Comment #24
clemens.tolboomI do miss this from latest patch(es). iirc this is now unnecessary. See patch from #12
Comment #25
hchonov@clemens.tolboom:
If we remove it, then calling
Url::fromRoute('rest.entity.' . $this->testEntityType . '.POST')->setAbsolute()->toString()
throws an exception :
exception 'Symfony\Component\Routing\Exception\MissingMandatoryParametersException' with message 'Some mandatory parameters are missing ("entity_test") to generate a URL for route "rest.entity.entity_test.POST".' in Drupal\Core\Routing\UrlGenerator->doGenerate() (line 174
Comment #26
clemens.tolboom@hchonov ic ... thanks for testing :-)
Did you do some manual testing too. My angular app is broken :(
Comment #27
hchonov@clemens.tolboom
yes, I did manual testing as well. So I guess the patch looks good now?
Comment #28
dawehnerIn case we do that, we should hae some form of change record, given that this is kinda some API change isn't it? I think the REST interface is considered as real API
Comment #29
klausiThis would be a pretty heavy API change at this point. We left out the creation POST route on purpose, because it does not align with any other route that serves HTML (the node creation form for example is on a completely different path).
So we need a pretty strong reason in the issue summary why we should change this at this point, can anyone do that? For backwards compatibility we could register 2 routes that do the same thing, one on /node and the legacy one on /entity/node.
I would close this as won't fix, let's ask Crell, the second rest.module maintainer what he thinks.
Comment #30
Wim LeersComment #31
dawehnerNow a couple of months later there is even less reasons for that. I guess we just have to say no and maybe think about it in D9? Providing some alias as well seems a bit weird.
Comment #32
Wim LeersThat's almost exactly what I first wanted to post. :)
Then I deleted what I wrote. Because this *is* very weird, a Drupal WTF. It's very counterintuitive. That's why I think an alias would actually be a good thing.
Comment #33
dawehnerAgreed.
Comment #34
dawehnerI would even argue at that time the sane option should be the default, and the BC layer should be the alias.
Comment #35
Wim LeersRebased #23, it no longer applied.
Comment #37
Wim LeersA regular URL alias won't work AFAIK. I think we can solve this using an inbound path processor? Something like this? Ideally, we'd even only conditionally enable this, i.e. only for sites that were originally installed using 8.0.x.
Comment #39
dawehnerYeah, we could have a container parameter to turn off this behaviour. This sounds like a great idea
Comment #40
Wim LeersPatch is against 8.1. This should happen in 8.1 anyway, otherwise it's a BC break.
Comment #41
Wim LeersRe-tested, but no idea if it'll now pick 8.1. May need to reupload.
Also, will need a CR.
Comment #44
Wim LeersYet another person who ran into this problem and found it very confusing: #2667186: Can't make a post with cookie authentication.
Comment #45
Wim LeersClearer title.
Comment #46
dawehnerLet's see whether this works.
Comment #47
Wim LeersNice! This looks great :)
Yeah, I was thinking the same. We definitely need test coverage for that.
Path processors run in a certain order, so we need to make sure that this runs after the language one.
We have a leading slash in the other places, so let's do that here too?
Comment #48
dawehnerThis is the interdiff
Comment #49
Wim LeersLOL, apparently I rolled this patch previously. I'll fix my own remarks then :P
Comment #50
Wim LeersActually, we don't need to do anything. Precisely since the language prefix needs to processed before everything else, the priority of the language path processor is very high. So it's done first. Precisely to allow contrib modules to not have to think about this. So, we can keep the default priority, by not specifying any priority, which gets us priority 0.
Fixed all my feedback in #47.
Comment #51
dawehnerLooks pretty sane now.
One issue I found is that both
\Drupal\views\Plugin\views\display\PathPluginBase::alterRoutes
and\Drupal\page_manager\Routing\PageManagerRoutes::alterRoutes
gives a damn about HTTP methods, so they potentially override the POST route now.I think this is simply never the case at the moment, because the default canonical routes are registered first.
Comment #54
dawehnerNote: the patch in #2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON fixes the failures.
Comment #55
Wim LeersOh, great! Postponing this on that issue (#2449143: REST views specify HTML as a possible request format, so if there is a "regular" HTML view on the same path, it will serve JSON) then.
Comment #56
dawehnerComment #57
dawehnerAdded a new issue: #2730497: REST Views override existing REST routes
Comment #58
Wim LeersSuperb! I reviewed (and RTBC'd) that one :)
Comment #59
dawehnerThis should pass now.
Comment #61
Wim LeersRandom migrate fail:
Retesting.
Comment #62
dawehnerI just checked
page_manager
and it indeed had a similar problem than node module. Given its an existing bug in contrib I'm not sure we should care about it: #2736385: Page Manager routing is overly complex and brittle: breaks REST, active trail, fallback pagesComment #63
Wim Leers#62: I think you wanted to post this on #2730497: REST Views override existing REST routes, not this issue?
Comment #64
dawehner@Wim Leers
No, I'm actively reporting it here. There could be a chance that some people used page manager for /node, so when they do and we commit this patch, their sites would be broken.
Comment #65
Wim LeersAh, ok, thanks for the clarification!
Comment #66
Wim LeersI rerolled this patch thrice:
Clearly, I only played a tiny role in writing the actual patch. So I think it's fine for me to RTBC this issue.
I think the patch is ready. Still to be done: issue summary update & change record. I did both of those things (CR: https://www.drupal.org/node/2737401). I also updated the existing documentation (https://www.drupal.org/node/2098511/revisions/view/9709507/9730869). Let's get this in!
Except… there is no test coverage yet to prove that
/entity/{entity_type}
still works. So we need a small test proving that still works, then this is ready.Comment #67
dawehnerWell, I still think we should consider to fix page_manager first. Just imagine
/user
etc. to be broken on actual sites. There is certainly a non null probability for it.Comment #68
dawehnerGood point, let's add the tests
Comment #69
dawehnerFor me this seems to be ready to role, anyone wants to object ;)
Comment #70
Wim LeersIt looks perfect:
Tests new URL and BC for POSTing entities of type
entity_test
.Tests new URL and BC for POSTing entities of type
node
.Tests new URL and BC for POSTing entities of type
comment
.Tests new URL and BC for POSTing entities of type
user
.Yay, let's finally get this in :)
Comment #72
Wim LeersIt looks like this needs a reroll now that #2724823: EntityResource: read-only (GET) support for configuration entities has landed:
Those are specifically referring to the config entities that can now be POSTed. The path processor needs to only strip the leading
/entity
if the entity type has a canonical path.Comment #73
dawehnerComment #74
Wim LeersWonderful, thanks!
Comment #75
alexpottWhat does this mean for
From #2724823: EntityResource: read-only (GET) support for configuration entities?
Comment #76
Wim LeersAs the patch shows:
/<config entity type>/<entity ID>
.Comment #77
alexpott@Wim Leers but doesn't that conflict with
Comment #78
Wim LeersLet me take a deeper look at this tomorrow, so I can formulate a proper answer.
Comment #79
dawehnerI would indeed argue that we have an out of scope change here. The default canonical aka. fallback path for entities is
/entity/{$entity_type_id}
, so config entities are currently exposed under /entity/config_test/{config_test} for example.We introduce a BC break in a way, or at least a change which is not part of the problem in most cases.
One could say
\Drupal\rest\Tests\ReadTest::getReadUrl
is a sign of a too abstracted test, as its not entirely obvious what is going on.Comment #81
dawehnerThis was just a GET URL involved
Comment #82
Wim LeersThe patch in #73 and all prior patches changed the default canonical URI path in
EntityResource
. That URI path was used if the entity type didn't specify a specific canonical URI path. Technically it is indeed out of scope to do that, and from that POV #79 and #81 are correct.However… in doing this, AFAICT this will start to break in #2300677: JSON:API POST/PATCH support for fully validatable config entities: because
PathProcessorEntityResourceBC
always strips the/entity
prefix for POST requests, you won't be able toHTTP POST /entity/taxonomy_vocabulary
, for example: the inbound path processor that this patch will always rewrite that toHTTP POST /taxonomy_vocabulary
, which points to a non-existing path.I'd argue that it is in the spirit of this patch to change all entity REST URIs to use
/{entity_type}/{entity_id}
, and therefore to change the default/fallback canonical entity resource URI paths. In which case the previous patch, i.e. the one in #73, is the appropriate one to commit.In doing so, we'd have:
Comment #83
dawehnerTechnically it was decided in #2019123: Use the same canonical URI paths as for HTML routes to do so, so sure, let's rename the issue title then.
Comment #84
dawehnerWe agreed on changing it
Comment #85
Wim LeersComment #87
Wim LeersUpdatePathWithBrokenRoutingFilledTest
failed. Let's see if it was a random fail or whether HEAD is broken.Comment #88
dawehnerYeah at that point in time basically every update test is failing randomly.
Comment #89
alexpottComment #90
alexpottNow we've changed this issue to also change the default for canonical - we'd need to also fix GETs but that seems super risky.
Comment #91
dawehnerWhat @alexpott and myself talked about is the following:
/entity
prefix but also really just change/entity/\d+
All in all this would result in a more minimal BC break.
Comment #92
Wim Leers#90: Damn, good point :/
#91:
But we want custom/contrib entity types to also post to
/<entity type>
, not to/entity/<entity type>
. We want that to still be consistent, so we should still do it for all entity types that do specify a canonical URL.So, I agree with everything in #91, except for one thing:
— we shouldn't do that, we should do it for all entity types that do specify a canonical URL in my opinion.Comment #93
dawehnerThere is no reason that they cannot do that. They can add the link template. Risking BC when we don't have to, is IMHO an antipattern.
Comment #94
Wim LeersAhh! Now I understand.
So they'd each have to do something like this:
(Look at that very last line in the annotation.)
That makes sense.
+1
Comment #95
dawehner@Wim Leers Exactly, do you think less risk is better here?
Comment #96
Wim LeersYes, +1.
Comment #97
Wim LeersActually, @EclipseGc made an excellent remark yesterday in personal chat. Why are we not simply using the
collection
link relation? Creating a new entity of a certain type is the same as POSTing something to a collection. That makes perfect sense. It'd also mean we can deprecate the sillyhttps://www.drupal.org/link-relations/create
link relationship. https://en.wikipedia.org/wiki/Representational_state_transfer#Relationsh... also explicitly mentions this.So I did some digging. Apparently #2019123: Use the same canonical URI paths as for HTML routes introduced the
https://www.drupal.org/link-relations/create
link relationship. Despite that issue being done by resident REST experts @klausi and @Crell, there were zero mentions of (well, there were mentions of that particular word in the context of a route collection, but the link relationship was not mentioned once).Now, the funny thing is that we have 14 entity types in core that specify a
collection
relationship. Butnode
isn't one of them. Nor iscomment
.So, it looks like in order to do what is really right, we'll first have to fix the link relations on many entity types.
Thoughts?
Comment #98
Crell CreditAttribution: Crell at Platform.sh commentedThe "every resource type has one definitive collection and you POST to that to create something" is not, AFAIK, part of REST style as defined by Fielding. It's a common pattern seen in demo applications and other cases where "All the Thingies" is the only reasonable collection of Thingies, but that's not at all universal. In fact, I have built systems for clients where we had /thingie/1/stuff/4 as a "stuff" resource, and the "collection" was therefore /thingie/1/stuff, and that's where you'd POST. There was also /thingie/8/stuff if you wanted to add things to THAT collection. (In this case, it was a collection of stuff that belonged to that particular thingie.)
Thus, a blanket statement that all entities are primarily a part of the "all the thingies" collection, and that is their primary existence is, I would argue, incorrect. That's why we didn't do that. Arguably we could make the assumption that all comments are first and foremost part of the node/X/comment collection (or whatever other base entity they are on, since in D8 comments are not node-specific), but I don't think any such assumption could be made for nodes.
Wim, you don't mention which those 14 entities are that have a collection relationship, but I'll venture a guess that many/most are config entities, or other entities with very low cardinality and relationship complexity. In those cases the simplified default assumption probably dos hold. But that's not the universal REST Way, just a common degenerate case, and so not an assumption we should bake into the whole system.
That is, "Creating a new entity of a certain type is the same as POSTing something to a collection" is only true sometimes. It is not a universal.
Comment #99
tstoecklerThat might be complete bogus in a REST world, but we currently use
collection
routes for entity list builders. The fact that not very many entity types actually provide link templates for thecollection
link templates is owed to the fact that we are not capable of auto-generating a list-builder-route yet (!). Once we can do that, we will add collection link templates, but they will point to paths like/admin/content
and/admin/config/user-interface/shortcut
. While - again - that might make the stomache of someone coming from a REST POV twist, I don't think that's something we can change at this point.Might also be that this is completely beside the issue and I'm just rambling, in which case sorry and carry on, but the mention of the
collection
relation above made me stutter for a moment.Comment #100
Wim Leers#98: thanks a lot for that comment, that was very interesting!
#99: indeed, many of the collection routes are admin-UI-specific URLs, which makes them… less than ideal for REST purposes. However, I'm not sure I understand your conclusion: are you saying you're agreeing with #97, or that you were frustrated by the explanation in #98, or …?
Comment #101
EclipseGc CreditAttribution: EclipseGc at Acquia commented@Crell
So, while I don't have a problem with your basic argument, your example does nothing to convince me. That's still defining the collection of the stuff entity as /thingie/{thingie}/stuff. It is then perfectly logical that posting to that will make a stuff entity related to the thingie in question. This is EXACTLY how I'd want comments (and taxonomy terms) to behave. Of course none of this has to be forcefully thrust upon all entity types, but as a guiding principle, I think it's a pretty good (and common) one. No?
Eclipse
Comment #102
Crell CreditAttribution: Crell at Platform.sh commentedWell, it's true in many cases, certainly. But to take comments, we support comments on nodes, on users, on commerce products, etc. Technically, we even support multiple comment fields/threads on a single entity. (I think? I've not actually tried and I don't know why you would, but isn't that a side effect of comment-config-as-field?) That means simply defining the comment "collection" route as /node/{node}/comments isn't sufficient, because comments would also make sense at /user/{user}/comments, /node/{node}/internal_comments (alternate field), /commerce/product/{product}/comments, etc.
That is, simply standardizing all entities to a "collection" route, in their annotation, of /{entity_type} is not going to work. That would be true in some cases, perhaps a majority, but would break badly in many. We would need to at least allow for collection to be defined contextually at runtime instead.
Comment #103
dawehnerWhile I think the discussion around the collection link relation is worth to do, I think this issue is about a bug in REST itself and its inconsistency. Fixing that is a small iteration step, vs. what we discuss which is a big step compared to it. Its totally worth to do, but it should be its own thing.
Regarding the collections, I think those collections should be created, but in order to design properly, one needs domain knowledge, rest module cannot have. REST module made the tradeoff early that its primarily a generic export all the things mechanism, rather something well tuned like a RESTAPI of github for example.
Comment #104
tstoecklerI only brought the collections thing up because - per this issue's title - I thought the intent was to make the
"collection"
link in the entity annotation point to e.g."/node"
. That would be a pretty big conceptual (and API) change, which I tried to explain above. If that is in fact not the case, then this can be ignored. I'm not sure reading the latest comments what the intent of the issue is, TBH.Comment #105
Wim LeersSo let's get this issue back on track! What still needs to happen for this issue to go back to RTBC?
collection
link relationship's URI.collection
, then it'd still be inappropriate to do so here, because it requires many more changes.So, basically, #97 through #104 was a big, distracting that would lead to a different direction for the patch. And it'd be very valuable, if it turns out to be possible.
But this issue is just about fixing the WTF that is
GET /node/42
,PATCH /node/42
,DELETE /node/42
versusPOST /entity/node
. That should just bePOST /node
(every person who's used REST so far has found that unintuitive/strange). That's all this is about. And that's something we can do without the even bigger changes proposed in #97–#104.Therefore let us go back to #90 + #91–#96, where we were so close to a patch that resolves this big REST Developer Experience WTF.
This implements everything discussed in #90–#97. CR updated as well: https://www.drupal.org/node/2737401/revisions/view/9730871/9919603.
I think this idea in #91 would be a splendid minor performance improvement for a follow-up:
But for now, I've gone with the simplest possible approach that works, so that this can still go in before feature freeze next week.
Comment #106
Wim LeersAlready updated the patch, CR and IS to reflect the new direction. Now also updating the title.
This is so much better. Thanks, @alexpott and @dawehner!
Comment #108
dawehnerCan we ensure to just match on /entity/{entity_type} and nothing more?
IMHO this path is a bit weird. Don't we want
/taxonomy/term
instead?Comment #109
Wim LeersThat's what the code after that does?Hm, I see it's not strict enough. Let me fix that.GET|PATCH|DELETE /taxonomy_term/42
already. It'd be much more confusing to haveGET /taxonomy_term/42
, butPOST /taxonomy/term
Comment #110
dawehnerYeah this was kinda the point alexpott and I talked about in #90 and #91
Well, GET is certainly on
"/taxonomy/term/{taxonomy_term}"
but yeah we some some inconsistency there. nevermind.Comment #111
dawehnerSo yeah please ignore me little rambling there :)
Comment #112
dawehnerThese test failures are kind of interesting to be honest.
They are basically really related with #2113345: Define a mechanism for custom link relationships as in we have, names for our link relations, and external URLS where they are defined.
In case we would have #2113345: Define a mechanism for custom link relationships in, there would be also a way to use "create" in the link templates, but then still expose the right link relation to the public.
Comment #113
tedbowIs possible to fix without #2113345: Define a mechanism for custom link relationships?
Classes like \Drupal\node\Controller\NodeViewController assume call \Drupal\Core\Entity\Entity::url which will call \Drupal\Core\Entity\Entity::toUrl
Entity::toUrl assumes it can make a route name with $rel.
That is why you get the error in #112.
Also on a related note should we actually be updating Entity class files like Node.php?
Isn't
"https://www.drupal.org/link-relations/create" = "/node",
Only valid if the REST module enabled or maybe I am missing something?
Should we be using hook_entity_type_alter to add this links to the entity definition? This would probably fix a lot of the test because they won't turn on the REST module but it won't fix the underlying problem.
Comment #114
dawehnerYeah let's be honest, this gives us the abstractions we need
Comment #115
dawehnerComment #118
Wim Leers#2113345: Define a mechanism for custom link relationships is FINALLY in, so now this can continue! We should still try to get this in 8.3.
I will reroll this first thing in the morning unless somebody beats me to it.
Comment #119
Wim LeersFirst, a straight reroll.
#2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method completely changed the test coverage, and thus those portions no longer apply:
core/modules/rest/src/Tests/CreateTest.php
andcore/modules/rest/src/Tests/ReadTest.php
no longer exist. This means we must still do the work to restore the removed test coverage! (I've attached an interdiff that shows which parts of the patch I removed. It's not actually an interdiff, it's the portion of #105 that I've literally cut away, because it cannot apply at all anymore.)I expect this to have a lot of fails.
Comment #120
tstoecklerCouldn't we return
$entity_type->getLinkTemplate(...)
instead of always using the entity type ID? I.e. for taxonomy terms maybe we want to use/taxonomy/term
?Edit: I see that that has been discussed in #108 and following, but it still seems that in general this should be configurable and we should use the provided link template?
Comment #122
xjmComment #123
Wim Leers#120: I think you're right :) That definitely needs to be done before this can be RTBC. But first, bigger worries: getting this patch to green. Details about the trickiness to follow…
Comment #124
Wim Leers#2113345: Define a mechanism for custom link relationships added link relations, but did so with
https://drupal.org/link-relations/…
URIs instead ofhttps://www.drupal.org/link-relations/…
which was used in this patch. Rather than modifyingcore.link_relation_types.yml
, I think it's better to modify this patch.Not only that, but it's probably better/more accurate to use the key specified in
core.link_type_relations.yml
: i.e. usecreate
, nothttps://drupal.org/link-relations/create
. Just like entity type annotations.This made me realize: then how can you discern registered link relation types versus extension link relation types?! We should use identify them by their name and URI respectively. That'd match the spec. So we should continue to do it like this patch is already doing: using URIs!
However… that won't work because Drupal 8 shipped with violations of this principle already, and we can't change that because it'd break BC. We should have done #2113345: Define a mechanism for custom link relationships before D8 shipped, then we wouldn't have made this mistake. For example for
Node
in HEAD:Of those, only
canonical
,edit-form
andversion-history
are registered link relation types (i.e. registered with and accepted by the IANA). So that meansdelete-form
andrevision
are already extension link relation types (i.e. Drupal-specific).(This also means that if the IANA accepts new link relation types that happen to use the same names as the ones Drupal used, but with a different meaning, that we will be forced to break BC. For D9, we should change the entity type
links
annotation to use URIs as keys for extension link relation types, and we should also makelink_relation_types.yml
use URIs as keys for those.)So the existing entity type annotations (and associated logic) requires us to not use URIs. But OTOH the REST module's logic requires us to use URIs. There's no way to reconcile these. We must break one of the two!
I think there is one of those that is clearly more important (and would cause a bigger disruption): the entity type annotation. Very few
@RestResource
plugins have ever figured out how to define a custom deriver. Plus we can add a BC layer in\Drupal\rest\Plugin\ResourceBase::routes()
, which means we can actually guarantee zero disruption. Whew!Ironically, this (the
Node
entity type annotation cited above) was introduced by #1970360: Entities should define URI templates and standard links, which itself was all about using link relation types + associated link templates for REST… that enabled #2019123: Use the same canonical URI paths as for HTML routes, which is what made the REST module actually use entity type-defined link templates. It already pointed to #2113345: Define a mechanism for custom link relationships as the place to settle on a final solution, but said it should not be blocked on that issue. That was a huge mistake. That's why we have this mess. That's why this issue is still not solved, more than 3.5 years after #2019123 was opened.In fact, @linclark predicted that the logic in the REST module would have to change depending on how #2113345: Define a mechanism for custom link relationships would be solved. See her comment at #2019123-43: Use the same canonical URI paths as for HTML routes. Also, it's that same issue (#2019123) that made bold claims about how much better REST URLs became (see the change record), but it also forgot to mention the fact that it only did so for GET/PATCH/DELETE, it did not make the POST case better. Which is what we're still trying to solve here, almost 3 years after that was committed.
Therefore, we have but one choice:
Change hunks like these to use
"create" = "…"
instead. Did that.This causes some disruption, but we can mitigate it completely by doing
— did that too.Comment #126
Wim LeersAs you can tell, the test results of #124 are littered with PHP errors like this:
This is because
\Drupal\node\Controller\NodeViewController::view()
(soon hopefullyEntityViewController
— see #2282029: Automate link relationship headers for all entity types, not just nodes) and\Drupal\rest\Plugin\rest\resource\EntityResource::addLinkHeaders()
call$node->toUrl('create')
(which is correct btw). That in turn tries to generate the URL for theentity.node.create
route. Which… fails. Because it doesn't exist.It doesn't exist, because despite
create
being a link relation type specifically for REST,\Drupal\rest\Plugin\ResourceBase::routes()
always wants to create routes with names likerest.entity.node.POST
.So this breaks both REST responses and HTML responses for
/node/NID
, with the fatal error mentioned above.We could make all code that iterates over link templates should support the fact that a route may not be defined. Then the
create
link would never be listed, but at least it wouldn't cause fatal errors.Let's at least do that for now, that should reduce the number of failures.
Comment #127
Wim LeersSo #126 was not a full solution. We are not yet able to generate
create
link headers, because the necessary route does not exist.Symfony does not support the concept of "route aliases" or "route pointers". We'd have to have
rest.entity.node.POST
andentity.node.create
. And they'd have to have identical route definitions. That's the only way to make two route names point to the same route.We can't drop
rest.entity.node.POST
either, doing so might break BC. Nor can we dropentity.node.create
, because that would break both\Drupal\Core\Entity\Entity::toUrl()
and the entity type link templates API.Letting
\Drupal\rest\Plugin\rest\resource\EntityResource::routes()
generate aentity.node.create
route with such a duplicate definition doesn't work, because\Drupal\rest\Routing\ResourceRoutes::alterRoutes()
prefixes all routes with 'rest'.I can make that work by also modifying
ResourceRoutes::alterRoutes()
to no longer set therest.
prefix on route names, and just setting that prefix inEntityResource
andDBLogResource
. But this would break BC for other REST resources — their route names would change.That almost leaves me with only one choice: a
\Drupal\Core\Routing\RoutingEvents::DYNAMIC
event subscriber that iterates over allRestResourceConfig
entities, looks which ones are for theEntityResource
@RestResource
plugin, and adds aentity.<entity type>.create
route with the same definition as therest.entity.<entity type>.POST
.But this feels quite strange IMO.
Modeled after
\Drupal\Core\EventSubscriber\EntityRouteProviderSubscriber
.And UGH… in the process I noticed that
\Drupal\rest\Routing\ResourceRoutes
is usingRoutingEvents::ALTER
, but it really should have been usingRoutingEvents::DYNAMIC
! To be fair, that was added in #2158571: Routes added in RouteSubscribers cannot be altered, long after this class was created. But irony oh irony, this very class was mentioned in #2158571 many times as an example of what should useDYNAMIC
, but then it didn't do that. So, fixing that too.Comment #128
Wim LeersNow addressing #120:
Fixed both of those things.
Comment #129
Wim LeersAdding back test coverage that was lost in #119. Thanks to the overhauled test coverage, adding test coverage has just become trivial.
Comment #130
Wim LeersRound of self-review.
This is a config entity. We shouldn't add this link template for config entities yet.
(Config entities currently only support
GET
.)s/for/that use/
The comment doesn't match the logic. And actually, the logic is very wrong too.
My changes in #128 were completely wrong. It's been a long day :(
This means that e.g. for taxonomy terms, this will fail.
We shouldn't change whitespace.
This needs a better comment.
Unnecessary whitespace.
\Drupal\rest\Plugin\rest\resource\EntityResource
still has:That should become
create
.PathProcessorEntityResourceBC
be renamed toPathProcessorEntityResourcePostRouteBC
?Addressed everything except the last point.
Comment #132
Wim LeersCR updated.
IS updated.
Comment #137
Wim LeersSigh, complete DrupalCI failure. We'll have to wait for DrupalCI to work again before we can retest.
#127 through #130 will need to be retested.
Comment #138
tedbowIs it preferable here to catch an exception rather than check the route exists?
something like:
Probably not but thought I would mention it.
If a particular entity resource is configured not to enabled POST $rest_post_route will be null and then a fatal error for adding the collection.
I hit this b/c the site I was debugging had 'file' entity type setup this way.
Fixed in attached patch.
Comment #139
Wim LeersFor the problems I found with
@EntityType
annotations'links
portion, I opened #2848945: EntityType link templates: using link relation type names as keys is a problem.Comment #140
Wim Leers#138.2: HAH! Yes, thanks :)
#138.1: The current approach works for any URL, yours requires more logic, and ends up doing the same thing. So not a fan of the approach you mentioned at first sight, but also not opposed.
Comment #141
Wim LeersComment #142
tedbow@Wim Leers for #138.2 that sounds like it is fine already the way it is.
Comment #144
dawehnerAnother possible exception could be a messing parameter in the generator URL, like when the relation type needs more than one parameter.
\Symfony\Component\Routing\Exception\MissingMandatoryParametersException
would be thrown in that case.+1
Comment #145
Wim Leers#144.1:
Right, currently there are nocreate
link templates that use parameters, but that's indeed possible. Fixing that…Although hm, no, that's not possible. Look at
\Drupal\Core\Entity\Entity::toUrl()
. It automatically retrieves the necessary route parameters using\Drupal\Core\Entity\Entity::urlRouteParameters()
. That already works for all other link relation types. So it will work fine for thecreate
link relation type too. So the code is fine as-is.Comment #146
Wim LeersApparently it's
taxonomy_page_attachments_alter()
that's adding link relation headers for the canonical Term route! We'll need to update that logic just like we updatedNodeViewController
's.That will fix some failures.
Comment #148
Wim LeersThere are a bunch of these failures:
The root cause is simple: that service needs the request object (hence that error), but it's not tagged correctly.
Comment #150
Wim Leers100% of the remaining failures is in
User
entity type REST tests.Why? Because thanks to #2293697: EntityResource POST routes all use the confusing default: use entity types' https://www.drupal.org/link-relations/create link template if available, we have comprehensive test coverage, including of edge cases. The failing assertions are asserting that we're getting an error response when POSTing to
/user
. But with this patch applied, we're not getting a 4xx response, we're getting a 200! Why? Because/user
matches theuser.page
route:… which requires the user to be logged in. If the user is not logged in, one is redirected automatically to
/user/login
. So, Guzzle simply follows this redirect and then repeats the same POST request, but this time to/user/login
.This then matches three routes:
user.login
(/user/login
), which has no_format
or_content_type_format
requirementsuser.login.http
(/user/login
), which has only a_format
requirementuser.entity.canonical
(/user/{user}
), which has no_format
or_content_type_format
requirementsThe second one is filtered away by
\Drupal\Core\Routing\MethodFilter
during route matching. That leaves the first and the third. The first one is selected and since there's no actual request body (i.e.$_POST
is empty),UserLoginForm
happily returns the form (since an empty request body does not trigger form submission) and hence it results in a 200 response.Now, it may be easy to blame the form system for sending a 200 response to an empty POST request. But the actual problem lies much earlier: in the redirect. It's the redirect that's causing all these problems. The redirect should never have happened! The redirect should only happen for HTML requests. What identifies HTML requests, is the absence of a
?_format
string AND aContent-Type
request header that is either:GET
request)application/x-www-form-urlencoded
multipart/form-data
In other words: our pre-D8 days are coming to haunt us, the decision to let D7 menu callbacks become
_controller
callbacks (first_content
) that return render arrays without needing to specify any additional requirements, is actually the problem here.Changing the defaults (i.e.
_content_type_format
) set for such routes (see\Drupal\Core\EventSubscriber\RouteMethodSubscriber
) is probably a bad idea; it's likely it'll cause some disruption. Plus,multipart/form-data
is actually also very commonly used for REST, i.e. for non-HTML routes. So we can't assign that anyway.I see only two solutions:
EntityResourceTestBase
uses Guzzle: make it no longer follow redirects. The 200 will become a 301. This is something we should probably do anyway.But it doesn't solve the actual problem: the redirection does not make sense.
to:
This will reject any POST requests for that route, which is fine, because it's a pure convenience route anyway, that always redirects (to either
/user/login
or to/user/{user}
, depending on whether you're logged in or not).The current 200 response (well, the 301 redirect, then 200) will become a 405 response (method not allowed).
But it doesn't solve the problem generically, it solves the problem only for this particular entity type, because it happens to have this particular route.
So I implemented this solution by implementing both parts. Part 1 is not necessary but helps prevent surprises. Part 2 is essential. It does mean that there is some special casing in
EntityResourceBase
forUser
, but it's very very limited.Comment #151
Wim LeersWTF d.o. I pressed "upload", and you submitted it.
Here are then the files that belong to #150.
Comment #152
Wim LeersTests were able to get further in the test scenario, but they got stuck on the final bit when posting users.
Here's the last necessary fix, to prevent those 422 responses (due to
Unprocessable Entity: validation failed. name: The username Dramallama ic28G5Or is already taken
) and get them to be 201 responses again.Comment #153
Wim LeersIn another round of self-review, I can find only one nit:
Unnecessary blank line.
Hopefully this is now getting close to RTBC :)
Comment #156
Wim LeersThe two remaining tests are both Contact-related. Reproduced.
Comment #157
Wim LeersSo, this test is failing in
ContactSidewideTest
:$this->updateContactForm($id, $label = $this->randomMachineName(16), $recipients_str = implode(',', array($recipients[0], $recipients[1])), $reply = $this->randomMachineName(30), FALSE, 'Your message has been sent.', '/user');
. Note that it's setting the redirect URL to/user
. So now it's validating the/user
URL using thePathValidator
. And remember that in #150/#151, we configured that route to only allow theGET
method to be used.The root cause:
PathValidator
reuses the current request. Hence it inherits thePOST
method ifPathValidator
is called in a form's submit handler. And hence\Drupal\Core\Render\Element\PathElement::validateMatchedPath()
now complains about/user
, because of the changes in #150/#151.Specifically that method calls:
\Drupal\Core\Path\PathValidator::getUrlIfValid()
, which calls\Drupal\Core\Path\PathValidator::getUrl()
, which does\Drupal\Core\Path\PathValidator::getPathAttributes()
calls\Drupal\Core\Routing\Router::match()
, which calls\Drupal\Core\Routing\Router::matchRequest()
, which calls\Symfony\Component\Routing\Matcher\UrlMatcher::matchCollection()
, which does NOT receive the request object, and instead retrieves the method like this:i.e. it reads the request method from the global request context, and hence incorrectly concludes that it's matching a collection for a POST request!
We need to make sure that we've updated the global request context object (the
router.request_context
service, actually) to use our fake request object that we use purely for validation purposes.Ideally, we should be able to just push our fake request onto the request stack… but sadly Symfony is
incredibly poorly architectednot taking care of that automatically, leaving us with the request stack and request context getting out of sync… So the solution is to not push onto the request stack, but rather to get the current request from the request stack, set our fake request on the request context, and then restore it onto the request context (because yay, the request context service/object has no way to read all of its information, so we can't read from it to then restore it).Whew. This is a great showcase of why I proposed https://events.drupal.org/baltimore2017/sessions/maintainability-only-co....
Comment #158
dawehnerWe have this RTBC issue: #2822190: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions which fixes exactly that problem.
Comment #159
Wim Leers#158: THANK YOU. I was dreading writing test coverage for this. And it really belongs in a separate issue. Very glad it already exists, and is already RTBC!
Comment #160
Wim LeersComment #162
dawehnerOh yeah fair. When we run into the problem it would be a problem everywhere else?
+1
We need a documentation for these things
I think this solution makes sense. In general I'm wondering whether it might be a problem on way more routes conceptually.
Comment #163
Wim Leers#157 failed with 2 fails, but they're now in
PathValidatorTest
. Because I failed to update that test; it needs additional services to be injected. Rather than fixing that, I'm now postponing this issue on #2822190: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions. Once that's in, #157's interdiff can be reverted and this patch will be green!Comment #164
Wim LeersThis also soft-blocks #2800097: Confusing UI: POST URL differs from GET/PATCH/DELETE URL, but UI doesn't indicate this! in the REST UI contrib module.
Comment #165
Wim LeersSo, let's revert #157's changes, and roll #152 + #2822190-40: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions. Let's see if it passes. (It passes locally.)
Comment #166
Wim LeersComment #167
Wim Leers#2815845: Importing (deploying) REST resource config entities should automatically do the necessary route rebuilding conflicted with this. Rerolling this patch. #152 is still the actual latest patch. #165 is just #152 + #2822190-40: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions.
So, rerolling #152. This should fail again with the same failures as #152: those in
ContactSitewideTest
andContactStorageTest
. No interdiff because rebase, with the only conflict (inrest.services.yml
) resolved.Comment #168
Wim LeersAn alternative could be to not block this on #2822190: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions and just modify
ContactSitewideTest
. This is the line that's causing a failure:Specifically, the
/user
parameter at the end. That's filling out a "contact form" config entity configuration form to set the redirect path to be/user
. This then causes problems because:/user
route to only allow theGET
method to be usedPathValidator
has an important bug, described & being fixed in #2822190ContactSitewideTest
just happens to have chosen this particular URL!What an unfortunate coincidence.
So why did that test use that particular path? Well, turns out that is coincidence too: @naveenvalecha wrote this at #306662-135: Add redirect option to site-wide contact forms:
i.e. for no other reason that that is the default front page, and is guaranteed to be available to redirect to, and there are no other non-admin URLs installed in this particular test. We could install the
system_test
module and just point somewhere else.But why even set this redirect path? Because we're testing the redirect functionality of the Contact module, DUH! Well… not setting any redirect path actually is also okay, because we're explicitly testing the redirect functionality separately anyway — see this further down in
ContactSitewideTest
:Conclusion: we can safely change that
'/user'
toFALSE
, and tests pass. There was no good reason for us to pick that particular path. And in fact, we don't even need that at all, and we still don't lose any test coverage.Is this incredibly silly, to the point of being hilarious? YES.
Comment #170
tedbowI think the this update makes sense because as @Wim Leers said we are testing the redirect functionality elsewhere in this test.
This looks good for testing core entities because obviously client code will already be using /entity/[entity_type] path and we should make sure it is still working.
The only thing I am wondering about is:
Since all core entity types now do have a "create" link template we are no longer testing that an entity type does not have this. Obviously there will be custom and contrib entity types that do not have "create" link.
So we should test that this still works.
Chatted with @Wim Leers about this and he suggested removing
"create" = "/entity_test",
from \Drupal\entity_test\Entity\EntityTest to prove that it still works if "create" is not there.
That sounds like a good idea to me. Also we should probably comment in EntityTest that "create" is intentionally omitted so we don't accidentally add it back thinking it was not on purpose.
Comment #171
Wim LeersGreat find :)
So, removed the
create
link template fromEntityTest
's annotation. Left a comment.This then still passes tests because the test coverage was already written with entity types that don't specify
create
link templates in mind:Comment #172
Wim LeersI hadn't fixed #162.2 yet. Done.
Comment #173
effulgentsia CreditAttribution: effulgentsia at Acquia commented+1, per #168. I'm glad that makes this issue's tests pass.
I don't think we can commit this line to core prior to #2822190: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions being fixed. Even though I just bumped that issue to Critical on its own, committing this change would enlarge that problem, because anywhere that a
'/user'
path is being validated (contact form redirect, menu link form, path alias form, ...) would stop working. That's already a problem for other routes with_method
requirements, but we shouldn't expand the breakage to more routes until we fix that issue.Marking this postponed to reflect this. But if folks here want to reset to Needs review in order to keep reviewing the rest of the patch, that's fine.
Comment #174
effulgentsia CreditAttribution: effulgentsia at Acquia commentedUm, what if we just remove this (and the second hunk of #173) from this patch and punt it to a follow-up that's postponed on #2822190: PathValidator validates based on a RequestContext leaked from the current request, resulting in false negatives during CLI requests and POST submissions? But then the rest of this can proceed without waiting for that?
Comment #175
Wim LeersThat is correct! Let's do that. Great alternative. :)
Comment #176
effulgentsia CreditAttribution: effulgentsia at Acquia commentedNW per #174/#175.
Comment #177
Wim Leers"create" = "/user"
and make tests pass.Comment #178
tedbow#177 looks good. Just #172 with adding "create" link template for user entity and removing necessary test changes. Follow up and change record look good.
RTBC!
Comment #179
alexpott@effulgentsia said he might look at this. I've been staring at this issue and have a couple of thoughts. I'm not sure if they block or commit or not and I'm happy to defer to @effulgentsia's judgement if he decides to commit this.
Ideally this would be in a separate issue as it is not related to the issue being fixed.
This chance is a shame. It means that this is likely to require fixes to contrib that have similar code for.
Comment #180
Wim Leers#179
\Drupal\Core\Entity\Entity::uriRelationships()
fromto
i.e. update its logic to only return URI relationships for which there actually is a URI. That is actually even consistent with its documentation:
It's then still possible to get all link relation types (and link templates) for an entity type, by calling
EntityType::getLinkTemplates()
.Thoughts?
Comment #181
effulgentsia CreditAttribution: effulgentsia at Acquia commentedKind of weird to be defining a link template that doesn't have a route unless you enable other modules. Even weirder is that a non-REST route does exist at this path if you have the Views module and the Frontpage View enabled. And weirdest of all, that route has nothing to do with "creating".
So, if I enable rest, hal, and basic_auth modules, then this code runs for the node entity type, due to
rest/config/optional/rest.resource.entity.node.yml
. But if I then disable one of those modules, e.g., hal, this code doesn't run. This has an interesting side-effect: if I disable the Frontpage View, and enable rest, hal, and basic_auth modules, then when I navigate to the frontpage or '/node', I get the "The website encountered an unexpected error. Please try again later." WSOD. If I then disable HAL module, I stop getting that error. So, setting this issue to NW at a minimum for that.#180.2 sounds appealing, but let's discuss it in a separate issue, since it has BC implications. But even if we do that, this part of the code is still weird, because in the case that there is a
entity.node.create
route, we end up outputting<link rel="create" href="/node?node=1" />
to the HTML view of/node/1
. But that makes no sense, since per #1, with the regular HTML browser, I can't actually create a node at that URL.Comment #182
Wim LeersWhy is it strange that something else may use that same path? It's a key principle of the web that you can do
and
and get completely different responses. The same URL can serve different representations. This is exactly like that. Your example is exactly like that.
If you follow your argument, then the logical conclusion is that we should have used a path prefix for every REST URL.
I don't see what the problem is with
<link rel="https://www.drupal.org/link-relations/create" href="/node" />
being output? Link relation headers are for machines. It's up to machines to interpret/use those headers. Not all of these links are necessarily intended for HTML use only.E.g. the
dns-prefetch
link relation type is entirely unrelated to HTML. So arehosts
,lrdd
and others. These are all official ones, see https://www.iana.org/assignments/link-relations/link-relations.xhtml.Comment #183
Wim LeersSo, #181 was marking NW for something that doesn't need work IMHO.
But #181.3 (regarding #180.2) about
NodeViewController
is something that would need to happen, but apparently @effulgentsia believes this can happen in a follow-up. And apparently the logic for generating link headers inNodeViewController
is broken anyway, because we indeed get<link rel="create" href="/node?node=1" />
, whereas we should be getting<link rel="https://www.drupal.org/link-relations/create" href="/node" />
. So we need to fixNodeViewController
anyway; it's already broken in HEAD.Therefore I'm fine with postponing #180.2/#181.3 to a follow-up.
That then means this is still RTBC IMHO.
Comment #184
alexpottI think #180.2 looks like a good idea and will mean that contrib code doesn't need to change because of this change. I think the BC implications of not making that change are way worse than doing it. If we don't do it, then code that used to not throw an exception now will.
Comment #185
Wim LeersDoes 180.2 block commit or can it be a major follow-up that lands in a few days?
Comment #186
xjmWe should never assume that any followup will ever land. :) We should only put things in followups if we are willing to accept them never landing.
Personally I think a followup sounds like a good idea if needed for any of those points in #180; let's check with effulgentsia though before making a commit(ment) here.
Comment #187
alexpottFor me #180.2 blocks commit - but @effulgentsia is not awake yet - going to discuss with him once he is up.
Comment #188
Wim LeersImplemented #180.2.
Comment #189
Wim LeersI personally agree with that.
Comment #190
Wim LeersBonus:
19 files changed, 236 insertions(+), 18 deletions(-)
18 files changed, 211 insertions(+), 15 deletions(-)
Fewer changes :)
Comment #191
alexpottThanks @Wim Leers I think the interdiff shows why #180.2 was good to fix.
Also as pointed out by @Wim Leers the docs for uriRelationships() says that it returns supported relationships. Returning a relationship that can cause RouteNotFoundException would be against that.
Comment #192
effulgentsia CreditAttribution: effulgentsia at Acquia commented+1. This is great.
Regardless of where the flaw lies, I don't think the patch is committable if by encountering the flaw, it's creating a WSOD. Clarifying #181.2. If I do this in HEAD:
rest
,hal
, andbasic_auth
modules./
or/node
.Then in visiting either of those last 2 URLs, I get a nicely themed "Page not found" page, which is correct.
If I then apply #188, then visiting either of those URLs shows me a white screen of death, with the infamous
"The website encountered an unexpected error. Please try again later."
message.The above problem on its own would be at minimum a Major bug ("Trigger a PHP error through the user interface, but only under rare circumstances") if you consider disabling the Frontpage View to be a rare circumstance. Except this can also be triggered by disabling the Views module itself too. Furthermore, you can fix the error at
/
by editing the frontpage path at/admin/config/system/site-information
, but the WSOD still remains when you visit/node
. And I think that's made even worse by the<link rel="create" href="/node?node=1" />
being output from/node/1
, since then you're telling the user agent that there's a resource at/node
but there isn't (at least not one available to this user agent).Setting to Needs review for more discussion on whether others agree or disagree with me about this being a commit blocker.
Comment #193
dawehnerTo be honest though its kind of good pratice to disable the frontpage view. I'm looking into that for a while.
Comment #194
dawehnerTo solve this properly we IMHO should catch any client side errors.
After applying the latest changes you get a
page, not rendered as nicely as the 403/404 pages, but at least something.
Comment #197
alexpottIf we do add this to solve the problem described in #192 we need to test it.
Comment #198
xjmComment #199
Wim LeersThe website encountered an unexpected error. Please try again later.
That's what you would get when you have verbose errors disabled. But people developing sites don't have that disabled; they have it enabled. Then they get this instead:
That's actually a very helpful error!
You can easily also get exactly the same at every
https://www.drupal.org/link-relations/create
link relation URL. For example, for taxonomy terms:Now
GET /taxonomy/term
results in the exact same error as the one you reported for/node
.The reason for those error responses is that you're accessing this URL without specifying a format (i.e. without
?_format=…
). Therefore the default of HTML is assumed. And the default exception subscriber for HTML only handles 401, 403 and 404.I'm not sure #194 is the best approach. It's technically correct ("handle all 4xx errors other than 401/403/404"), but does it also match the expected behavior? Wouldn't it from a HTML/browser POV make more sense to map e.g. 405 errors (which is what we're seeing here) to a 404, because… there is no HTML to be found? Which is semantically equivalent with a 404. EDIT: that also happens to be the behavior before this patch, but that's not even why I'm proposing this.
I disagree that improving the handling of accessing a "REST-only route" without specifying a format is something we need to improve here. It's a pre-existing problem. It's a problem that's out of scope for this issue. And I think the scenario in #192 is fairly contrived. I don't see what the big problem is with making this a bit more likely to occur. It's a problem we need to solve regardless of this issue.
(And this once again goes to show that Drupal is still far away from being "API-first".)
Comment #200
dawehnerThis is weird. This kinda of mixes up different layers. Why should HTML care about the status code and wise versa.
It feels like we are so used to the idea of 403/404 as users, that we somehow think that they are special.
Comment #201
effulgentsia CreditAttribution: effulgentsia at Acquia commentedTrue, this is a pre-existing problem. Even without this patch's addition of the "create" relation, we're already encountering this problem with the "delete-form" relation. The closest issue we have currently open for fixing it is #2848945: EntityType link templates: using link relation type names as keys is a problem. This was actually leading to HTML validation problems, which we provided a partial workaround for in #2406533: edit-form, delete-form etc. <link> tags added on /node/{node} are invalid according to W3C Validator. Except that workaround only sidestepped it for anonymous users, which means we're still producing invalid HTML for authenticated users. Plus, that workaround was only applied to nodes, which means we're still producing invalid HTML for authenticated and anonymous users for taxonomy terms: #2821635: edit-form, delete-form etc. <link> tags added on /taxonomy/term/{taxonomy_term} are invalid according to W3C Validator.
The issue summary of #2406533: edit-form, delete-form etc. <link> tags added on /node/{node} are invalid according to W3C Validator says:
Do we know for sure that Google and other service engines access all link relations? Even ones not registered with IANA, such as "delete-form" and "create"? In which case, this patch is adding another one ("create") for it to try to access and fail. Maybe that's ok? It's just one more 4xx response for the engine to deal with. Except, while in HEAD, we have pre-existing ones that return 403s, this patch adds one that returns a 405. Again, maybe that's ok, but might be good to do some research on whether it is or not. #2406533-113: edit-form, delete-form etc. <link> tags added on /node/{node} are invalid according to W3C Validator said that even the 403s for the relation links coming from node pages was a severe bug. I don't know if adding links to 405 pages for authenticated node traffic or anonymous taxonomy term traffic would add additional substantial severity to the remaining existing problems in HEAD.
Comment #202
Wim LeersDo we know people at Google who can answer this with confidence? The fact that it's so hard to find information about this when searching the web means we're probably the first ones to do this, or the first major CMS at least.
Comment #203
xjmKeep in mind that there's a reasonable chance that someone at Google couldn't answer the question because I'd bet it might be covered by NDA. (Since gaming search engine rankings is something they have to be continually watchful about, and etc.)
Comment #204
Wim LeersAnother two weeks that have passed without progress. We're effectively blocking a massive DX improvement/the removal of a major Drupal WTF from our REST API on SEO matters. That's unfortunate :(
I also pinged somebody at Google, hopefully he'll point us in the right direction: https://twitter.com/wimleers/status/837293938084040704.
Comment #205
Wim LeersI got a response from the person at Google:
That's no conclusive evidence, but a strong indicator that Google indeed does not penalize this.
In the mean time I've found more evidence that link relations (
<link rel="">
orLink
header) indeed do not need to be tied to a HTML presentation:<link rel="search" type="application/opensearchdescription+xml" href="https://s1.wp.com/opensearch.xml" title="WordPress.com" />
— this is clearly serving an XML file that is completely unusable for a human user, but is relevant to a particular type of user agent: one that renders HTML.<link rel='dns-prefetch' href='//s2.wp.com' />
— this is not serving anything, and is a directive for low-level implementation details of HTML-rendering browsers, to accelerate the fetching of referenced resources (CSS, JS, images)element for autodiscovery, and uses a single XML-RPC call for notifying the target site of the link on the source site.
This is NOT something that a browsing user agent even can do. It's information specifically for other sites. Not a robot or human browsing the internet. It's for a site to act as a user agent whenever content is posted that references content hosted on a pingback-enabled site.
describes
link relation type specifically allows one resource accessible via HTTP to express that it describes the structure of another resource. This is RFC6892. It mentions RDF, JSON and XML. And it has this gem:(Emphasis mine.)
In other words: link relations are not meant to be HTML-specific or HTML-centric. They are specifically meant to be representation-agnostic. It's up to the user agent to first inspect the
rel
, and check if this particular link relation type is something that this user agent A) supports, B) surfaces to the end user in some way.I also find this confirmed in the very introduction of the RFC that standardized the concept of link relation types, RFC5988:
And from section 4:
Comment #206
Wim LeersI believe the above presents overwhelming evidence that the approach in the current patch is justified. And that we do not need to distinguish between "HTML link relation types" and "non-HTML link relation types".
Hence we wouldn't need to include @dawehner's proposed changes in #194.
But @dawehner's changes still help the edge case that @effulgentsia described in #192: of removing the front page view and enabling the REST+HAL modules, then getting plain text 405 responses for
/node
+/
.So, let's keep the changes made by @dawehner in #194, but make them pass tests.
#2248983: Define the revision metadata base fields in the entity annotation in order for the storage to create them only in the revision table conflicted with this slightly. Rebased too.
Comment #207
jofitz CreditAttribution: jofitz at ComputerMinds commentedPatch no longer applies after #2776975: March 3, 2017: Convert core to array syntax coding standards for Drupal 8.3.x RC phase.
Comment #208
jofitz CreditAttribution: jofitz at ComputerMinds commentedRe-rolled, ensuring [] array syntax.
Comment #209
Wim LeersManually diffed #206 and #208. Confirming this is a correct straight rebase. (The size difference is attributable to the fact that #206 is rolled with diff stats enabled, and #208 is not.)
Comment #210
Munavijayalakshmi CreditAttribution: Munavijayalakshmi at Valuebound commentedLine exceeding 80 characters.
Line exceeding 80 characters.
Comments should (noramlly) begin with a capital letter and end with a full stop / period .
Comment #211
Munavijayalakshmi CreditAttribution: Munavijayalakshmi at Valuebound commentedComment #212
dawehnerThe only weird bit about this code is that we now generate each URL twice. Once here, and once when we actually render the link relationship. Is this something we should worry about?
I'm wondering whether we should add the route inside
ResourceBase
Comment #213
Wim Leers#212:
RouteProvider::getRouteByName()
… but it'd be rather difficult to actually do that, because we also allow an entity type to specify anuri_callback
, which is what is used if a particular link template is not specified:The support for
uri_callback
means we cannot simply rely onRouteProvider::getRouteByName()
. Which means we need to call the URI callback, which means we need to actually generate the URL after all AFAICT :(ResourceBase
(or any\Drupal\rest\Plugin\ResourceInterface
) implementation returns in itsroutes()
method, the route names that that returns are always prefixed byrest.
in\Drupal\rest\Routing\ResourceRoutes
. Let me quote what I wrote in #127:Comment #214
dawehnerAt least for me the
uri_callback
concept is just fundamentally flawed and I would be surprised it actually works properly in all cases still.Ah, why can't everything be nice. I wonder whether you could still generate the routes there, but then in
EntityResourcePostRouteSubscriber
just renames the routes?Comment #215
Wim LeersWell,
uri_callback
is still an official API, and supported, so we can't really ignore that I think?Regarding generating routes in
ResourceBase
and then renaming them in the event subscriber: that'd be possible too, yes. I'm not sure whether it's better. Both approaches are very similar. I don't have a strong opinion on either approach. Do you want me to change it to what you proposed?Comment #216
dawehnerIt would be nice if plugins could easily change the route somehow, if needed.
Comment #217
Wim LeersBut when would that ever be needed? The whole point is that this
create
route is just pointing to the exact same route definition. You can change the route definition to anything you want in yourResourceInterface
-implementing plugin. Theentity.$entity_type_id.create
that we create automatically would just use exactly that same route definition.That's all this is: two route names using the same route definition, so that you can generate URLs to this route using either route name.
Comment #218
dawehnerOh sorry, I totally forgot what this patch was doing, aka. exactly what I though should be done :)
Comment #219
Wim LeersHah! Great :D
Comment #221
jofitz CreditAttribution: jofitz at ComputerMinds commentedPatch no longer applies - re-rolled.
Comment #223
jofitz CreditAttribution: jofitz at ComputerMinds commentedSorry, messed up that re-roll. Let's try again.
Comment #224
Wim LeersConflicted in
EntityResourceTestBase
due to changes introduced in #2853211: EntityResource::post() incorrectly assumes that every entity type has a canonical URL.Back to RTBC!
Comment #226
dawehnerThat seemed to be a random fluctuation.
Comment #228
Wim LeersAgain.
Comment #230
dawehnerJust a quick reroll
Comment #231
Wim LeersComment #233
Wim LeersStraight rebase. (This conflicted with #2848927: EntityResource::addLinkHeaders() should use LinkRelationType plugin instances rather than definitions, which was committed yesterday.)
Comment #235
Wim LeersComment #237
dawehnerComment #239
Wim LeersThis conflicted with #2869415: EntityResourceTestBase::getUrl() overrides BrowserTestBase::getUrl(), yet offers different functionality, but the conflict was trivial to resolve :) (
EntityResourceTestBase::getPostUrl()
was renamed to
EntityResourceTestBase::getEntityResourcePostUrl()
.)Comment #240
alexpottCommitted 4398109 and pushed to 8.4.x. Thanks!
I've checked all the credit assigned to reviewers and it looks correct.
Comment #242
Wim LeersThis finally unblocked #2800097: Confusing UI: POST URL differs from GET/PATCH/DELETE URL, but UI doesn't indicate this!, which I indicated in #164 was being blocked by this.
It also unblocks a next step: #2851984: Add "create" link template to User entity type annotation, to allow POSTing to /user instead of /entity/user.
So glad this is finally in!
It’s been in reroll hell since March 21 (almost 2 months ago). Before that, it was RTBC on Feb 14. And before that it was RTBC in June 2016! It took a year to get this fixed!
Comment #243
andypostChange record said about 8.3 looks it needs update
Comment #244
Wim Leers#243: I noticed that myself too, already fixed :)