Updated: Comment #136
Problem/Motivation
In some formats, lists of items is not just a list of items. They are a collection with metadata containing a list. This is important to provide context about the list, such as overall size, a description of why those particular items are grouped together, and other contextual or semantic details.
We are currently in violation of the HAL specification for how collections should be handled, which can be seen in section 6 of the specification. This should look like:
{
"_links": {
"self": { "href": "/view-uri" }
},
"_embedded": {
"items": [
Array of serialized entities goes here
]
}
}
Because the collection members are nested, the top-level namespace can be cleanly reserved the HAL _links
element, and could also be home for other metadata. Making space for the _links
element also allows for Collection level operations, such as pagination across the collection.
The use of the HAL _embedded
structure makes it semantically clear that the contents of the collection canonically live as part of a different resource. Tools exist to interpret this structure, as is described under testing instructions.
Another strong example of the need for a more robust collection structure is RSS feeds, which contain top-level channel properties such as:
pubDate - The publication date of this set of content, in RFC 822 date format
language - The language the channel is written in
category - One or more (specified by multiple tags) categories the channel belongs to
Note that denormalization of collections and the means collection members are identified is currently out of scope, but this could be a good foundation to facilitate both.
Proposed resolution
Create a new Collection concept that serves as a structured intermediary between data and the serialization process. Views or other lists of data can initialize a collection instance then use that as the data for REST responses. This will allow individual serializers to decide how they need to handle collection formatting by implementing an applicable normalizer.
- Add a Collection class to Serialization module.
- Change Views REST Export to pass a Collection to serializers rather than an array.
- Add a Normalizer to HAL for EntityCollection to match the structure above and incorporate pagination, title, and description links.
- Use a link manager to determine the link relations from collection to item
How to manually test
To test the HAL-specific structuring brought in this patch, you can use the HAL Browser tool to explore collections once the change is in place.
- Enable Hal module.
- Update http://drupal.d8/admin/structure/views/view/frontpage/edit
- Add a Rest export display and set it's path to /node
- Install in your drupal root
git clone https://github.com/mikekelly/hal-browser
- Visit http://drupal.d8/hal-browser/browser.html
- Result should look similar to http://haltalk.herokuapp.com/explorer/browser.html#/users
Remaining tasks
- Add a test for EntityCollection output in HAL
- Fix failing tests
Unaddressed Review Items
Fix item #11 and #19 from #2100637-36: REST views: add special handling for collections, #2100637-89: REST views: add special handling for collections (All points except 3, 4, 7, & 9) with #2100637-99: REST views: add special handling for collections, #2100637-124: REST views: add special handling for collections,
The items from #36
#11
+++ b/core/modules/rest/src/LinkManager/CollectionLinkManagerInterface.php
@@ -0,0 +1,25 @@
+/**
+ * Interface for mapping collection (e.g. Views) link relations.
+ */
+interface CollectionLinkManagerInterface {
It is odd that we talk about mapping link relations but the class name nor namespace mentions it.
#19
+++ b/core/modules/serialization/src/Collection.php
@@ -0,0 +1,214 @@
+ * Provides a wrapper for a collection of entities, e.g. a feed channel.
...
+ * The entities in the collection.
...
+ * var array
Are these any kind of things or just entities?
Comment | File | Size | Author |
---|---|---|---|
#154 | 2100637-154.patch | 56.63 KB | tedbow |
#150 | special_handling_collections-2100637-150-collections-do-not-test.patch | 57.34 KB | dmouse |
#145 | interdiff.txt | 3.5 KB | Grayside |
#145 | 2100637-145.patch | 56.63 KB | Grayside |
#142 | interdiff-2758563-140-142.txt | 1.07 KB | Grayside |
Comments
Comment #1
linclark CreditAttribution: linclark commentedHere is a patch that adds basic support.
Comment #2
linclark CreditAttribution: linclark commentedThis patch adds a LinkManager for Collections. This is used to determine the link relation from the collection to its items.
Comment #4
linclark CreditAttribution: linclark commentedFixing the test fails...
FieldHelpTest passes on my local. I'm not sure what's up with LocaleUpdateTest.
Comment #4.0
linclark CreditAttribution: linclark commentedUpdated issue summary.
Comment #5
linclark CreditAttribution: linclark commentedAdded a PHPUnit test for HAL's EntityCollectionNormalizer.
Comment #5.0
linclark CreditAttribution: linclark commentedUpdated issue summary.
Comment #6
klausiCould you extend the example HAL representation in the issue summary with the title, description and other properties, so that we get a picture what the full result looks like?
So this method is empty, why? Are we not going to support denormalization? Please add a comment.
So as you said this works well for full entities, could we also use the EntityCollection class to pass in primitive rows built by views?
So this looks like a data junk class. Basically we are just using this class so that the serializer can detect it and knows what to do, right? Actually we are just dealing with some metadata like title, URI and description of the collection + an array of entities, that's it. Could we just use an getCollectionMetadata() method that returns an array with all those properties? Is there a compelling reason why we want that all as separate properties? I'm not sure about this, the empty data junk class just looks a bit weird to me. And what properties do we define, which ones are required?
Overall I think we are on the right track here. Haven't looked to closely at the link manager changes, so someone else should review that.
Comment #7
linclark CreditAttribution: linclark commentedI will work on this.
We might as well support denormalization. I'll probably implement that in a follow-up though, so I'll add a comment.
If those rows can be deserialized individually, then they could be deserialized in the EntityCollection. They currently cannot be deserialized individually, though, and I don't think we should support that in core since we don't have a use case for it.
There might also be other properties that one would want to add. For example, RSS has the following commonly used channel elements:
I think it is more self documenting to have these as getters and setters. This is also the way that ZendFeed does it. For example:
Comment #8
linclark CreditAttribution: linclark commentedAdded the comment.
Comment #8.0
linclark CreditAttribution: linclark commentedAdded code example
Comment #9
moshe weitzman CreditAttribution: moshe weitzman commentedThis affects Views primarily so perhaps this tag is appropriate.
Comment #10
frega CreditAttribution: frega commentedRerolled linclark's patch.
Added & adjusted integration test cases in core/modules/rest/lib/Drupal/rest/Tests/Views/StyleSerializerTest.php
Added & adjusted unit tests in core/modules/hal/tests/EntityCollectionNormalizerTest.php
There are a few todos and prickly points left, and in general we should consider renaming EntityCollection to RowCollection (as we could leverage it for paging all kinds of views), we need to move the collection factory (getEntityCollectionFromView) from \Drupal\rest\Plugin\views\style\Serializer to somewhere more reasonable ...
Comment #12
frega CreditAttribution: frega commentedMissed re-adding the CollectionLinkManager to the LinkManager instantiation in NormalizerTestBase (changed signature of RelationLinkManager). Interdiff and new patch attach.
Comment #13
klausihttp://tools.ietf.org/html/draft-kelly-json-hal-05#section-6 HAL spec info
Comment #14
frega CreditAttribution: frega commented- Improved test coverage (more comprehensive integration tests for views.view.test_serializer_display_entity.yml and views.view.test_serializer_display_field.yml including paging; added unit test for \Drupal\serialization\Collection).
- Refactored/Renamed \Drupal\serialization\EntityCollection to \Drupal\serialization\Collection
- Use injected UrlGenerator service instead of url() style/Serializer.php-Plugin
- drupalcs love.
Hefty interdiff due to lot's of renaming.
Comment #16
dawehnerImpressive work, indeed.
Is there any way to spin of part of the big patch?
I wonder whether we should implement a last page but rather not use a count query.
Comment #17
frega CreditAttribution: frega commentedRerolled patch as i botched the rebase in #14 :(
@dawehner: happy to split, slice and dice as you see fit :) I'll ping you on IRC.
Comment #18
klausiWhy don't we support JSON? I think it should just look the same as HAL.
I think this @todo can be removed? It should work regardless of what the items are?
Comment #19
frega CreditAttribution: frega commentedhej klausi,
re: 1. that's been like that since lin's patch in #8 (https://drupal.org/files/2100637-08-collections.patch); in principle we can of course serialize to JSON, but there would be no defined way to add link relations (prev, next, etc.) I dumping the collection to JSON is a piece of pie.
re: 2. i think that's true - i would remove and revisit all todos in one go if that's ok?
best, f.
Comment #20
clemens.tolboomAdded a manual test section
Comment #23
clemens.tolboomDid a reroll. Let's see what fails.
[edit]Pager is working fine[/edit]
Comment #25
clemens.tolboom[edit]deleted as pager is working fine.[/edit]
Comment #26
clemens.tolboomFixed the duplicate 'use ...String' failure.
Comment #27
clemens.tolboomViews preview is not working while it was in the old rest_display
I get
[Tue Jul 29 13:05:16 2014] [error] [client 127.0.0.1] Uncaught PHP Exception Symfony\\Component\\Serializer\\Exception\\UnexpectedValueException: "Serialization for the format drupal_ajax is not supported" at /Users/clemens/Sites/drupal/d8/www/core/vendor/symfony/serializer/Symfony/Component/Serializer/Serializer.php line 77, referer: http://drupal.d8/admin/structure/views/view/frontpage/edit/rest_export_1
Comment #29
clemens.tolboomFixed failing tests (at least locally)
Comment #30
clemens.tolboomComment #31
clemens.tolboomSorry testbot for misnaming the interdiff :-(
Comment #33
Crell CreditAttribution: Crell commentedI think, based on #31?
Comment #35
clemens.tolboomPatch adds documentation and variables annotations. This make the code a little more navigable.
Comment #36
dawehnerThe one line description is a copy and paste
Do we have a follow up for this already?
Let's use absolute documentation.
If we document this here, which sounds like a good idea, let's not try to shortcut it.
???
Can't you just typehint type data itself? this is where getParent lives.
absolute.
It is just crazy how much this patch is blown up by unrelated changes
absolute again.
Let's use @coversDefaultClass and @covers here.
It is odd that we talk about mapping link relations but the class name nor namespace mentions it.
Ensure that all files have an empty line before the last }
Isn't that solved already, as the row plugin would be different and so different values are returned? I guess we can drop the todo here? Afaik we should have test coverage so we are safe.
I would rather go with $view->getTitle() to be honest.
Mh, this works fine for the normal cases, but we might have a different route name if something overrides an existing route. You could also use something like $state = \Drupal::state(); $route_names = $state->get('views.view_route_names'); $route_names["$view_id.$display_id"];
we don't need getInfo any longer.
there is no reason for the initDisplay() call. setDisplay() already does that.
i don't really see a value in this line of documentation, do you?
Are these any kind of things or just entities?
Comment #37
clemens.tolboom#4
Do you mean add a sentence?
I will work through the list from #36 tomorrow.
Comment #38
clemens.tolboom#2 No follow up found. Not sure what to create yet.
#5 See #2
#6 PHP Storm seems to need this
#8 I wasn't aware of https://www.drupal.org/coding-standards
so will rethink next issue.
#10 not sure what to add as a whole. Added only @coversDefaultClass ... https://www.drupal.org/node/2325985
#11 skipped for now
#13 @todo removed
#16 removed the
public static function getInfo() {
but should we save to info data?#19 skipped for now
Added to summary:
#2 No follow up found. Not sure what to create yet.
#11 skipped for now
#19 skipped for now
Comment #40
clemens.tolboomArgh
Is that required?!? See http://phpunit.de/manual/current/en/appendixes.annotations.html#appendix...
Added to #2325985: No documentation about @group @coversDefaultClass @covers
Comment #41
clemens.tolboomAdded @group back to test
Comment #43
clemens.tolboomJust a reroll.
Comment #45
clemens.tolboomSomehow the rest.services.yml missed
That makes the views Rest export display working. Tests still failing. It cannot load the test views somehow.
Comment #47
clemens.tolboomFixed missing pagers thanks to @dawehner
This still needs some work.
Comment #48
clemens.tolboomComment #49
clemens.tolboomAdded outstanding issue from #36 to summary.
Comment #50
dawehnerJust some short feedback. I think this patch should get some review from damiankloip, as he is aware of both serialization and views.
So let's do it here, I guess? This then also needs a test coverage.
Nitpick: Add a newline between.
Ah I see, so normalize() exists on the serializer, not on the interface, maybe we want to typehint against the NormalizerInterface in that case.
Comment #51
clemens.tolboom@dawehner
#1 : This needs some hints :-/
Fix #2 and #3
Comment #53
clemens.tolboomreroll
Comment #54
clemens.tolboomComment #55
clemens.tolboomNeeded a reroll for fgm
Comment #57
damiankloip CreditAttribution: damiankloip commentedGenerally, this is looking good. An epic bit of work!
I don't think we should check if it's an array here, that silent failure stuff...
Fair enough, let's create a follow up and link it here.
Let's add a todo here too to remove this method based on above.
When do we return a regular object?
The serializer implements the DenormalizerInterface which has this method on.
Not inheritdoc and other docs. Just the first :)
That's implicitly done anyway, so this doesn't make any function change. Not really needed, but meh. Don't care either way on that.
Same as previous comment. NormalizerInterface has this.
Won't make any difference to the tests, but we don't have a cache.cache bin anymore, Just cache.default.
Don;t need getInfo for unit tests anymore. The group and @coversDefaultClass/@covers is good.
stub/mock
That is not true, Serializer implements many interfaces. Can also pipe| the phpunit mock object class to that.
This is added twice
Needs a one line description like 'The URL generator' pointless, yes pretty much. That's just the way it is.
This should be injected too.
Move the processing of the rows to near here. Easier to read and keep track of then.
The url generator
Maybe at least initStyle() should be called here?
Two separate assertions please.
Is this getting done? :)
This does not have to apply only to entities. E.g. what happens if you use the field data row plugin? I guess it's fine just docs need updating?
Should typehint an array
Should always be an array, so just return
(bool) count()
expected value first.
Comment #58
damiankloip CreditAttribution: damiankloip commentedAlso, another problem is that this will totally break RestExport/Serializer if hal is not enabled. As serialization module does not have a normalizer for collections.
There is also quite a lot of processing we don't want to do unless we are using the hal format. e.g. regular json does not care about links etc... I just want the raw data.
Comment #59
fgmI rerolled the patch this morning too: very little difference, mostly:
- removed a duplicate service definition
- fixed the JSON namespace import
- 2 instances of == NULL replaced by === NULL for accuracy
- removed various unused imports and out of date /** var ...*/
Comment #60
fgmForgot to set CNR to trigger bot.
Comment #62
fgmRerolled:
- $this->getName is now $this->getMachineName() in tests
- url() is now Url::fromUri().
Comment #65
fgmNote: the massive fails are due to a problem with the testbots (too many SQL connections), not with the test itself, so retrying until the bots work again.
Comment #70
clemens.tolboom@fgm do you have an interdiff and have you time for the feedback of @damiankloip in #57 or should I take those?
Added #2350401: [PP-1] Collection denormalization thus removed one TODO from summary.
Comment #72
fgm@clemens.tolboom : here is the diff between the two versions, in patch format. Yes, I can continue with Damiankoip's remarks today.
Comment #74
clemens.tolboom@fgm awesome. Your interdiff contained unrelated js stuff .. np. The interdiff could have the .txt extension to skip testing. Thanks for working on this :0
Comment #76
damiankloip CreditAttribution: damiankloip commented#58 is the one the biggest problems here at the moment, I think.
Comment #77
clemens.tolboom@damiankloip
Is this patch causing this? Or is the current 'Rest export display' breaks when disabling hal?
Guess that's a serialization problem right. What about #2343431: Make JSON data non-Drupalish and #2339795: Add "REST modes", just like we have "view modes" and "form modes" AND depth argument to reduce # of requests
Comment #78
fgmRegarding 58, I think Hal should be just a serialization format, and "plain JSON", whatever it is, be another, just like XML, or site-specific serializations like application/myawesomesite+json
Comment #79
fgmHopefully everything passing this time.
Comment #80
fgmSince the bot doesn't seem to have sent its notification yet: the test suite passes.
Comment #81
fgmStarted work on issues in #57, time for a first check:
1. I think we need to check if it is an array (would we accept an iterator ?), to avoid an error in the subsequent foreach.
1bis. Since $object is not type-hinted, we need to behave propertly if the method is not called with a Collection object.
2: Followup is #2350401: [PP-1] Collection denormalization.
3. Done.
4. Actually, we only return an array now. Phpdoc changed.
5. Actually, the code only implies that ContentEntityNormalizer, not
ContentEntityNormalizer->serializer implements DenormalizerInterface: the
serializer property is declared in SerializerAwareNormalizer as implementing
only SerializerInterface, which does not include (de)normalize methods. So
that remains a @todo
6. Done.
7. Probably more readable that way, let's keep it.
8. Same remaining problem as 5.
9. Bin changed to "default", but the MemoryBackend ignores the bin anyway.
10. Done.
11. Done.
12. Done. Loooon type info: does it really make sense ?
13. Done.
14. Done.
Also removed a duplicate copy of CollectionNormalizerTest (one in src/Tests/, one in tests/ ).
Comment #82
fgmAddressed the rest of the #57 points:
15. Done
16. Done
17. Done
18. Probably not ? $view->getStyle() includes a call to initStyle()
19. Done
20. Not yet
21. Done
22. Done
23. Done
24. Done
Comment #84
fgmSo the failing test is a consequence of point 22 : typehinting an array on Collection::setLinks().
There are two ways to fix this:
- removing the type hint
- removing the failing test : this is part of a unit test in which there is an explicit call to setLinks(NULL), which does not seem to make sense as all other use cases correctly pass a (possibly empty) array
So rerolling with the setLinks(NULL) test removed and the type hint kept : this makes the API more consistent.
Also fixed a remining getName() -> getMachineName() call and removed another duplicated test.
Comment #85
larowlanFirst observation - there are a lot of out of scope coding-standards changes in this patch that make reviews difficult - please split them into a separate patch/issue to reduce the size of this patch. Given they're all minor those changes could be in by now if they were in a separate issue and reviews would be more forthcoming here.
There are two issues I see here, one is there is no interface for Collection object, the other is the line that does $this->view = $this->view, which seems like an error. The rest is CS nitpicks in the new code or CS changes in existing code that are out of scope.
Other than that, looks great.
out of scope
missing a blank line before the last }
this looks wrong
out of scope
out of scope
is there an issue for this?
There are a lot of public methods here so we should be adding an interface too, and type-hinting that
nitpick needs space before the last }
out of scope
Comment #86
clemens.tolboomAs discussed on IRC we need to remove the out of scope changes. I'll try to do that over the weekend.
Comment #87
larowlanReverted out of scope stuff (git checkout -p is the business)
Comment #88
larowlanre-roll
Comment #89
Crell CreditAttribution: Crell commentedFinally reviewing this!
The no foreach here seems suspect. If it actually works, why is this different than the way links work/are added?
I'm not sure that we need denormalization of collections in the first place, in practice. Is there a use case?
Mmm... standards. :-)
That said, do we need a whole service to define one string? Couldn't that just be a container parameter or something if we want it swappable?
The description for this method makes no sense to me. Wha?
Strictly speaking, first/last have been removed from the standards since few if any implementers used them. That said, I'm fine with leaving them as I think implementers who didn't implement them are idiots. :-)
The first link,though, should not specify a page parameter at all in order to be consistent with the way we use pagers elsewhere and to have a consistent URI.
If the previous page is the first page, then we should omit the page property as above.
All of these setLink() lines would probably be easier to read with [] short array syntax. Especially for the empty arrays.
Strictly speaking, this test is order-sensitive. HAL, however, is not order-sensitive on links. That the order is the same here is coincidental and based on knowledge of how the code happens to be written right now.
It would be more stable to do a series of assertTrue(in_array()) calls, or else order the keys alphabetically and then check against a known alphabetized list.
Actually, even the latter is wrong because it doesn't allow for the presence of additional links. The presence of additional links is entirely legal; we just want to be sure these links are present.
So that means the most correct test would be:
Or something like that (I didn't actually test the above line.)
Initialize to empty array. []
Human-readable? Translated? If so, should it be called label (since that's what we call it on entities) or NOT be called label (since that's what we call it on entities)?
As above.
What is the structure of this array? And if each link is itself structured, should those be objects of their own (a la HtmlFragment, RIP)?
Also, the list of link names should be explicitly listed as examples. Having other links on a collection is entirely legal and something we want to encourage.
From the code below, it appears that we only support the link url itself here. However, that is insufficient as it is completely legal in HAL to have links that have additional metadata properties, such as hreflang, templated = true, etc. We need to allow for those.
Wouldn't this be for iteration?
When would we want to set them all at once?
Should this be a string, or a Url object? Url object would be consistent with the rest of core, but string would be more decoupled. Hm. I am not sure here.
While a collection won't have multiple first/last/prev/next links, realistically, it is 100% legal for there to be multiple links of the same relationship from one resource to another. We need to allow for that. Hence, this should really be addLink(), and we store potentially multiple links of the same relationship. and then getLink() returns an array of links, or [] if there are none of that type.
Given the structure this could have (as above), this method strikes me as unweidly in practice.
This is a similar situation as is being discussed for the PSR-7 HTTP standard interfaces in FIG, where we recently removed the set-multiple option to avoid the confusion that it brought.
I do not use case for this method. I can see hasLink($relationship), but not hasAnyLinksAtAll().
Comment #90
rteijeiro CreditAttribution: rteijeiro commentedI re-rolled the patch and fixed a few of the things @Crell has commented (3, 4, 7 and 9) but it still needs work. I can continue working on it but need some guidance or feedback.
Provided an interdiff for easier review.
Comment #92
rteijeiro CreditAttribution: rteijeiro commentedOps! It seems that test fails due to change suggested by @Crell in point 9. Any thoughts?
Comment #93
Crell CreditAttribution: Crell commentedWell, [] is not null. :-)
Comment #94
R.Muilwijk CreditAttribution: R.Muilwijk commentedChanged assertion to check whether it's an empty array.
Comment #95
Crell CreditAttribution: Crell commentedWhat assistance do you still need beyond the recommendations I provided in #89?
Comment #96
clemens.tolboomPatch needed a reroll.
Comment #97
clemens.tolboom#50 Open #1?
#89 Open all but 3,4,7,9
Comment #99
clemens.tolboomA reply to #89
#1
The relation is single: _embedded.item. The call adds the items so no need for a for loop right?
#2
One such usecase could be the new `RSS` importing a HAL collection news feed. Not sure that is a good usecase. We do have #2350401: [PP-1] Collection denormalization
On ie a twitter timeline there is no real first/last same as watchdog log. There are always new items added in time.
I think views should not add page# but item.id instead :p
But let's do it the views way
array_diff()
the result should be empty.Where is that title used? And even tested? I do not see it in the output for hal+json or json. Should it be the views title?
Title is used on HTML and Feed. Feed has outputs description from setting in views.
I guess we should output title and description too.
Keyed array of Link Objects.
We use the keys previous, next, first, last to support pagination.
Linked Objects are defined on https://tools.ietf.org/html/draft-kelly-json-hal-06#section-5
Please explain?
My guess is views adds all items at once.
Collection->setUri is done in core/modules/rest/src/Plugin/views/style/Serializer.php:194 which seems views related. And is a string.
Taken from https://tools.ietf.org/html/draft-kelly-json-hal-06#section-4.1.1
Comment #100
clemens.tolboomPatch needs a reroll from
Comment #101
cyborg_572 CreditAttribution: cyborg_572 at Northern Commerce commentedAt Drupalcon LA, working on the re-roll
Comment #102
cyborg_572 CreditAttribution: cyborg_572 at Northern Commerce commentedRerolled the patch. I'm not 100% sure about the resolution in core/modules/rest/src/Tests/Views/StyleSerializerTest.php though.
Comment #103
clemens.tolboom@cyborg_572 thanks for the reroll. Let's see what the testbot thinks.
Comment #105
clemens.tolboomFailing tests are
Drupal\comment\Tests\Views\CommentRestExportTest
ExpandDrupal\rest\Tests\Views\StyleSerializerTest
I tried to fix StyleSerializerTest but failed. The exceptions for StyleSerializerTest are gone
This obviously need work :-/
Comment #107
clemens.tolboomPatch needed a reroll done from 29429bda910e398601fbace6a86f094000641ef4
I managed to come close. But /node is now taken over by rest/hal
http://drupal.d8 produces json NOT OK
http://drupal.d8?_format=json produces json OK
http://drupal.d8?_format=hal_json produces hal+json OK
Let's see what else breaks.
Comment #110
clemens.tolboomThis needs a reroll either from #105 (guess that's the best option) or #107 (which was my bad effort)
Check the "How to test" from the issue summary too :-)
Comment #111
nlisgo CreditAttribution: nlisgo commentedComment #112
nlisgo CreditAttribution: nlisgo commentedStepping away from it for now. Will take more time than I have today to make sure that I prepare the reroll properly as there are quite a few conflicts. I will look back in next week and help out if I have some more time.
Comment #113
dawehnerAt that point in time, this is more of a 8.1.x feature to be honest.
Comment #114
marthinal CreditAttribution: marthinal commentedRerolled
Fixed CommentRestExportTest and StyleSerializerTest problems.
We need the route parameters on Serializer::getCollection() to fix CommentRestExportTest.
1- Added a couple of TODOs to StyleSerializerTest
Not sure if I can remove this assertion.
2- Not sure if we should use
instead of
I had problems using drupalGetHalJson() to obtain the hal_json format and using drupalGetWithFormat() it works as expected. Anyway we can refactor later.
Locally looks ok but let's take a look at the test results.
Comment #116
marthinal CreditAttribution: marthinal commentedRemoving debug() from tests. Locally the tests are working as expected ... Not sure where is the problem for the moment
Comment #118
marthinal CreditAttribution: marthinal commentedOk Let's try again.
Comment #119
marthinal CreditAttribution: marthinal commentedNot sure why we have this result for totalItems using mini pager.
Comment #120
aneek CreditAttribution: aneek as a volunteer commented@marthinal,
I did some debug and found that this is coming from the mini pager plugin
core/modules/views/src/Plugin/views/pager/Mini.php:84
. Well, it says thatceil()
will not fail but eventually it fails as per your attached screenshot. I believe, the reason behind is if a number has more digits than 13 it will automatically converted to a float value (see the references) also according to this S.O post.So this can be solved as per the patch I am attaching by just using number_format. But from user end giving a big number as the last link is not a good way. I think in case if a view has a mini pager selected we should only show the "Previous" and "Next" links with "Self" as the HATEOAS pagination links. This should match the same behavior as per the website's views.
I am attaching a new patch based on my views above. Some tests might fail due to the changes that I've done. I'll have more updates on this after first bot review. However, feel free to review. :-)
REF:
https://en.wikipedia.org/wiki/Scientific_notation
http://php.net/manual/en/language.types.integer.php#language.types.integ...
Thanks!
Comment #121
Wim LeersThis would allow to reduce the # of requests also. That's why this would also address part of #2339795: Add "REST modes", just like we have "view modes" and "form modes" AND depth argument to reduce # of requests.
Comment #122
Wim LeersComment #123
Wim LeersAlso related: #2662010: Add a derivative view for each comment field providing a REST export of comments for commented entities.
Comment #124
Wim LeersThis looks extremely valuable.
Let's document why this priority is 10: which things should run before/after?
This needs to be updated. It's identical to
\Drupal\hal\Normalizer\EntityReferenceItemNormalizer
's documentation right now.Nit: s/an/a/
https://www.drupal.org/node/2350401 was opened for that, let's point to that issue URL.
Nit:
\n
in between."normalize supports"?
@covers supportsNormalization
@covers normalize
Nit: s/array()/[]/
Nit:
\n
in between.Why does this live in
rest
andCollection
lives inserialization
?Nit:
\n
in between.Need fixing
Hm… this is returning entity field data, plus URLs. Both of those things come with cacheability metadata. So this sounds like this object should implement
\Drupal\Core\Cache\CacheableDependencyInterface
.Nit: extraneous
\n
.Misplaced. (And perhaps intended to be deleted?)
Far too much
\n
.Too much
\n
.Looks like this needs to be resolved.
Needs some love.
Nit:
\n
in between.Nit:
array()
->[]
Comment #125
Wim LeersWhat is not 100% clear to me yet is whether #121 is actually correct, i.e. whether this would allow you to for example retrieve a Node and have all the Taxonomy Terms it references be embedded in the response. i.e. removing the need to request each of the Taxonomy Terms separately.
Comment #126
clemens.tolboom@win-leers in [#2100637#121] you say "This ...". What are you referring to 'this'?
AFAIK this collection will not be a compound request having a node and it's comments/terms. I used collection to fetch all comments/terms for a node but that still requires 2 requests (1 for node and 1 for comments/terms by nid).
Comment #127
Wim Leers"This" = this issue/patch.
Comment #129
tedbowAssigning to myself to re-roll and fix as much as I can from @Wim Leers' review in #124
Comment #130
tedbowOk actually just a re-roll for now.
I think some tests are going to fail but I am getting some weird time out issues when I try to run tests locally.
Comment #132
tedbowUn-assigning myself but I will look at this issue and the failing test tomorrow.
Comment #133
tedbowUpdate constructor call in SerializerTest to match updates.
Comment #136
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedTried #133 out with Drupal 8.1 as the basis for a REST plugin to expose custom entity queries as resources. Works well. Title and Description are not surfaced in the normalizer.
Comment #137
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedTook a stab at an issue summary.
Points that struck me while summarizing:
This issue facilitates functional use cases, developer learnability, and developer experience.
Comment #138
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedRegarding #36 item (19) called out in the summary, Collections should be agnostic about what kind of thing can be a member. Some facility to declare a type for collection members would help enforce consistency, but the flexibility should be there as a starting point.
In my own implementation, I've extended the Collection class as a RefinableCacheableDependency so I can bubble up entity metadata in custom REST resources. The serializers do not care.
Comment #139
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedWorking on 8.3 reroll.
Comment #140
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedSince #133 did not apply, I attempted to use the interdiff tool but it also failed. Tested the attached reroll manually and saw collections for a simple HAL JSON view.
Comment #142
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedComment #144
Grayside CreditAttribution: Grayside at Phase2 commentedUnfortunately out of time to continue exploring this right now.
The fail in core/modules/rest/tests/src/Unit/Plugin/views/style/SerializerTest.php seems to be an insufficiency in how the Views object is being mocked.
Comment #145
Grayside CreditAttribution: Grayside at Phase2 commentedMinor cleanups - removed unneeded file docbocks and an instanceof check in the CollectionNormalizer out of step with how other normalizers are set up.
Comment #146
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedComment #149
Grayside CreditAttribution: Grayside at Phase2 commentedIt was pointed out in by @dawehner in #2803413: [PP-1] REST views: add HTTP Link pager that the total count should not be executed for mini pager so there is an ability to have some pager links without the overhead of a potentially complex count query. (Biggest previous discussion on mini pager was in #119 and #120)
Changes in that case will require mini_pager logic that is more prone to providing pager links that are not needed. I tend to think a next page link which gets no further results is better than no pager link at all, I don't recall how mini pager is handled as a UI element.
Comment #150
dmouseThis patch is for 8.2.x branch, I use this in my current project, I just update if anyone needs it
Comment #151
fengyun CreditAttribution: fengyun commentedI apply the patch for v8.2.3. it does not work. instead of the error below.
The website encountered an unexpected error. Please try again later.
TypeError: Argument 3 passed to Drupal\rest\LinkManager\LinkManager::__construct() must implement interface Drupal\rest\LinkManager\CollectionLinkManagerInterface, none given, called in /var/www/cylife/web/core/lib/Drupal/Component/DependencyInjection/Container.php on line 272 in Drupal\rest\LinkManager\LinkManager->__construct() (line 38 of core/modules/rest/src/LinkManager/LinkManager.php).
Drupal\rest\LinkManager\LinkManager->__construct(Object, Object) (Line: 272)
Drupal\Component\DependencyInjection\Container->createService(Array, 'rest.link_manager') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('rest.link_manager', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'serializer.normalizer.file_entity.hal') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('serializer.normalizer.file_entity.hal', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 522)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'serializer') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('serializer', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'simple_oauth.repositories.access_token') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('simple_oauth.repositories.access_token', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'simple_oauth.server.resource_server') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('simple_oauth.server.resource_server', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'simple_oauth.authentication.simple_oauth') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('simple_oauth.authentication.simple_oauth', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 329)
Drupal\Component\DependencyInjection\Container->createService(Array, 'authentication_collector') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('authentication_collector', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'authentication') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('authentication', 1) (Line: 494)
Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 236)
Drupal\Component\DependencyInjection\Container->createService(Array, 'authentication_subscriber') (Line: 177)
Drupal\Component\DependencyInjection\Container->get('authentication_subscriber') (Line: 108)
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch('kernel.request', Object) (Line: 125)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 64)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 652)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
Comment #152
xjm@fengyun, that error is typical of applying a code change without clearing the cache or running update.php. I'd suggest testing the patch against a clean install.
The core committers and Views maintainers to make this issue a major task, since there is a lack of feature completeness for serialization here.
Comment #153
fengyun CreditAttribution: fengyun commented@xjm, yeah, by clean the cache, it works! Thanks. its seems the patch has not apply in drupal core 8.3.x-dev.
Comment #154
tedbowJust a re-roll #145 for now. Then will look at test failures.
Comment #156
Wim LeersLately I’ve been thinking we should “won’t fix” this in favor of JSON API. The recommendation would become:
I just think that it’s a LOT of effort to replicate all the collection handling that JSON API has in the crazy non-structured/non-strict world of
@RestResource
plugins: how will@RestResource
plugins even be able to define their own collections? They're currently centered on individual resources. Or would we need to add "collection" resources?The issue summary actually specifically calls out Views. But … REST Export views are not a generic collection solution: views only works for entities.
I'm not saying we MUST not do this. I'm saying that this requires a lot of work, that may be better spent elsewhere. Especially now that the JSON API module is in a usable shape, and has well-documented (because the spec defines it) methods for filtering, sorting and paging collections.
OTOH, the concept of a
Collection
value object is also something the JSON API module has. But it may be smarter to first focus on JSON API complying with the spec, and then (once we have JSON API in core) consider doing this issue. Bothrest.module
/serialization.module
andjsonapi.module
would then be able to use this commonCollection
value object concept. We would have two implementations using it, which is always better than just one: more validation, more certainty.In other words: I think it's best to mark this issue #90 (the end of 2014) anyway.
. Which it effectively has been since roughlyI'd love to hear others' opinions. Again: I'm not saying this won't happen, I'm saying this is not the biggest win right now. It's been up for the taking for years for somebody to bring to completion, and it hasn't happened yet. It seems like we can do without for a bit longer, especially if JSON API can fill this gap for the time being.
Comment #157
dawehnerI have to agree, a custom not flexible solution feels like a waste of time. On top of wasting time now we would have to support the feature in the future. If we want to do that, than I would strongly suggest to put it into a experimental module and drop it maybe after a while.
That's simply not true. Views can query everything which is needed, like for example dblog entries.
Comment #158
Wim LeersI was thinking about that just after I posted #156. You're of course right. But of course, not everybody with custom data is going to be writing views integration. Perhaps we should be recommending that, and perhaps even require that if the future "collections" support in REST depends on views.
Comment #159
dawehnerSo yeah the question is, do we somehow want to recommend people how they can integrate themselves with our REST interfaces.
Would we recommend them to somehow magically integrate themselfes into JSONAPI?
Comment #160
Grayside CreditAttribution: Grayside at Phase2 commentedI found Views too limiting but the Collection serializer very useful to facilitate uniform list structure with views output. For creating custom list resources I think this has value in core that would be very limited in contrib.
Comment #162
Wim LeersComment #163
Wim LeersFor the same reasons in #156, we should also postpone, and actually even close #2662010: Add a derivative view for each comment field providing a REST export of comments for commented entities. So, did that.
Comment #164
Wim LeersNote that since this blocks #2099281: [PP-1] REST views: pagination link relations, that issue is also affected.
This is a feature request, that has been open for years, and we just decided to postpone it even further. I think demoting to
priority makes sense.Comment #165
dawehnerI mean honestly, people can append the ?page query parameter in their client code and they probably much more likely do that than wanting to look at some page relation header/JSON.
Comment #166
Wim LeersBesides #2099281: [PP-1] REST views: pagination link relations (as mentioned in #164), this also blocks #2803413: [PP-1] REST views: add HTTP Link pager.
Comment #167
clemens.tolboomI guess the JSON API mentioned is #156 is actually RESTful Web Services API right?
The way I see it is JSON API is drupalish JSON and HAL is machine readable specific JSON (for pagination, type look up, find). We the people have to work out drupalism for values in both APIs. If we decide to support HAL we have to implement all features including collections. Otherwise move it to contrib.
Comment #168
dawehner@clemens.tolboom
JSON API is the JSON API module: http://drupal.org/project/jsonapi which implements the jsonapi spec: http://jsonapi.org/
Comment #169
clemens.tolboom@dawehner tnx ... that enlightening :-)
So JSON API is a competing against HAL+JSON https://tools.ietf.org/html/draft-kelly-json-hal-05 ... they look similar. I'm unbiased on what prevails (apart from waisted time). I'll try https://www.drupal.org/project/jsonapi.
Comment #170
dawehner@clemens.tolboom
Its more than that, as for example you can have collection of things.
Comment #171
Wim LeersWhat @dawehner said :)
Comment #177
Wim Leers2.5 years later, nothing has happened.
Since then, Drupal 8.7.0 has been shipping for 6 months with the JSON:API module which does support this. Adding this to the REST module is simply not going to happen.
Comment #178
Wim Leers