Problem/Motivation
The oembed Resource value object strictly validates that the width and height of certain resource types are able to be cast to integers greater than zero.
The issue encountered is that various oembed providers may send either one, none or both of these values in their responses. This excludes a lot of providers from being usable with the core oembed system.
Proposed resolution
Be more permissive with NULL values, allow either one, both or neither of width and height to be NULL. Developers may need to decide how these resources end up being displayed, but this is acceptable for each provider given they are required to be enabled via a hook.
Remaining tasks
Review/patch.
User interface changes
None.
API changes
- More permissive internal
Resource value object.
- More permissive underlying oembed system with respect to a large group of providers.
Data model changes
None.
Release notes snippet
Comments
Comment #2
Roensby commentedIn short, the cache can fail if entity has validation errors that haven't been caught when creating the entity.
Obviously, the above description fails to capture the real issue: that the filter result of the jsonapi query includes the erroneous entity, which is why the queries sometime cache and sometimes don't.
In my case, a faulty oembed url, but it could probably be any validation error.
An entity unable to cache can be a pretty big issue if it slows down every query that happens to include it. It would be nice if Drupal logged whenever an entity fails to cache.
Comment #3
wim leersGlad you were able to figure this out!
Although I honestly do not understand how a faulty oembed URL can cause this. Could you please elaborate? I'd love to understand :)
Comment #4
Roensby commentedSure, I programmatically created an entity with a media field. The media entity referenced contained an oembed resource that had an illegal height dimension. Because entity validation was skipped when creating the entity, it would continously return an oembed validation error whenever loaded.
I haven't investigated what exactly happens in code, but the issue appear to have cascaded all the way through to the cache and manifested to the user as an overview of entities loading really slow via jsonapi (because the entity was part of the overview).
I make assumptions: based on what I see, entity validation errors causes the cache layer to refuse caching the entity. It could obviously be something else, but after fixing the offending oembed resource, the entity caches correct.
If my assumption is correct, it may place undue responsibility on entity validation (especially for external resources such as oembeds) with regards to the integrity of the cache layer.
Comment #5
wim leersAhhh! Now I understand :)
No entity validation occurs during entity rendering. But it's possible the the Media module's oEmbed support does do some validation during rendering, to ensure it's actually possible to render correctly. This is what I suspect is going on. @phenaproxima was closely involved with Drupal 8's Media oEmbed support, so assigning to him for feedback.
Comment #6
wim leersComment #7
Roensby commentedRewritten to reflect the actual issue.
Comment #8
phenaproximaI imagine that could absolutely be what's happening. When rendering media, it's very likely that the resource fetcher service will be invoked (it is used in order to get metadata which may well be used during rendering). The resource fetcher will load raw resource data from the persistent cache, then send that into one of the static factory methods of the Resource class. This is where the exception might happen, since the Resource class does a lot of validation on instantiation.
I think, looking at the code, Resource::setDimensions() is being a bit overzealous. For certain resource types, like photos or videos, requiring a width and height absolutely makes sense. But for things like rich content -- i.e., tweets -- or links, it doesn't. Resource::rich() is validating the dimensions of rich content (video, which is considered a subset of rich content, gets this as well), so your otherwise valid resource is failing.
My suggestion would probably be to modify the setDimensions() method so that it accepts a null value for $height, and doesn't bother validating $height if it's null. We could also separate the width and height setters into different methods, and have the ::rich() and ::video() factory methods call them separately as needed.
Comment #9
Roensby commentedI'm a fan of this option, and suggest 0 instead of null values. Attached patch provides sample implementation.
The problem is that any oembed provider could one day decide to not live up to the standard, which would result in my entities not getting cached - my point being that my website speed would be at the mercy of 3rd party oembed providers.
Comment #10
wim leersPer #8.
Comment #11
phenaproximaUn-assigning from myself, and tagging as an oEmbed issue.
Comment #12
xjmAm I reading correctly that this could cause an exception to be thrown under normal site operation for affected media? (In addition to any performance implications.) If so then that probably qualifies as a major bug.
Comment #13
xjmDidn't mean to remove a tag.
Comment #14
Roensby commented@xjm I don't think so. Although an exception is thrown, it is not unhandled. In Drupal, you will just get a themed error message that the entity failed validation (whenever the entity is loaded).
To the best of my understanding, this post deals exclusively with the issue of validation that happens post entity creation. And how that is a problem for decoupled systems that rely on Drupal cache.
Comment #15
wim leersComment #16
sam152 commentedI've got a similar issue with embedded Facebook videos. The height is always NULL, and thus embeds can't be used anywhere.
By the sounds of it, there are two issues being discussed though. The overzealous exception and what should happen up the callstack when such exceptions are encountered. It might be worth opening up a new issue to look specifically at what JSONAPI does in these circumstances.
Comment #17
sam152 commentedI've read carefully over #8 and #9, here is how I think we should proceed.
NULL is already a hinted return type of both
::getWidthand::getHeightand @Roensby is correct, providers will likely have various capability to send either one, both or neither of these values. Lets make the foundation of the handling permissive with respect to these dimensions, for all resource types. In my case, despite requiring some special consideration for Facebook embeds during display, the underlying system rejecting the NULL value made the oembed system unusable.I think it's acceptable for developers to make specific decisions and considerations towards the display of non-conforming resources, especially given these resources need to be explicitly enabled with a hook.
With that in mind, here is the test, IS/title update.
@Wim Leers, @Roensby, do you think JSON API needs a dedicated issue to discuss the exception handling component? I think this is probably not an API first initiative issue with the modified scope.
Comment #18
phenaproximaCould we get a FAIL patch here, to prove the bug and the fix?
Comment #19
phenaproximaI find this logic a bit hard to follow. Rather than doing
$var !== NULL, we can use isset() instead? Also, I'm not sure we need to cast $width and $height to ints when using comparison operators; I'm pretty sure PHP will cast them automatically. So this can be:Oh my God, it's gorgeous!!! 😍 But, since this patch does some type juggling, can we use assertSame() instead of assertEquals()?
Comment #20
sam152 commented1. Sure thing, great improvements.
2. Great point, fixed.
Comment #21
phenaproximaI love this. RTBC when green on all backends. Thank you for the great bugfix!
Comment #22
phenaproximaActually, one thing that concerns me...the
photoresource type really requires dimensions. Should we perhaps add something to theResource::photo()factory method, and related test coverage, to ensure that both width and height have a value (and throw exceptions if either doesn't)? Something like this:Also, the
rich()factory method currently doesn't explicitly allow NULL values for $width and $height. Should we make that change here too?Comment #23
sam152 commentedIf the system is really unusable for photos without dimensions, I definitely agree. Otherwise I think it would probably be a matter of time before, in order to support
$some_provider, the constraints would simply be relaxed once more.Comment #25
sam152 commentedHere is a test to prove that photos render without dimensions, the
hook_themedefaults them toNULLandtemplate_preprocess_imageoptionally includes them as attributes.Comment #27
acbramley commentedNice work @Sam152 looks like #22 is addressed
Comment #28
alexpottCommitted f85f69c and pushed to 9.1.x. Thanks!
I think this is a good example of the robustness principle.
Going to get a +1 from release managers for a backport to 8.9.x and 9.0.x as they are in rc.
Fixed spelling mistake on commit.
Comment #32
alexpottDiscussed with @catch and we agreed to backport.