Problem/Motivation

The oembed Resource value object strictly validates that the width and height of certain resource types are able to be cast to integers greater than zero.

The issue encountered is that various oembed providers may send either one, none or both of these values in their responses. This excludes a lot of providers from being usable with the core oembed system.

Proposed resolution

Be more permissive with NULL values, allow either one, both or neither of width and height to be NULL. Developers may need to decide how these resources end up being displayed, but this is acceptable for each provider given they are required to be enabled via a hook.

Remaining tasks

Review/patch.

User interface changes

None.

API changes

  • More permissive internal Resource value object.
  • More permissive underlying oembed system with respect to a large group of providers.

Data model changes

None.

Release notes snippet

Comments

Roensby created an issue. See original summary.

Roensby’s picture

Status: Active » Fixed

In short, the cache can fail if entity has validation errors that haven't been caught when creating the entity.
Obviously, the above description fails to capture the real issue: that the filter result of the jsonapi query includes the erroneous entity, which is why the queries sometime cache and sometimes don't.
In my case, a faulty oembed url, but it could probably be any validation error.

An entity unable to cache can be a pretty big issue if it slows down every query that happens to include it. It would be nice if Drupal logged whenever an entity fails to cache.

wim leers’s picture

Issue tags: +API-First Initiative

Glad you were able to figure this out!

Although I honestly do not understand how a faulty oembed URL can cause this. Could you please elaborate? I'd love to understand :)

Roensby’s picture

Sure, I programmatically created an entity with a media field. The media entity referenced contained an oembed resource that had an illegal height dimension. Because entity validation was skipped when creating the entity, it would continously return an oembed validation error whenever loaded.

I haven't investigated what exactly happens in code, but the issue appear to have cascaded all the way through to the cache and manifested to the user as an overview of entities loading really slow via jsonapi (because the entity was part of the overview).

I make assumptions: based on what I see, entity validation errors causes the cache layer to refuse caching the entity. It could obviously be something else, but after fixing the offending oembed resource, the entity caches correct.

If my assumption is correct, it may place undue responsibility on entity validation (especially for external resources such as oembeds) with regards to the integrity of the cache layer.

wim leers’s picture

Title: Caching issue when filtering on referenced uuid » Caching issue when referencing an oEmbed resource with an illegal height dimension
Assigned: Unassigned » phenaproxima
Status: Fixed » Active

Because entity validation was skipped when creating the entity, it would continously return an oembed validation error whenever loaded.

Ahhh! Now I understand :)

based on what I see, entity validation errors causes the cache layer to refuse caching the entity. It could obviously be something else, but after fixing the offending oembed resource, the entity caches correct.

No entity validation occurs during entity rendering. But it's possible the the Media module's oEmbed support does do some validation during rendering, to ensure it's actually possible to render correctly. This is what I suspect is going on. @phenaproxima was closely involved with Drupal 8's Media oEmbed support, so assigning to him for feedback.

wim leers’s picture

Issue tags: +Media Initiative
Roensby’s picture

Issue summary: View changes

Rewritten to reflect the actual issue.

phenaproxima’s picture

I imagine that could absolutely be what's happening. When rendering media, it's very likely that the resource fetcher service will be invoked (it is used in order to get metadata which may well be used during rendering). The resource fetcher will load raw resource data from the persistent cache, then send that into one of the static factory methods of the Resource class. This is where the exception might happen, since the Resource class does a lot of validation on instantiation.

I think, looking at the code, Resource::setDimensions() is being a bit overzealous. For certain resource types, like photos or videos, requiring a width and height absolutely makes sense. But for things like rich content -- i.e., tweets -- or links, it doesn't. Resource::rich() is validating the dimensions of rich content (video, which is considered a subset of rich content, gets this as well), so your otherwise valid resource is failing.

My suggestion would probably be to modify the setDimensions() method so that it accepts a null value for $height, and doesn't bother validating $height if it's null. We could also separate the width and height setters into different methods, and have the ::rich() and ::video() factory methods call them separately as needed.

Roensby’s picture

StatusFileSize
new950 bytes

My suggestion would probably be to modify the setDimensions() method so that it accepts a null value for $height, and doesn't bother validating $height if it's null.

I'm a fan of this option, and suggest 0 instead of null values. Attached patch provides sample implementation.

We could also separate the width and height setters into different methods, and have the ::rich() and ::video() factory methods call them separately as needed.

The problem is that any oembed provider could one day decide to not live up to the standard, which would result in my entities not getting cached - my point being that my website speed would be at the mercy of 3rd party oembed providers.

wim leers’s picture

Version: 8.7.5 » 8.7.x-dev
Component: jsonapi.module » media system
Category: Support request » Bug report
Issue tags: +Needs title update, +Needs issue summary update

Per #8.

phenaproxima’s picture

Assigned: phenaproxima » Unassigned
Issue tags: +oembed

Un-assigning from myself, and tagging as an oEmbed issue.

xjm’s picture

Priority: Normal » Major
Status: Active » Needs work
Issue tags: -Needs title update +Needs tests

Am I reading correctly that this could cause an exception to be thrown under normal site operation for affected media? (In addition to any performance implications.) If so then that probably qualifies as a major bug.

xjm’s picture

Issue tags: +Needs title update

Didn't mean to remove a tag.

Roensby’s picture

@xjm I don't think so. Although an exception is thrown, it is not unhandled. In Drupal, you will just get a themed error message that the entity failed validation (whenever the entity is loaded).

To the best of my understanding, this post deals exclusively with the issue of validation that happens post entity creation. And how that is a problem for decoupled systems that rely on Drupal cache.

wim leers’s picture

Title: Caching issue when referencing an oEmbed resource with an illegal height dimension » Loading Media entities using the @MediaSource=oembed may trigger PHP exceptions if the referenced oEmbed URL does not comply with the spec
Issue tags: -Needs title update
sam152’s picture

I've got a similar issue with embedded Facebook videos. The height is always NULL, and thus embeds can't be used anywhere.

By the sounds of it, there are two issues being discussed though. The overzealous exception and what should happen up the callstack when such exceptions are encountered. It might be worth opening up a new issue to look specifically at what JSONAPI does in these circumstances.

sam152’s picture

Title: Loading Media entities using the @MediaSource=oembed may trigger PHP exceptions if the referenced oEmbed URL does not comply with the spec » The oembed Resource value object should be more permissive for NULL dimensions
Version: 8.7.x-dev » 8.9.x-dev
Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -API-First Initiative, -Needs issue summary update, -Needs tests
StatusFileSize
new4.05 KB

I've read carefully over #8 and #9, here is how I think we should proceed.

NULL is already a hinted return type of both ::getWidth and ::getHeight and @Roensby is correct, providers will likely have various capability to send either one, both or neither of these values. Lets make the foundation of the handling permissive with respect to these dimensions, for all resource types. In my case, despite requiring some special consideration for Facebook embeds during display, the underlying system rejecting the NULL value made the oembed system unusable.

I think it's acceptable for developers to make specific decisions and considerations towards the display of non-conforming resources, especially given these resources need to be explicitly enabled with a hook.

With that in mind, here is the test, IS/title update.

@Wim Leers, @Roensby, do you think JSON API needs a dedicated issue to discuss the exception handling component? I think this is probably not an API first initiative issue with the modified scope.

phenaproxima’s picture

Could we get a FAIL patch here, to prove the bug and the fix?

phenaproxima’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/media/src/OEmbed/Resource.php
    @@ -510,25 +510,20 @@ protected function setThumbnailDimensions($width, $height) {
    -    $width = (int) $width;
    -    $height = (int) $height;
    -
    -    if ($width > 0 && $height > 0) {
    -      $this->width = $width;
    -      $this->height = $height;
    -    }
    -    else {
    -      throw new \InvalidArgumentException('The dimensions must be numbers greater than zero.');
    +    if (($width !== NULL && (int) $width <= 0) || ($height !== NULL && (int) $height <= 0)) {
    +      throw new \InvalidArgumentException('The dimensions must be NULL or numbers greater than zero.');
         }
    +    $this->width = $width !== NULL ? (int) $width : NULL;
    +    $this->height = $height !== NULL ? (int) $height : NULL;
    

    I find this logic a bit hard to follow. Rather than doing $var !== NULL, we can use isset() instead? Also, I'm not sure we need to cast $width and $height to ints when using comparison operators; I'm pretty sure PHP will cast them automatically. So this can be:

    if ((isset($width) && $width <= 0) || (isset($height) && $height <= 0)) {
      throw;
    }
    $this->width = isset($width) ? (int) $width : NULL;
    $this->height = isset($height) ? (int) $height : NULL;
    
  2. +++ b/core/modules/media/tests/src/Unit/ResourceTest.php
    @@ -0,0 +1,87 @@
    +  /**
    +   * Test cases for ::testSetDimensions.
    +   */
    +  public function setDimensionsTestCases() {
    +    return [
    +      'Standard rich dimensions' => [
    +        'rich',
    +        5,
    +        10,
    +      ],
    +      'Negative width and height' => [
    +        'rich',
    +        -5,
    +        -10,
    +        'The dimensions must be NULL or numbers greater than zero.',
    +      ],
    +      'Zero width' => [
    +        'rich',
    +        0,
    +        5,
    +        'The dimensions must be NULL or numbers greater than zero.',
    +      ],
    +      'NULL width' => [
    +        'rich',
    +        NULL,
    +        10,
    +      ],
    +      'NULL height' => [
    +        'rich',
    +        NULL,
    +        10,
    +      ],
    +      'NULL width and height' => [
    +        'rich',
    +        NULL,
    +        NULL,
    +      ],
    +      'Cast numeric dimenions' => [
    +        'rich',
    +        "1",
    +        "45",
    +        NULL,
    +        1,
    +        45,
    +      ],
    +      'Cast invalid zero value' => [
    +        'rich',
    +        "0",
    +        10,
    +        'The dimensions must be NULL or numbers greater than zero.',
    +      ],
    +      'Cast negative value' => [
    +        'rich',
    +        "-10",
    +        10,
    +        'The dimensions must be NULL or numbers greater than zero.',
    +      ],
    +    ];
    +  }
    +
    +  /**
    +   * @covers ::setDimensions
    +   * @dataProvider setDimensionsTestCases
    +   */
    +  public function testSetDimensions($factory, $width, $height, $exception = NULL, $expected_width = NULL, $expected_height = NULL) {
    +    if ($exception) {
    +      $this->setExpectedException(\InvalidArgumentException::class, $exception);
    +    }
    +    $resource = Resource::$factory('foo', $width, $height);
    +    $this->assertEquals($expected_width ?: $width, $resource->getWidth());
    +    $this->assertEquals($expected_height ?: $height, $resource->getHeight());
    +  }
    

    Oh my God, it's gorgeous!!! 😍 But, since this patch does some type juggling, can we use assertSame() instead of assertEquals()?

sam152’s picture

Status: Needs work » Needs review
StatusFileSize
new2.01 KB
new2.27 KB
new4.06 KB

1. Sure thing, great improvements.
2. Great point, fixed.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

I love this. RTBC when green on all backends. Thank you for the great bugfix!

phenaproxima’s picture

Status: Reviewed & tested by the community » Needs review

Actually, one thing that concerns me...the photo resource type really requires dimensions. Should we perhaps add something to the Resource::photo() factory method, and related test coverage, to ensure that both width and height have a value (and throw exceptions if either doesn't)? Something like this:

function photo ($url, $width, $height...) {
  // Do some stuff

  // isset() is variadic and only returns TRUE if all arguments are set
  if (empty($width) || empty($height)) {
    throw new \InvalidArgumentException();
  }

  // More stuff
}

Also, the rich() factory method currently doesn't explicitly allow NULL values for $width and $height. Should we make that change here too?

sam152’s picture

If the system is really unusable for photos without dimensions, I definitely agree. Otherwise I think it would probably be a matter of time before, in order to support $some_provider, the constraints would simply be relaxed once more.

The last submitted patch, 20: 3071682-20--test-only.patch, failed testing. View results

sam152’s picture

StatusFileSize
new1.33 KB
new5.39 KB

Here is a test to prove that photos render without dimensions, the hook_theme defaults them to NULL and template_preprocess_image optionally includes them as attributes.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

acbramley’s picture

Status: Needs review » Reviewed & tested by the community

Nice work @Sam152 looks like #22 is addressed

alexpott’s picture

Version: 9.1.x-dev » 9.0.x-dev
Status: Reviewed & tested by the community » Patch (to be ported)

Committed f85f69c and pushed to 9.1.x. Thanks!

I think this is a good example of the robustness principle.

Going to get a +1 from release managers for a backport to 8.9.x and 9.0.x as they are in rc.

diff --git a/core/modules/media/tests/src/Unit/ResourceTest.php b/core/modules/media/tests/src/Unit/ResourceTest.php
index 6b398cfda7..5043565cd3 100644
--- a/core/modules/media/tests/src/Unit/ResourceTest.php
+++ b/core/modules/media/tests/src/Unit/ResourceTest.php
@@ -48,7 +48,7 @@ public function setDimensionsTestCases() {
         NULL,
         NULL,
       ],
-      'Cast numeric dimenions' => [
+      'Cast numeric dimensions' => [
         'rich',
         "1",
         "45",

Fixed spelling mistake on commit.

  • alexpott committed f85f69c on 9.1.x
    Issue #3071682 by Sam152, Roensby, phenaproxima, Wim Leers: The oembed...

  • alexpott committed 9e3592a on 9.0.x
    Issue #3071682 by Sam152, Roensby, phenaproxima, Wim Leers: The oembed...

  • alexpott committed 46b22b3 on 8.9.x
    Issue #3071682 by Sam152, Roensby, phenaproxima, Wim Leers: The oembed...
alexpott’s picture

Version: 9.0.x-dev » 8.9.x-dev
Status: Patch (to be ported) » Fixed

Discussed with @catch and we agreed to backport.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.