Problem/Motivation

If an oEmbed remote URL cannot be retrieved, a vague "Could not retrieve the remote URL" message is logged, with no mention of the underlying exception, which makes troubleshooting difficult.

Steps to reproduce

  1. Use Media module to display remote media (e.g. YouTube videos)
  2. Prevent the server from accessing the remote media URL, e.g. via firewall rule, dysfunctional DNS server, etc.
  3. A vague "Could not retrieve the remote URL" error will be logged.

Proposed resolution

Add the underlying exception to the "Could not retrieve the remote URL" error message.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#6 3402540-6.patch980 bytesmfb
#2 3402540.patch910 bytesmfb

Issue fork drupal-3402540

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

dadderley created an issue. See original summary.

mfb’s picture

Status: Active » Needs review
StatusFileSize
new910 bytes

It seems like there isn't enough error logging here to really be able to debug what's going on.

Here's a patch to improve logging somewhat (although, maybe still not enough, depending what the exception message is :)

needs-review-queue-bot’s picture

Status: Needs review » Needs work

The Needs Review Queue Bot tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request. Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

dadderley’s picture

Thanks mfb, have applied the patch.
Waiting for new errors.

dadderley’s picture

Further update.

The patch adds to the error message.
Could not retrieve the remote URL (https://youtu.be/ZwEYpptNyek): Could not retrieve the oEmbed resource.

In a browser where I am not logged in I go the affected page with the embedded video.
Of course the video does not show.

I visit the same page in as a logged in user.
The video shows.

I flush the caches.

I return to the page In a browser where I am not logged in.
The video shows now whereas it did not show before.

Interesting caching issue looks like.

I do not drupal core caching enabled.
I have advaag disabled.

I am using redis and I am using cloudflare.

Any ideas?

mfb’s picture

StatusFileSize
new980 bytes

There's hopefully another exception inside with a better error message? Here's another patch.

Contrib loggers that can show nested exceptions make debugging a lot easier than drupal core on its own.

dadderley’s picture

Applied the patch.

The error message now reads.
Could not retrieve the remote URL (https://youtu.be/Fw6Rz2GAxc0): cURL error 28: Resolving timed out after 5000 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://www.youtube.com/oembed?url=https://youtu.be/Fw6Rz2GAxc0&maxwidth=560&maxheight=315
So this looks helpful.

Are there any quick fixes for this?
Thanks in advance.

As a non coder, this whole thing is pure voodoo.
As this seems to happen mostly for anon users and authenticated users see the video.

mfb’s picture

Status: Needs work » Needs review

From here, it'll be up to you to figure out by talking to whoever is responsible for the network where your site is hosted. You now have an error you can show them - it should not take over 5000 milliseconds to resolve www.youtube.com - maybe the DNS server you are using is over-saturated, or caching needs to be enabled, or something like that.

This logging change might be worthwhile for other folks who are debugging this component, so I'll set the patch to "needs review"

smustgrave’s picture

Category: Bug report » Support request

This seems more like a support request ticket

dadderley’s picture

Thanks for the good advice.
I talked to the server admin and he seems to have fixed this issue.
This is his explanation of what was going on.

so the issue on that video thing was that the DNS resolution _on the server_ was broken.

Our servers sometimes need to resolve DNS like anything else, and we had them pointed to Cloudflare public DNS (1.1.1.1) as well as Google Public DNS (8.8.8.8).

When the nodes are being saved the server is reaching out to Youtube to fetch some info, using cURL, and that is the error that we were seeing, that CURL call failing because DNS would not resolve for the youtube name.

Coincidentally (or not), Cloudflare.com was down last week due to DDoS attack (south Sudanese hackers claimed responsibility).

Both Cloudflare and Google are under constant DDoS attack (see https://www.zdnet.com/article/google-cloud-aws-and-cloudflare-report-lar...) .

...

This is the 2nd time we have received direct reports of the cloudflare DNS flaking out, so we have switched to another DNS provider.

We are in Canada so are using the CIRA CAnadian Shield DNS.

To fix this problem, replace the DNS on your server (/etc/resolv.conf) with a more reliable alternative (one from your ISP, or a local caching DNS server, etc). Unreliable DNS is the cause.

mfb’s picture

@smustgrave what do you think about the patch - this is rather difficult to debug unless the logging is improved.

smustgrave’s picture

So this could be repurposed with an updated title and issue summary.

New ticket opened up and this one is postponed on that.

Either or.

mfb’s picture

Title: Seemingly random 'Could not retrieve the remote URL' errors » 'Could not retrieve the remote URL' error message is vague
Issue summary: View changes
smustgrave’s picture

Version: 10.1.x-dev » 11.x-dev
Category: Support request » Feature request
Status: Needs review » Reviewed & tested by the community

So revisting this and seems the request is to add additional logging information for possible future debugging or issues. On that idea +1 from me.

sachinsnimbalkar@gmail.com’s picture

I have created the twig file based on my content type template/field/field--node--field-article-video--article.html.twig
with following code and
{#
/**
* @file
* Default template for a field.
*/
#}

{% if gallery_video_thumbURL %}

{% else %}

{% for item in items %}
{{ item.content }}
{% endfor %}

{% endif %}

In my theme file created function
function extractYouTubeVideoID($url) {
// Regular expression pattern to match YouTube video IDs
$pattern = '/(?<=watch\?v=|\/videos\/|embed\/|youtu.be\/|\/v\/|\/e\/|watch\?v%3D|\?v%3D)([\w-]{11})/i';

// Perform the regular expression match
preg_match($pattern, $url, $matches);

// Check if the video ID is found and return it, or return null if not found
return isset($matches[1]) ? $matches[1] : null;
}
function crystalrun_preprocess_field(&$variables) {
$entity = \Drupal::routeMatch()->getParameter('node');

if ($entity instanceof \Drupal\node\NodeInterface) { // Check if it is a node
$nid = $entity->id(); // You also have the node object right here$

$node = \Drupal\node\Entity\Node::load($nid);
$content_type= $node->getType();
/**********/
if ($content_type == "article"){
// check the field value exits or not.
$url = $node->field_article_video->getValue();
if (!empty($url))
{
$field_value = $node->field_article_video->entity->field_media_oembed_video->getValue()[0]['value'];
$videoID = extractYouTubeVideoID($field_value);
$variables['gallery_video_thumbURL'] = 'https://www.youtube.com/embed/'.$videoID ? 'https://www.youtube.com/embed/'.$videoID :'' ;
}else
{
$variables['gallery_video_thumbURL']="";
}
}
/*********/
}

Its absolutely working fine for me, please share you feedback and comment on the resolution, can be used with other contenttype simillary be creating twig file and handling the logic.

quietone’s picture

Status: Reviewed & tested by the community » Needs work

Can we have some tests of this new message?

mfb’s picture

Status: Needs work » Needs review

I'm not super familiar with media module, so not sure this is the best place for it, but here is a hopefully a working test

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Restoring status as test coverage is there https://git.drupalcode.org/issue/drupal-3402540/-/jobs/555070

longwave’s picture

Status: Reviewed & tested by the community » Needs work

Made a suggestion to improve the test.

mfb’s picture

Status: Needs work » Needs review
smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Feedback appears to be addressed

longwave’s picture

Status: Reviewed & tested by the community » Fixed

Not eligible for backport as it changes a user-facing string.

Committed 8f38fc7 and pushed to 11.x. Thanks!

  • longwave committed 8f38fc7e on 11.x
    Issue #3402540 by mfb, dadderley, smustgrave, longwave: 'Could not...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.