Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
Some examples of random test failure by MediaStandardProfileTest::testMediaSources
Test the standard profile configuration for media type 'remote_video'.
https://www.drupal.org/pift-ci-job/1250132
https://www.drupal.org/pift-ci-job/1210218
There was 1 failure:
1) Drupal\Tests\media\FunctionalJavascript\MediaStandardProfileTest::testMediaSources
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Drupal Rap Video - Schipulcon09'
+'example_2.jpeg'
/var/www/html/core/modules/media/tests/src/FunctionalJavascript/MediaStandardProfileTest.php:400
/var/www/html/core/modules/media/tests/src/FunctionalJavascript/MediaStandardProfileTest.php:80
Test the standard profile configuration for media type 'image'.
https://www.drupal.org/pift-ci-job/1209481
There was 1 failure:
1) Drupal\Tests\media\FunctionalJavascript\MediaStandardProfileTest::testMediaSources
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'example_2.jpeg'
+'example_1.jpeg'
/var/www/html/core/modules/media/tests/src/FunctionalJavascript/MediaStandardProfileTest.php:249
/var/www/html/core/modules/media/tests/src/FunctionalJavascript/MediaStandardProfileTest.php:79
Proposed resolution
Add waitForLink($locator, $timeout = 10000)
after pressButton('Save').
$page->pressButton('Save');
+ $result = $assert_session->waitForLink('Add media');
+ $this->assertNotEmpty($result);
+ $assert_session->addressEquals('admin/content/media');
Remaining tasks
None
Comments
Comment #2
Krzysztof Domański-- edited --
Comment #4
Krzysztof DomańskiComment #5
Krzysztof Domański1. Ignore previous patches. Another cause of the random test failure.
2. Looks like problem with
pressButton('Save')
. We should wait to make sure the media will be added.3. See also:
#2936432: Review use of pressButton() in functional javascript tests in the Media module
#2934997: Intermittent failure in MediaUiJavascriptTest
Comment #7
Krzysztof DomańskiComment #8
Krzysztof Domański#3059022: If Vimeo is down our tests break should solve the problem when Vimeo is down.
However, this may not be enough. See https://www.drupal.org/pift-ci-job/1209481. This is "Test the standard profile configuration for media type 'image'". The test failure is not related to vimeo.
Comment #16
alexpottThis is still happening today.
Comment #17
alexpottI think the problem here is that we're going to the next page prior to checking that the save has finished.
Comment #19
alexpottMissed a place where we are pressing a button... and then not checking that it is done...
Comment #21
alexpottComment #23
alexpottSo here's an image of what is happening when the issue fails...
https://dispatcher.drupalci.org/job/drupal_patches/152779/artifact/jenki...
We get an error "The provided URL does not represent a valid oEmbed resource".
Comment #24
andypostProbably that's a cause of error when fetching resource, maybe it's caused by rate limit or quota limits?
https://developers.google.com/youtube/v3/determine_quota_cost
Comment #25
andypostI can't find reference to rate limits but there's https://developers.google.com/youtube/v3/getting-started#quota
Comment #26
alexpott@andypost we mock all of the requests... I can run this test successfully with no internet. Also that wouldn't explain how this is random - it would start failing at some point during the day and then stop failing.
Comment #27
alexpottComment #28
alexpottComment #30
alexpottComment #34
alexpottSo the request to
media_test_oembed/resource
is failing for some reason!Comment #35
alexpottHere's the exception we're getting from #30 - https://dispatcher.drupalci.org/job/drupal_patches/152811/artifact/jenki...
Hopefully #34 will give us more info.
Comment #37
alexpottSo https://dispatcher.drupalci.org/job/drupal_patches/152812/artifact/jenki... is very interesting...
A local DNS resolution is taking move than 5000ms! I think something is wrong with the container and DNS.
Comment #38
alexpottHere's an even easier to read version of the exception... https://dispatcher.drupalci.org/job/drupal_patches/152812/artifact/jenki...
Comment #39
alexpottLet's try increasing the timeout.
Comment #41
alexpottComment #43
alexpottComment #48
alexpottI've made a commit to 10.1.x through to 9.4.x to reduce JS test concurrency based on #41.
Comment #49
alexpottComment #50
alexpottComment #51
alexpottComment #53
alexpottComment #55
alexpottComment #57
alexpottComment #58
alexpottSo #57 is a good fix for requests back to the PHP container - but we still have the chromedriver container to contend with. The other random fails are to do with slowness there and given the issue identified here it might well be DNS too.
Comment #59
alexpottLet's see if resolv.conf gives us some clues.
Comment #60
alexpottComment #61
alexpottLet's see if we can fix resolv.conf
The whole problem is here:
The line
search us-west-2.compute.internal
causes DNS lookups to append .us-west-2.compute.internal to every lookup we do... so before we lookupphp-apache-jenkins-drupal-patches-152812
we lookupphp-apache-jenkins-drupal-patches-152812.search us-west-2.compute.internal
- the docker nameserver will fall back to amazon's at this point and then eventually we will get rate limited.We need to prevent this from happening on all the containers but let's start with the PHP container.
Comment #62
alexpottOkay so let's have concurrency 20... with a fixed resolv.conf and without....
Comment #64
alexpottLet's test that the resolv.conf fix does fix the original random error.
Comment #66
alexpottI've created #3316016: DNS lookups on containers can be throttled to fix this on the containers properly but here's a fix for the PHP container... I don't think it is early enough to help the chromedriver container so I still expect this to cause the occasional random fail during heavy JS testing.
And committing the patch here will certainly do no harm. But we need to ensure that this change doesn't break any of the other test types. I'm pretty sure it won't but you never know.
Comment #69
alexpottOh maybe we have to restart networking... interesting.
Comment #70
alexpottComment #72
andypostbtw in new releases of Debian this file is generated and symlinked so edits are not saved
lrwxrwxrwx 1 root root 39 сен 30 17:42 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
Comment #73
alexpott@andypost yeah I was wondering about that... obviously I am managing to change it... but does something come along and revert it.
Comment #75
alexpottLet's see something...
Comment #80
Wim Leers#61 & #66: 🤯
Comment #81
alexpottAs per #75 this has been fixed by #3316016: DNS lookups on containers can be throttled
Comment #83
quietone CreditAttribution: quietone at PreviousNext commented