Problem/Motivation

In some cases, tests are wrong because similar links exists.

Example with Tour module, in TourHelpPageTest::verifyHelp(), assertNoLink() on 'Translation' will return 'Interface Translation' and make assert result FALSE, even if the 'Translation' link is not here.

In WebAssert::linkExists() and WebAssert::linkNotExists(), we should call $this->session->getPage()->findAll(); with 'named_exact' instead of ''named'.
This can be done by default for linkNotExists(), but i think some tests with linkExists() expect to search on approximative label.

Proposed resolution

  • Replace 'named' by 'named_exact' in WebAssert::linkNotExists().
  • Add option to AssertLegacyTrait::assertLink() and WebAssert::linkExists() to filter on 'named_exact' in place of 'named'.

Comments

GoZ created an issue. See original summary.

goz’s picture

klausi’s picture

I think we should use named_exact in our WebAssert because we really want to check that the link name is there standalone, not as part of some other link.

goz’s picture

I think we should use named_exact in our WebAssert because we really want to check that the link name is there standalone, not as part of some other link.

Even for AssertLegacyTrait::assertLink() and WebAssert::linkExists() ?
Previously trying this made some existing tests fail. See #2767275-16: Convert web tests to browser tests for tour module

goz’s picture

Issue tags: +DevDaysSeville
klausi’s picture

Yes, let's try to use named_exact for both and see where the fails are. From that issue I only see that tour tests were failing which are not converted yet anyway.

goz’s picture

Status: Active » Needs review
StatusFileSize
new1.51 KB

Status: Needs review » Needs work

The last submitted patch, 7: add_exact_option_to-2860175-7.patch, failed testing.

goz’s picture

Status: Needs work » Needs review
StatusFileSize
new2.17 KB
new1.31 KB

Let tests define if we want named_exact or named.

klausi’s picture

I looked into the test fails from #7 and the problem is for example that a link Manage fields <span class="visually-hidden">(active tab)</span> of course does not match with named_exact when only "Manage fields" is passed in.

That makes me think that our current behavior of linkExists() is correct. There might be other parts in a link label (icons, images, hidden spans for accessibility) that are not relevant for testing. So just using the "named" selector which does partial matches seems fine to me.

So we could just change the behavior of linkNotExists() to named_exact, but that also feels wrong. It really should work the same as linkExists() except of being reversed.

dawehner’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests
+++ b/core/tests/Drupal/Tests/WebAssert.php
@@ -221,13 +221,22 @@ public function titleEquals($expected_title) {
-  public function linkExists($label, $index = 0, $message = '') {
+  public function linkExists($label, $index = 0, $message = '', $exact = FALSE) {

I would make exact the default, its always better to be strict than less strict. If you don't need the strict behaviour then you'll see it in your test.
In other others explicit > implicit. I believe this is exactly what @klausi wrote before.

goz’s picture

Making exact default will break existing tests (see #7). Are we ok with that ?

dawehner’s picture

Making exact default will break existing tests (see #7). Are we ok with that ?

Mh, that's tricky, I guess we cannot risk that now.

Another idea I just had, have a method linkExistsExact to make it a bit easier to read. Any thoughts/opinions about that?

mile23’s picture

The dockblock says: "Passes if a link with the specified label is found." This implies the exact label.

+++ b/core/tests/Drupal/Tests/WebAssert.php
@@ -221,13 +221,22 @@ public function titleEquals($expected_title) {
+  public function linkExists($label, $index = 0, $message = '', $exact = FALSE) {

How about we reverse this? $exact should default to TRUE, since the expectation is that we're looking for the exact label.

Then for special-case failing legacy cases, we should add FALSE to the assertion with a todo/follow-up.

goz’s picture

Status: Needs work » Needs review
StatusFileSize
new2.86 KB

How about we reverse this? $exact should default to TRUE, since the expectation is that we're looking for the exact label.

Then for special-case failing legacy cases, we should add FALSE to the assertion with a todo/follow-up.

With default to TRUE, we need to change a lot of tests, which is not the purpose to of #2767275: Convert web tests to browser tests for tour module. Seems we are agree with dawehner in #12 and #13 to avoid that.

The dockblock says: "Passes if a link with the specified label is found." This implies the exact label.

Dockblock can be fixed to match with new argument.

Another idea I just had, have a method linkExistsExact to make it a bit easier to read. Any thoughts/opinions about that?

It's an option.

So we could just change the behavior of linkNotExists() to named_exact, but that also feels wrong. It really should work the same as linkExists() except of being reversed.

To also take care of klausi comment #10, may be we should do the same for linkNotExists.

Let's make a patch with this one, so we move a step forward.

dawehner’s picture

I like it "obviously". Do you mind adding some test coverage for those?

goz’s picture

Status: Needs review » Needs work

Yes, i'll add tests

goz’s picture

Status: Needs work » Needs review
StatusFileSize
new4.96 KB
new2.02 KB

And here are tests

dawehner’s picture

Issue tags: -Needs tests
+++ b/core/tests/Drupal/FunctionalTests/BrowserTestBaseTest.php
@@ -138,6 +138,58 @@ public function testPipeCharInLocator() {
+    try {
+      $this->assertSession()->linkExistsExact('foo|bar');
+      $this->fail('The exactly "foor|bar" label link was found.');
+    }
+    catch (ExpectationException $e) {
+      $this->pass('testLinkNonExistsExact correctly failed. The exactly "foor|bar" label link was not found.');
+    }
...
+    try {
+      $this->assertSession()->linkNotExistsExact('foo|bar|baz');
+      $this->fail('The exactly "foo|bar|baz" label link was not found.');
+    }
+    catch (ExpectationException $e) {
+      $this->pass('testNotLinkNonExistsExact correctly failed. The exactly "foo|bar|baz" label link was found.');

You can use setExpectedException here. Any reason why not?

goz’s picture

StatusFileSize
new1.73 KB
new4.64 KB

No reasons except i miss this one.

dawehner’s picture

+++ b/core/tests/Drupal/FunctionalTests/BrowserTestBaseTest.php
@@ -152,15 +152,10 @@ public function testLinkExistsExact() {
-    catch (ExpectationException $e) {
...
+    $this->setExpectedException('\Exception', 'Link with label foo|bar found');

@@ -178,15 +173,10 @@ public function testNotLinkExistsExact() {
-    catch (ExpectationException $e) {
...
+    $this->setExpectedException('\Exception', 'Link with label foo|bar|baz not found');

Note: You can specify which exception you expect. Let's do that as we did before :)

mile23’s picture

Status: Needs review » Needs work
Anonymous’s picture

Yeah, like this:

use Behat\Mink\Exception\ExpectationException;
...
$this->setExpectedException(ExpectationException::class, ...)

Also maybe replace names:

public function testNotLinkExistsExact() {
public function testInvalidNotLinkExistsExact() {

on

public function testLinkNotExistsExact() {
public function testInvalidLinkNotExistsExact() {

It will be better to match the name of the tested method: linkNotExistsExact().

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new1.53 KB
new4.15 KB

Specified expected exception.
Edited function names to match tested method.

Anonymous’s picture

Status: Needs review » Reviewed & tested by the community

Thanks, looks nice! All feedback has been addressed.

dawehner’s picture

Nice! I think this is good to go now

larowlan’s picture

Status: Reviewed & tested by the community » Fixed

dded @dawehner to issue credits for numerous reviews.
Added @klausi for investigating test fails and providing reviews.

Committed as e562b4e and pushed to 8.4.x.

Unpostponed #2767275: Convert web tests to browser tests for tour module

  • larowlan committed e562b4e on 8.4.x
    Issue #2860175 by GoZ, Jo Fitzgerald, dawehner, klausi: Add exact option...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.