Problem/Motivation

There are many core issues where JavaScript tests have gone awry because they expect certain conditions to be true (i.e., certain text or elements to be present) when they are not. Usually these tests try to use JsWebAssert::assertWaitOnAjaxRequest() as a way to wait for the browser to get things in the right state, but that method seems to be quite unreliable.

A working theory that myself, @bnjmnm, and @alexpott have come up with is that, although assertWaitOnAjaxRequest() waits for jQuery to complete AJAXing and animating, it does not have verify that any code in the AJAX command callbacks (i.e., anything in Drupal.AjaxCommands.prototype) is not running. (That's one example that springs to mind, but there could be others.) These kinds of small timing differences may be the reason for the unreliability.

Child issues:

Optional:

Proposed resolution

Deprecate assertWaitOnAjaxRequest() and change the waitFor*() methods in JsWebAssert so that they actually assert their expectations, rather than returning boolean status values. This is also much more developer-friendly because it allows JavaScript tests to assert what they actually expect, rather than trying to hack into and around the low-level workings of whatever is happening in the browser.

Remaining tasks

That thing.

User interface changes

None.

API changes

A widely-used JavaScript testing method will be deprecated, and other assertion methods will change behavior slightly.

Data model changes

None.

Release notes snippet

TBD

Comments

phenaproxima created an issue. See original summary.

phenaproxima’s picture

Issue summary: View changes
phenaproxima’s picture

Issue summary: View changes
wim leers’s picture

I pointed @phenaproxima to #1988968: Drupal.ajax does not guarantee that "add new JS file to page" commands have finished before calling said JS. If we can confirm that this is is caused by that, perhaps we can finally get that issue fixed :) OTOH it's already marked Major and I don't think it can possibly meet the requirements for Critical.

phenaproxima’s picture

Status: Active » Needs review
StatusFileSize
new1.76 KB

A little something to get the ball rolling.

phenaproxima’s picture

StatusFileSize
new2.27 KB

A little clean-up, and fixing utterly wrong and incorrect doc comment on JsWebAssert::waitForText().

oknate’s picture

Nice. Could get a waitForTextRemoved() method too?

@bnjmn is using one for #3055648: Frequent random fail in \Drupal\Tests\media_library\FunctionalJavascript\MediaLibraryTest

He calls it WaitForNoText(), but I think to be more consistent waitForTextRemoved() is better.

Or should that be a different issue?

phenaproxima’s picture

I think that's out of scope here. Adding new assertions should be done in a separate issue(s); this is just about improving the ones we already have.

oknate’s picture

Perhaps it's an antipattern, but I have used $this->assertEmpty($assert_session-> waitForElementVisible('css', '.something', 1000)); The change to waitForElementVisible() will break those types of assertions. What should they be replaced with?

I guess waitForElementRemoved(), which is being added in 8.8?

phenaproxima’s picture

For those cases, we'd need to add a new assertion (waitForElementNotVisible()) or similar. Presumably, those kinds of breakages will be revealed by this patch.

oknate’s picture

I think waitForElementRemoved():
https://www.drupal.org/node/2906685

The last submitted patch, 7: 3061852-7.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 8: 3061852-8.patch, failed testing. View results

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

oknate’s picture

Status: Needs work » Needs review

Removing most (if not all) of the $this->assertNotEmpty()and $this->assertNotNull() code wrapped around JsWebAssert::waitFor*() methods.

oknate’s picture

StatusFileSize
new128.66 KB

Status: Needs review » Needs work

The last submitted patch, 18: 3061852-18.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

oknate’s picture

Broke off a side issue for one of the failures: #3102981: Invalid selector in EditModeTest

oknate’s picture

Status: Needs work » Needs review
StatusFileSize
new132.64 KB
new5.08 KB

Fixing a test or two, fixing some missed assertions testing the result of the updated methods which need to be removed or replaced.

oknate’s picture

oknate’s picture

Title: Deprecate assertWaitOnAjaxRequest() and make the JsWebAssert::waitFor*() methods behave like real assertions » [PP-1] Deprecate assertWaitOnAjaxRequest() and make the JsWebAssert::waitFor*() methods behave like real assertions

Postponing on

We can't really fix this one until those are fixed. But we should keep going through the failures here to see if there are similar issues where tests were passing erroneously.

Status: Needs review » Needs work

The last submitted patch, 21: 3061852-21.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

oknate’s picture

Status: Needs work » Needs review
StatusFileSize
new61.83 KB
new189.15 KB
  • Moving methods to JsWebAssert that were marked as @todos for this issue
  • Fixing some more tests.

Status: Needs review » Needs work

The last submitted patch, 25: 3061852-25.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

oknate’s picture

Status: Needs work » Needs review
StatusFileSize
new5 KB
new190.99 KB

more fixes

oknate’s picture

StatusFileSize
new717 bytes
new190.99 KB

making method public

Status: Needs review » Needs work

The last submitted patch, 28: 3061852-28.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

oknate’s picture

StatusFileSize
new1.69 KB
new192.05 KB

restoring one of the methods removed from the MediaLibraryTestBase.

oknate’s picture

StatusFileSize
new718 bytes
new192.05 KB

making a method public, same mistake as #28 on a different method.

oknate’s picture

Breaking off one part of this as a separate issue: #3103034: Invalid selector in MaximumFileSizeExceededUploadTest

I didn't include it in the file name, because I just created it.

Also, I checked and there are 240 occurrences of assertWaitOnAjaxRequest() in core. Given that this issue is already large, I think we should come up with a way of splitting this up.

Here's how we might do that:

  • Revert the changes where we're moving classes from media library and move that into a separate issue. (See #43 where I did this.)
  • Add a separate issue to deprecated assertWaitOnAjaxRequest, so that could go into a separate issue and actually remove the usages of assertWaitOnAjaxRequest

Another thing we could do is make this a META issue.

oknate’s picture

Issue summary: View changes
oknate’s picture

Status: Needs work » Postponed
StatusFileSize
new1.24 KB
new197.7 KB

Fixing the failure, and marking as postponed.

oknate’s picture

Title: [PP-1] Deprecate assertWaitOnAjaxRequest() and make the JsWebAssert::waitFor*() methods behave like real assertions » [META] Deprecate assertWaitOnAjaxRequest() and make the JsWebAssert::waitFor*() methods behave like real assertions

I think (IMHO), this should be a META issue, and we can do it as a series of child issues.

oknate’s picture

Issue summary: View changes
oknate’s picture

Issue summary: View changes
oknate’s picture

Status: Postponed » Active
oknate’s picture

Issue summary: View changes
oknate’s picture

Issue summary: View changes
oknate’s picture

StatusFileSize
new773 bytes
new198.16 KB

Deprecating assertWaitOnAjaxRequest().

Update: I guess this isn't enough, we'll need to add a @trigger_error: https://www.drupal.org/core/deprecation.

oknate’s picture

StatusFileSize
new811 bytes
new198.4 KB

Adding @trigger_error to assertWaitOnAjaxRequest().

oknate’s picture

StatusFileSize
new21.38 KB
new192.91 KB

1) In #25 I moved some methods from MediaLibraryTestBase to JsWebAssert. I think some of that was out of scope, so reverting those changes. I opened two separate issue to explore moving two of these methods to JsWebAssert:

2) The methods in JsWebAssert to which we added assertions, if they were returning a boolean, no longer need a return. If they fail, they'll throw an error, so there's no point of a return value, AFAIK. They either pass or throw an error. Those that return a node element can still return it, in case it's needed, but the documentation should be updated to indicate it won't return NULL, but with throw an error, for example:

-   * @return \Behat\Mink\Element\NodeElement|null
-   *   The page element node if found and visible, NULL if not.
+   * @return \Behat\Mink\Element\NodeElement
+   *   The page element node if found and visible, otherwise an error is thrown.
oknate’s picture

Issue summary: View changes
oknate’s picture

Issue summary: View changes
oknate’s picture

StatusFileSize
new1.29 KB
new193.82 KB

Updating the @todo items based on #43 to refer to those two new issues.

oknate’s picture

Issue summary: View changes
oknate’s picture

Issue summary: View changes
oknate’s picture

Issue summary: View changes
oknate’s picture

Issue summary: View changes

Created a subtask for the work related to updating the assertions: #3103101: Make the JsWebAssert::waitFor*() methods behave like real assertions

This leaves this issue as a META issue. It also allows 3103101 to be committed independent of #3103081: Deprecate JsWebAssert::assertWaitOnAjaxRequest() in Drupal 9.0 and Drupal 8.9 and Replace usages of JsWebAssert::assertWaitOnAjaxRequest() in core, which will take a lot more work to be ready to commit.

oknate’s picture

Issue summary: View changes

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

tim.plunkett’s picture

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

nod_’s picture

Because of the new add_js command, ajax.ajaxing will now wait for the commands to finish executing before being set to false. That should help a bit.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.