Problem/Motivation

Since we java javascript test coverage backed upon ower PHPUnit based testing framework we hopefully gained some stability in our JS.
As multiple people discussed though, testing javascript in PHP might not be ideal:

Problems of testing JS in PHP/Advantages of testing JS in JS

  • JS developers who want to participate into Drupal don't have to learn PHP / learn the phpunit testing framework as they are already familiar with nightwatch
  • The JS world is much bigger than the PHP world now, so they should provide better testing frameworks for their needs
  • Nightwatch makes it easy to open up a real browser to see what the test is doing
  • By splitting up the programming languages, we make it clear that Drupal is API first. There is no way of communication possible, which traditionally lead to A LOT of insane JS bugs.

Advantages of testing JS in PHP/ Disadvantages of testing JS in JS

  • One testing syntax to rule them all
  • People are aware of phpunit now already
  • We force the current test writers to learn another testing framework and we're seriously lacking people who are interested in writing tests already.
  • Jumping from BrowserTestBase to JavascriptTestBase is quite easy right now.
  • We split the functional testing into 2 separate frameworks. Upgrading an test to a JavaScript test will be more difficult and I really don't like the split in functional testing.
    Will 'the people' really come if we have another testing framework?
  • Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?

This was discussed on a BOF/meeting during Drupalcon Baltimore.

What we will use

We first agreed that testing JS in PHP was a good first step, but to really get serious about it, we have to leverage JS based tools.

As a tool we agreed on using nightwatch for the following reasons: It has a wide usage, it has good documentation. It can not only support end to end testing but also unit testing. It uses the webdriver protocol, which is an open w3c standard, so it is supported by every browser vendor potentially. On top of that it could support multiple sessions, aka. parallel testing.

How it will work

Each test will reinstall Drupal, just like in PHP.

Testbot side of things

After the agreement we talked about what we need on the testbot side of things. We agreed on the following approach:

  • Provide a package.json script which runs all tests
  • Let the testbot execute it and parse the JSON and call it a day
  • Hope that this is really enough ;)

TL;DR

  • It doesn't make sense to use PHP to test JS
  • Nightwatch is a good tool to do that given its wide usage, support for multiple browsers, its unit testing integration as well as good documentation.

Remaining tasks

  • Provide a more detailed example involving forms, an actual login process ..., fiddling with some ajaxy form. Ideally the follow tests:
    • Drag-n-drop: \Drupal\Tests\book\FunctionalJavascript\BookJavascriptTest.php
    • Dialog: \Drupal\Tests\views_ui\FunctionalJavascript\FilterCriteriaTest
    • Dialog:
    • Complex test: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest
  • Provide package.json integration so the testbot just calls this and doesn't care about how to run the test
  • Make the setup process for a test as simple as possible
  • Provide some basic documentation on drupal.org which points to the right places on the nightwatch documentation
  • Maybe do some research around parallel test running (something the current ecosystem can't provide)

API changes

A total new API for writing tests

Data model changes

How to use it

  1. Install drupal normally
  2. cd core
  3. yarn install
  4. set an environment variable: <code>export SIMPLETEST_BASE_URL="http://localhost/d8"
  5. ./node_modules/.bin/nightwatch -c nightwatch.json example-tests Run the example tests for now

Comments

dawehner created an issue. See original summary.

dawehner’s picture

Here is some start of an experiment, yes just an experiment.

It consists of a couple of pieces:

  • It provides a script to install a specific installation profile in a testing environment and it returns the DB prefix used for it
  • It installs nightwatch and configures a headless chrome
  • It uses the DB prefix to set the right cookie required to get testing running (this is not working properly yet)
dawehner’s picture

FileSize
112.05 KB

Here is the current status of the patch, it is not working properly yet, see comment #2.

drpal’s picture

dawehner’s picture

dawehner’s picture

FileSize
112.55 KB
1.34 KB

Note: This now actually sets up the browser in a way that the child site can be accessed.

dawehner’s picture

Issue summary: View changes

Yesterday we had a discussion involving mixologic, drpal, justafish, mateu, laurri, infiniteluke, wimleers, dawehner
I put the meeting notes/conclusions into the issue summary.

dawehner’s picture

Issue summary: View changes

In case someone wants to try out the patch the issue summary now has a way to describe how to use it.
The documentation is probably not complete yet, but its worth starting to document it :)

dawehner’s picture

Issue summary: View changes
FileSize
934 bytes
112.64 KB

I had to make some adaptions that it actually works. The steps should be complete now.

martin107’s picture

I hope this will be a useful comment,

I agree with

It doesn't make sense to use PHP to test JS

While we evaluate this further can I ask a question about nightwatch's abilities....

Over in #2862510: Convert system/tests/src/Ajax to JavascriptTestBase

we have been looking for a good pattern to use

Is it possible in nightwatch to control the browser to submit invalid form requests, to simulate a bad actor and attack the site.

it is something Mink does not entertain.

In short how would we convert this code segment from FormValuesTest::testSimpleAjaxForm()

    // We don't need to check for the X-Drupal-Ajax-Token header with these
    // invalid requests.
    $this->assertAjaxHeader = FALSE;
    foreach (['null', 'empty', 'nonexistent'] as $key) {
      $element_name = 'select_' . $key . '_callback';
      $edit = [
        $element_name => 'red',
      ];
      $commands = $this->drupalPostAjaxForm('ajax_forms_test_get_form', $edit, $element_name);
      $this->assertResponse(500);
    }
michielnugter’s picture

I'm still not a 100% sure on this but am very interested to see a couple of representative tests converted and working.

My doubts are as follows:

  1. We force the current test writers to learn another testing framework and we're seriously lacking people who are interested in writing tests already.
  2. Jumping from BrowserTestBase to JavascriptTestBase is quite easy right now.
  3. We split the functional testing into 2 separate frameworks. Upgrading an test to a JavaScript test will be more difficult and I really don't like the split in functional testing.
  4. Will 'the people' really come if we have another testing framework?
  5. Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?

The main pro's I see are:

  1. Same testing framework for functional javascript testing and javascript unit testing.
  2. Testing in the same language as the code itself.

In short my programming profile so my doubts can be put into perspective:
I'm mainly a Drupal backend PHP programmer and am one of the PHPUnit initiative leads. I have quite a lot of experience with building a JavaScript web-app on top of Drupal (I have built a complex backbone based progressive decoupled web-app). I'm certainly not protective of the JavascriptTestBase and am open to any alternative that integrates well with Drupal and the rest of the testing needs.

I would love to see a something converted that uses drag-n-drop and a dialog or contextual links though, these are the difficult parts to test in the current JavascriptTestBase.

martin107’s picture

Issue tags: +Needs reroll

Patch no longer applies... I have tried a reroll but it is very difficult to untangle all the changes to yarn.lock

#11 is an I interesting perspective...

I am in the process of forming an opinion

For what it is worth ---

1) This stuff was discussed here.

https://www.lullabot.com/podcasts/drupalizeme-podcast/modernizing-javasc...

2) nightwatch's documentation looks very good. and the pattern in the example look just like our Mink code.

http://nightwatchjs.org/

3) "More example of difficult to solve problems" ... would be a good way to go... I will help out where I can :)

droplet’s picture

I have similar thoughts with @michielnugter.

In most of the cases, we actually testing the UI functionality, not testing the JS code. On another side, PHP developers can't write JS also. You make changes to PHP & JS together. To understand basic PHP is a must in Drupal Core Development.

Also, many of tests get the dynamic data from DB and PHP.

A real case study:
#2725255: Unfiltered data in "Allowed HTML tags"
testJSInjection: ideally, it should provide by PHP Developers. (PHP Developer creates the UI. TDD.)

testParserError: should be a JS UnitTest (mocha)

What discourage our developer to contribute to tests:

[Slightly Off-topic] I think one of the main problems is CORE development always testbots & Core committers first, not contributors first. It squeezed our patience to last minute until you give up. Bug A creates Bug B, Bug B creates C. And Bug E blocked A bring into CORE..etc

- Slow test.
It spent few mins on a simple test in my local env. It easily wasted an hour or so on a random bug.
#2747075: [meta] Improve WebTestCase / BrowserTestBase performance by 50%

- Hard to debug.
Missing visual way to debug:
https://twitter.com/webdriverio/status/806911722682544128
#2857957: Explore a way to provide interactive shell for tests (or use Psysh)

Anyway, what test engines was rejected in the offline meetings?

dawehner’s picture

Issue summary: View changes

It would be great if you could all update the arguments in the issue summary. I'll try to do that, but feel free to edit it.

#10

Is it possible in nightwatch to control the browser to submit invalid form requests, to simulate a bad actor and attack the site.

While mink can't, guzzle can certainly generate invalid POST requests. Do you think this would be an okayish solution?
All those browser testing frameworks are about user interaction testing. Attackers use probably not a browser, but rather raw POST requests.

Does that makes sense?

#11
I added all your points to the issue summary.

Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?

I think the fact that people call out to the container in the middle of the test is way worse than people actually estimate. If you use the container in the middle of the test, you act on a different state of the world, than what is available via HTTP in the browser.
The setup step can totally call out to php, but once we are done with that, calling to PHP is problematic.

One point I added to the issue summary is that by separating the two layers by actually using a different programming language. In the world of API first, we need to think about those side effects.

In short my programming profile so my doubts can be put into perspective:
I'm mainly a Drupal backend PHP programmer and am one of the PHPUnit initiative leads. I have quite a lot of experience with building a JavaScript web-app on top of Drupal (I have built a complex backbone based progressive decoupled web-app). I'm certainly not protective of the JavascriptTestBase and am open to any alternative that integrates well with Drupal and the rest of the testing needs.

Educating people about other technologies aren't a bad thing. People are maybe scared, but I believe career wise its never a bad idea to learn things.

#12

I would love to see a something converted that uses drag-n-drop and a dialog or contextual links though, these are the difficult parts to test in the current JavascriptTestBase.

Do you have a partical example of a test in mind? It would be great to list 1-2 examples in the issue summary and provide an example of them in the test.

#13

In most of the cases, we actually testing the UI functionality, not testing the JS code

This is absolutely true. In a functional testing environment though you don't care that much how something is implemented. On the other hand I doubt our current way of rendering will stay like that.

Also, many of tests get the dynamic data from DB and PHP.

Which is problematic, as written earlier ...

[Slightly Off-topic] I think one of the main problems is CORE development always testbots & Core committers first, not contributors first. It squeezed our patience to last minute until you give up. Bug A creates Bug B, Bug B creates C. And Bug E blocked A bring into CORE..etc

The process is hard, and nearly impossible to understand. I can't agree more. On the other hand there are good reasons for it, like we still try to figure out what keeping BC actually means.

It spent few mins on a simple test in my local env. It easily wasted an hour or so on a random bug.

The issue you linked is mostly about optimizing the testbot performance, but of course, it would be really nice, no question, to improve the everyday test experience.

Anyway, what test engines was rejected in the offline meetings?

We agreed to go with nightwatch, because it for example had parallel test running built in: https://github.com/nightwatchjs/nightwatch-docs/blob/master/guide/runnin...

Just to be clear in general, BrowserTestBase also needed like basically 1 year to be actually used. Of course when we would switch to whatever new technology, we would not just burn everything else down.

michielnugter’s picture

Issue summary: View changes

@dawehner Thanks for the IS update!

Is it possible in nightwatch to control the browser to submit invalid form requests, to simulate a bad actor and attack the site.

I think this should be moved to BrowserTestBase and simulate the request directly, it should be possible there. Doing this in a real-browser env just seems wrong. Still wrong in BrowserTestBase but at least a little less wrong ;)

I think the fact that people call out to the container in the middle of the test is way worse than people actually estimate. If you use the container in the middle of the test, you act on a different state of the world, than what is available via HTTP in the browser.

Mostly it's done to quickly change a setting instead of clicking through the interface. It's a time-saver and makes the test more readable. Should we completely stop doing this? I agree it makes sense to only use the UI but it would mean a significant increase in test-complexity.

Testing from JS would make the impossible.

Educating people about other technologies aren't a bad thing. People are maybe scared, but I believe career wise its never a bad idea to learn things.

Oh sure, completely agree! I actually came to Drupal to learn stuff and actually like learning about this a lot. I just have doubts about this switch and if it will really improve Drupal in general. To be clear: doubts don't mean I'm against it, I just see some problems / considerations.

Do you have a partical example of a test in mind? It would be great to list 1-2 examples in the issue summary and provide an example of them in the test.

Drag-n-drop: \Drupal\Tests\book\FunctionalJavascript\BookJavascriptTest.php
Dialog: \Drupal\Tests\views_ui\FunctionalJavascript\FilterCriteriaTest
Dialog:
Complex test: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest

I updated the IS on this.

Nightwatch makes it easy to open up a real browser to see what the test is doing

The issue is open to move to Webdriver making this possible for the current phpunit JavascriptTestBase as well. Does NightwatchJS offer more?

Of course when we would switch to whatever new technology, we would not just burn everything else down.

Do you mean to support both NightwatchJS and JavascriptTestBase together?

To me: if we're going to switch we should deprecate JavaScriptTestBase and move core to the new testing framework.