Problem/Motivation

Since we java javascript test coverage backed upon ower PHPUnit based testing framework we hopefully gained some stability in our JS.
As multiple people discussed though, testing javascript in PHP might not be ideal:

Problems of testing JS in PHP/Advantages of testing JS in JS

  • JS developers who want to participate into Drupal don't have to learn PHP / learn the phpunit testing framework as they are already familiar with nightwatch
  • The JS world is much bigger than the PHP world now, so they should provide better testing frameworks for their needs
  • Nightwatch makes it easy to open up a real browser to see what the test is doing
  • By splitting up the programming languages, we make it clear that Drupal is API first. There is no way of communication possible, which traditionally lead to A LOT of insane JS bugs.

Advantages of testing JS in PHP/ Disadvantages of testing JS in JS

  • One testing syntax to rule them all
  • People are aware of phpunit now already
  • We force the current test writers to learn another testing framework and we're seriously lacking people who are interested in writing tests already.
  • Jumping from BrowserTestBase to JavascriptTestBase is quite easy right now.
  • We split the functional testing into 2 separate frameworks. Upgrading an test to a JavaScript test will be more difficult and I really don't like the split in functional testing.
    Will 'the people' really come if we have another testing framework?
  • Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?

This was discussed on a BOF/meeting during Drupalcon Baltimore.

What we will use

We first agreed that testing JS in PHP was a good first step, but to really get serious about it, we have to leverage JS based tools.

As a tool we agreed on using nightwatch for the following reasons: It has a wide usage, it has good documentation. It can not only support end to end testing but also unit testing. It uses the webdriver protocol, which is an open w3c standard, so it is supported by every browser vendor potentially. On top of that it could support multiple sessions, aka. parallel testing.

How it will work

Each test will reinstall Drupal, just like in PHP.

Testbot side of things

After the agreement we talked about what we need on the testbot side of things. We agreed on the following approach:

  • Provide a package.json script which runs all tests
  • Let the testbot execute it and parse the JSON and call it a day
  • Hope that this is really enough ;)

TL;DR

  • It doesn't make sense to use PHP to test JS
  • Nightwatch is a good tool to do that given its wide usage, support for multiple browsers, its unit testing integration as well as good documentation.

Remaining tasks

  • Provide a more detailed example involving forms, an actual login process ..., fiddling with some ajaxy form. Ideally the follow tests:
    • Drag-n-drop: \Drupal\Tests\book\FunctionalJavascript\BookJavascriptTest.php
    • Dialog: \Drupal\Tests\views_ui\FunctionalJavascript\FilterCriteriaTest
    • Dialog:
    • Complex test: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest
  • Provide package.json integration so the testbot just calls this and doesn't care about how to run the test
  • Make the setup process for a test as simple as possible
  • Provide some basic documentation on drupal.org which points to the right places on the nightwatch documentation
  • Maybe do some research around parallel test running (something the current ecosystem can't provide)

API changes

A total new API for writing tests

Data model changes

How to use it

  1. Install drupal normally
  2. cd core
  3. yarn install
  4. set an environment variable: <code>export SIMPLETEST_BASE_URL="http://localhost/d8"
  5. ./node_modules/.bin/nightwatch -c nightwatch.json example-tests Run the example tests for now
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

dawehner created an issue. See original summary.

dawehner’s picture

Here is some start of an experiment, yes just an experiment.

It consists of a couple of pieces:

  • It provides a script to install a specific installation profile in a testing environment and it returns the DB prefix used for it
  • It installs nightwatch and configures a headless chrome
  • It uses the DB prefix to set the right cookie required to get testing running (this is not working properly yet)
dawehner’s picture

FileSize
112.05 KB

Here is the current status of the patch, it is not working properly yet, see comment #2.

drpal’s picture

dawehner’s picture

dawehner’s picture

FileSize
112.55 KB
1.34 KB

Note: This now actually sets up the browser in a way that the child site can be accessed.

dawehner’s picture

Issue summary: View changes

Yesterday we had a discussion involving mixologic, drpal, justafish, mateu, laurri, infiniteluke, wimleers, dawehner
I put the meeting notes/conclusions into the issue summary.

dawehner’s picture

Issue summary: View changes

In case someone wants to try out the patch the issue summary now has a way to describe how to use it.
The documentation is probably not complete yet, but its worth starting to document it :)

dawehner’s picture

Issue summary: View changes
FileSize
934 bytes
112.64 KB

I had to make some adaptions that it actually works. The steps should be complete now.

martin107’s picture

I hope this will be a useful comment,

I agree with

It doesn't make sense to use PHP to test JS

While we evaluate this further can I ask a question about nightwatch's abilities....

Over in #2862510: Convert system/tests/src/Ajax to JavascriptTestBase

we have been looking for a good pattern to use

Is it possible in nightwatch to control the browser to submit invalid form requests, to simulate a bad actor and attack the site.

it is something Mink does not entertain.

In short how would we convert this code segment from FormValuesTest::testSimpleAjaxForm()

    // We don't need to check for the X-Drupal-Ajax-Token header with these
    // invalid requests.
    $this->assertAjaxHeader = FALSE;
    foreach (['null', 'empty', 'nonexistent'] as $key) {
      $element_name = 'select_' . $key . '_callback';
      $edit = [
        $element_name => 'red',
      ];
      $commands = $this->drupalPostAjaxForm('ajax_forms_test_get_form', $edit, $element_name);
      $this->assertResponse(500);
    }
michielnugter’s picture

I'm still not a 100% sure on this but am very interested to see a couple of representative tests converted and working.

My doubts are as follows:

  1. We force the current test writers to learn another testing framework and we're seriously lacking people who are interested in writing tests already.
  2. Jumping from BrowserTestBase to JavascriptTestBase is quite easy right now.
  3. We split the functional testing into 2 separate frameworks. Upgrading an test to a JavaScript test will be more difficult and I really don't like the split in functional testing.
  4. Will 'the people' really come if we have another testing framework?
  5. Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?

The main pro's I see are:

  1. Same testing framework for functional javascript testing and javascript unit testing.
  2. Testing in the same language as the code itself.

In short my programming profile so my doubts can be put into perspective:
I'm mainly a Drupal backend PHP programmer and am one of the PHPUnit initiative leads. I have quite a lot of experience with building a JavaScript web-app on top of Drupal (I have built a complex backbone based progressive decoupled web-app). I'm certainly not protective of the JavascriptTestBase and am open to any alternative that integrates well with Drupal and the rest of the testing needs.

I would love to see a something converted that uses drag-n-drop and a dialog or contextual links though, these are the difficult parts to test in the current JavascriptTestBase.

martin107’s picture

Issue tags: +Needs reroll

Patch no longer applies... I have tried a reroll but it is very difficult to untangle all the changes to yarn.lock

#11 is an I interesting perspective...

I am in the process of forming an opinion

For what it is worth ---

1) This stuff was discussed here.

https://www.lullabot.com/podcasts/drupalizeme-podcast/modernizing-javasc...

2) nightwatch's documentation looks very good. and the pattern in the example look just like our Mink code.

http://nightwatchjs.org/

3) "More example of difficult to solve problems" ... would be a good way to go... I will help out where I can :)

droplet’s picture

I have similar thoughts with @michielnugter.

In most of the cases, we actually testing the UI functionality, not testing the JS code. On another side, PHP developers can't write JS also. You make changes to PHP & JS together. To understand basic PHP is a must in Drupal Core Development.

Also, many of tests get the dynamic data from DB and PHP.

A real case study:
#2725255: Unfiltered data in "Allowed HTML tags"
testJSInjection: ideally, it should provide by PHP Developers. (PHP Developer creates the UI. TDD.)

testParserError: should be a JS UnitTest (mocha)

What discourage our developer to contribute to tests:

[Slightly Off-topic] I think one of the main problems is CORE development always testbots & Core committers first, not contributors first. It squeezed our patience to last minute until you give up. Bug A creates Bug B, Bug B creates C. And Bug E blocked A bring into CORE..etc

- Slow test.
It spent few mins on a simple test in my local env. It easily wasted an hour or so on a random bug.
#2747075: [meta] Improve WebTestCase / BrowserTestBase performance by 50%

- Hard to debug.
Missing visual way to debug:
https://twitter.com/webdriverio/status/806911722682544128
#2857957: Explore a way to provide interactive shell for tests (or use Psysh)

Anyway, what test engines was rejected in the offline meetings?

dawehner’s picture

Issue summary: View changes

It would be great if you could all update the arguments in the issue summary. I'll try to do that, but feel free to edit it.

#10

Is it possible in nightwatch to control the browser to submit invalid form requests, to simulate a bad actor and attack the site.

While mink can't, guzzle can certainly generate invalid POST requests. Do you think this would be an okayish solution?
All those browser testing frameworks are about user interaction testing. Attackers use probably not a browser, but rather raw POST requests.

Does that makes sense?

#11
I added all your points to the issue summary.

Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?

I think the fact that people call out to the container in the middle of the test is way worse than people actually estimate. If you use the container in the middle of the test, you act on a different state of the world, than what is available via HTTP in the browser.
The setup step can totally call out to php, but once we are done with that, calling to PHP is problematic.

One point I added to the issue summary is that by separating the two layers by actually using a different programming language. In the world of API first, we need to think about those side effects.

In short my programming profile so my doubts can be put into perspective:
I'm mainly a Drupal backend PHP programmer and am one of the PHPUnit initiative leads. I have quite a lot of experience with building a JavaScript web-app on top of Drupal (I have built a complex backbone based progressive decoupled web-app). I'm certainly not protective of the JavascriptTestBase and am open to any alternative that integrates well with Drupal and the rest of the testing needs.

Educating people about other technologies aren't a bad thing. People are maybe scared, but I believe career wise its never a bad idea to learn things.

#12

I would love to see a something converted that uses drag-n-drop and a dialog or contextual links though, these are the difficult parts to test in the current JavascriptTestBase.

Do you have a partical example of a test in mind? It would be great to list 1-2 examples in the issue summary and provide an example of them in the test.

#13

In most of the cases, we actually testing the UI functionality, not testing the JS code

This is absolutely true. In a functional testing environment though you don't care that much how something is implemented. On the other hand I doubt our current way of rendering will stay like that.

Also, many of tests get the dynamic data from DB and PHP.

Which is problematic, as written earlier ...

[Slightly Off-topic] I think one of the main problems is CORE development always testbots & Core committers first, not contributors first. It squeezed our patience to last minute until you give up. Bug A creates Bug B, Bug B creates C. And Bug E blocked A bring into CORE..etc

The process is hard, and nearly impossible to understand. I can't agree more. On the other hand there are good reasons for it, like we still try to figure out what keeping BC actually means.

It spent few mins on a simple test in my local env. It easily wasted an hour or so on a random bug.

The issue you linked is mostly about optimizing the testbot performance, but of course, it would be really nice, no question, to improve the everyday test experience.

Anyway, what test engines was rejected in the offline meetings?

We agreed to go with nightwatch, because it for example had parallel test running built in: https://github.com/nightwatchjs/nightwatch-docs/blob/master/guide/runnin...

Just to be clear in general, BrowserTestBase also needed like basically 1 year to be actually used. Of course when we would switch to whatever new technology, we would not just burn everything else down.

michielnugter’s picture

Issue summary: View changes

@dawehner Thanks for the IS update!

Is it possible in nightwatch to control the browser to submit invalid form requests, to simulate a bad actor and attack the site.

I think this should be moved to BrowserTestBase and simulate the request directly, it should be possible there. Doing this in a real-browser env just seems wrong. Still wrong in BrowserTestBase but at least a little less wrong ;)

I think the fact that people call out to the container in the middle of the test is way worse than people actually estimate. If you use the container in the middle of the test, you act on a different state of the world, than what is available via HTTP in the browser.

Mostly it's done to quickly change a setting instead of clicking through the interface. It's a time-saver and makes the test more readable. Should we completely stop doing this? I agree it makes sense to only use the UI but it would mean a significant increase in test-complexity.

Testing from JS would make the impossible.

Educating people about other technologies aren't a bad thing. People are maybe scared, but I believe career wise its never a bad idea to learn things.

Oh sure, completely agree! I actually came to Drupal to learn stuff and actually like learning about this a lot. I just have doubts about this switch and if it will really improve Drupal in general. To be clear: doubts don't mean I'm against it, I just see some problems / considerations.

Do you have a partical example of a test in mind? It would be great to list 1-2 examples in the issue summary and provide an example of them in the test.

Drag-n-drop: \Drupal\Tests\book\FunctionalJavascript\BookJavascriptTest.php
Dialog: \Drupal\Tests\views_ui\FunctionalJavascript\FilterCriteriaTest
Dialog:
Complex test: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest

I updated the IS on this.

Nightwatch makes it easy to open up a real browser to see what the test is doing

The issue is open to move to Webdriver making this possible for the current phpunit JavascriptTestBase as well. Does NightwatchJS offer more?

Of course when we would switch to whatever new technology, we would not just burn everything else down.

Do you mean to support both NightwatchJS and JavascriptTestBase together?

To me: if we're going to switch we should deprecate JavaScriptTestBase and move core to the new testing framework.

dawehner’s picture

Mostly it's done to quickly change a setting instead of clicking through the interface. It's a time-saver and makes the test more readable. Should we completely stop doing this? I agree it makes sense to only use the UI but it would mean a significant increase in test-complexity.

Well, I doubt people really understand the nature of having a different container/state of the system. This time safer argument is totally valid, but we should think more about it. Debugging those hard issues have lead to most probably hundreds of hours of hard to debug tests.

Testing from JS would make the impossible.

I don't think this is necessarily the case. We could still call out from node to PHP, but the good part, is that we never have state (like a container in some version) in the test process itself. Given the PHP script would bootstrap Drupal to some respect it would be more safe to do the necessary changes. Do you think this makes sense?

I just have doubts about this switch and if it will really improve Drupal in general. To be clear: doubts don't mean I'm against it, I just see some problems / considerations.

Sure, that's exactly why we have a discussion :) We don't want to blindly jump onto something.

The issue is open to move to Webdriver making this possible for the current phpunit JavascriptTestBase as well. Does NightwatchJS offer more?

Given that this property is purely based upon the fact that its using webdriver. I see a tiny advantage given that unlike JTB we wouldn't implement it for ourselves.

Do you mean to support both NightwatchJS and JavascriptTestBase together?
To me: if we're going to switch we should deprecate JavaScriptTestBase and move core to the new testing framework.

Well, this is what happened with phpunit tests, web tests and browsers tests as well. Unit tests got introduced quite early (2013) but it took time to be leveraged. The same thing happened with BrowserTestBase (which is 2 years old now). Future planning is part of big opensource projects :)

Mixologic’s picture

Priority: Normal » Major

Im bumping the priority of this.

PhantomJS is broken, and we didnt really run into it until we tried to repurpose some extra hardware in our racks for some testbots. Basically it freezes because the underlying networking subsystem is broken. (https://github.com/ariya/phantomjs/issues/14286) The only options to make it work at this point are upgrade it to a fully unsupported 2.5.0 beta version, and *hope* that fixes the problems, or get rid of it once and for all and convert everything to *not* use phantomjs.

Of course when we would switch to whatever new technology, we would not just burn everything else down.

Can we burn phantom js just a little?

michielnugter’s picture

@Mixologic

I think the road for this is still a bit too long to help you out in short term. Also as stated by @dawehner in #16 this issue would not mean removing the current form of JavascriptTestBase in the short run, we'll have to keep supporting JavascriptTestBase at least until 9.x.

I think the most efficient way of getting rid of PhantomJS for the test-bot is getting this issue in: #2775653: [PP-2] JavascriptTests with webDriver.

That's about switching to Webdriver for JavascriptTestBase, after that gets in we could do a follow-up to change the browser to chrome headless?

michielnugter’s picture

Priority: Major » Normal

I'm in favor of resetting the priority to normal here and upgrading the priority on the mentioned issue as this is really a huge change that really can't use a lot of pressure to get in, if we're going to switch we want to do it right.

Feel free to disagree ;)

Mixologic’s picture

Yes, perhaps this is not the issue that is the shortest path to gutting phantomjs. As long as there is a path forward that has some way that the testbots do not have to have that as an option, thats great (it certainly doesnt have to be this issue).

michielnugter’s picture

Debugging those hard issues have lead to most probably hundreds of hours of hard to debug tests.

Given the PHP script would bootstrap Drupal to some respect it would be more safe to do the necessary changes. Do you think this makes sense?

Nice to see this change forces us to think about how to test specific situations :)

Clicking through the UI won't make the tests better readable but it may make them better understandable.

Though if we separate environment preparation from the actual tests (could be a get call to a specific URL or a method on the test class itself) sounds like a good idea. It makes documentation better possible and more clear that the purpose is environment preparation.

Well, this is what happened with phpunit tests, web tests and browsers tests as well. Unit tests got introduced quite early (2013) but it took time to be leveraged. The same thing happened with BrowserTestBase (which is 2 years old now). Future planning is part of big opensource projects :)

For sure we have to keep support for JavascriptTestBase but if we change I would like all current tests converted to use the best framework. We don't have that many tests currently so it's still doable.

dawehner’s picture

Yeah let's be practical and do babysteps:

* Get chrome with webdriver working on the testbot
* We will make most probably run into stability issues again, nothing is ever easy!
* Then continue investigating on this issue here ... I'll try to keep this issue up to date in a state that people can give feedback.

For sure we have to keep support for JavascriptTestBase but if we change I would like all current tests converted to use the best framework. We don't have that many tests currently so it's still doable.

That is fair.

drpal’s picture

👍

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.