Problem

Goal

  • Pave the way for replacing Simpletest with PHPUnit.

Battle plan

  • There's no way we're going to convert 40,000+ assertions over night. We aim for concurrent/parallel testing framework support in D8.
  • drupal.org testing infrastructure won't support PHPUnit over night. The testing infra team works on a new job manager anyway already, so patching PIFR makes no sense.
  • The testing framework is technically detached from Drupal core. It was beneficial to add it to core for D7 to foster adoption, but otherwise it shouldn't have been added. The new web tests should be designed in a cross-major-version compatible way.
  • This effort requires involvement and hard work from all testing system experts, but everyone is extremely busy with getting D8 ready and soon out of the door. People will shoot us and get very angry if testing system changes will suddenly break their last-minute work. We will not do that. An effort like this should start at the beginning of a release cycle.
  • Nevertheless, we cannot leave Simpletest in its current state, since it is broken in many aspects and continuously hinders core developers. We have to improve it in parallel.
  • In concrete terms:
    Long-term (D9)
    1. PHPUnit for unit tests and + Mink + Guzzle + Goutte for web/integration tests. Quite potentially Behat for BDD tests.
    2. No more remnants of Simpletest in core and contrib.
    Short-term (D8)
    1. Improve Simpletest where possible and attempt to bring it more in line with PHPUnit to make the long-term transition easier.
    2. Try to allow developers to write PHPUnit tests (not in core) by preparing the necessary (code) infrastructure if possible, without officially supporting it. (i.e., base classes for running unit/web tests with PHPUnit.)
    3. Start to hammer on this after feature freeze until release.

Agreements

  • 1 test site (no parent site leaking into child site).
  • PHPUnit is used by everyone in the PHP world.
  • Remove dependency on full site + Simpletest module being installed.
  • Write results/log into JUnit .xml results file.
  • Install Drupal with (stripped down) testing profile + class-specific module(s).
  • TBD: Kick off test runs and show results in a separate web app ? - No Simpletest in core.
  • TBD: No testing framework in core. [barrier for novice contributors?]
  • Guzzle as internal browser, because we should have it in core anyway.
  • PHPUnit has assertSame() for type-strict comparisons (== current assertIdentical()).

Details

All tests
Unit tests
Web tests
Behat tests
  • Dedicated issue: #2232271: [Meta] Use Behat for validation testing
  • Leverage and re-use infrastructure of default testing configuration + test helpers for Behat natural language parsing, Feature contexts, steps.
  • Figure out the generic natural language that works for 10,000+ modules. Start with Drupal Commons tests. The steps are in Ruby but easy to convert.
Dependency management
  • Have all required libraries ship with core. Testing must not involve barriers.

    Moshe: Though nice, I don't think this is feasible. PHPUnit has versions, updates, installs via PEAR, etc. we can't take all that on.
    sun: The version is actually a reason for including it, so as to make sure that all tests are compatible. PHPUnit has BC breaks in new versions, too, like any other software.

Test Bot
Dissent / Unclear

Resources

PHPUnit + bridge efforts:
https://www.acquia.com/upal
http://denver2012.drupal.org/content/upal-proposed-test-framework-drupal-8
https://github.com/weitzman/upal (has a unnecessary dependency on Drush)

Behat:
#1551600: DBehave!
http://docs.behat.org/guides/1.gherkin.html#backgrounds
http://drupalcode.org/project/doobie.git/blob/725cc24ff3d4acca36d68f9f19...
http://drupal.org/node/1578324

not:
#249553: Rename SimpleTest to Testing (test.module)

Comments

sun’s picture

Project:Drupal Platform» Drupal core
Issue summary:View changes

Updated issue summary.

sun’s picture

Issue summary:View changes

Updated issue summary.

sun’s picture

Issue summary:View changes

Updated issue summary.

sun’s picture

Issue summary:View changes

Updated issue summary.

gagarine’s picture

Project:Drupal core» Drupal Platform

atoum

I just get a small presentation of it. 30s after I was searching if someone was thinking to move away of us hacked Simpletest.

It's nice, easy, fast, modern and flexible.

mageekguy’s picture

I'm the creator and lead developer of atoum and i have no problem to help you to use it instead of SimpleTest.

sun’s picture

@mageekguy: Thanks a lot for taking the time to register and letting us know! That means a lot. :)

README.txt

In general, this entire effort will still take some more time to actually take off. It will take a shitload of careful planning and architectural design work to make the transition possible, without losing or endangering the existing test coverage of Drupal core (with ~37,000+ assertions) as well as the existing test coverage of ~2,000+ contributed modules (total amount of assertions unknown).

In fact, this very issue totally is a placeholder at this point only. I created it after wasting entire days of developer time with duct-taping our current testing framework, and so the issue summary is totally not in a format that was intended for public consumption ;)

That said, regardless of which new testing framework we're going to choose, I actually see a very high chance for having to support multiple testing frameworks at the same time — i.e., kinda circling back into the above; we're not able to just simply throw away tens of thousands of lines of code of existing test coverage, and due to that massive amount, as well as a constantly/daily changing Drupal core HEAD target/tip, a end-day/deadline for migrating all tests also seems impossible. The only exception to that might be a "compatibility layer" between the new testing framework and our current (which basically is the idea of Upal; which attempts to bridge and port our current base test classes to PHPUnit).

Another reason for possibly supporting multiple frameworks could be that we're currently migrating Drupal core's base system to Symfony, and Symfony components ship with their tests bundled for each component starting from 2.1 (dunno whether Symfony changed that because of my complaints, but either way, totally appreciated ;)), and their tests are based on PHPUnit currently.

Speaking of base test classes above, Drupal implements two entirely different types of tests: straight unit tests and functional/integration tests. The former (unit) tests are fairly easy to migrate to "whatever", since they do not have or use any special methods that wouldn't exist in any other testing framework. But alas, we only have a very small amount of unit tests in the first place, since up until Drupal 8.x/HEAD, core didn't implement any DI container and pretty much no OO code in general, but most functions in Drupal require some other subsystems to be up and running, so it wasn't possible to mock anything for unit tests at all.

Thus, the majority of existing tests is based on our WebTestBase, which actually installs a complete Drupal site like the regular Drupal installer for every test being executed, using a random database prefix and some logic baked into Drupal's (regular runtime) core that re-routes all HTTP requests originating from WebTestBase into the "child site under test" (the db-prefixed one). Based on that, every test is able to specify additional modules to install into that child site, in order to test various combinations of modules and configuration and content/data and user input and whatnot.

It is fairly important to understand that Drupal, unlike PHP frameworks, is a modular, extensible, and entirely "configuration-based" application layer, whereas the configuration can be and is supposed to be changed via the UI. This inherently means that, for the most part, it makes more sense to test module functionality in an actual Drupal environment instead of attempting to test it in a mocked unit testing environment, since the results can easily vary. This doesn't mean there's no point in unit tests (there totally is and we totally should have more), but in most cases, the results are vastly different based on the order in which modules and thus their hooks/events are executed. Thus, right now, most of Drupal's tests are trying to verify the expected functionality from an end-user perspective, which apparently matters most for our end goal of delivering a product that works and can be configured and enhanced from the UI/frontend.

So for the most part, I doubt that those web/functional/integration tests are going to change dramatically, since neither PHPUnit nor atoum seem to provide any solution for what we're doing there — both of them seem to focus on unit tests only. In turn, this means we'll have to port that WebTestBase layer onto whichever testing framework anyway. We're currently trying to at least align those tests with PHPUnit's stance of setting up a testing environment as well as fixtures in #1411074: Setup test environment only once per test class; introduce setUpBeforeClass() and tearDownAfterClass() [PHPUnit] (which probably isn't unique to PHPUnit only).

Due to massive adoption of PHPUnit in other PHP projects and also common support in IDEs, there has been a some push towards PHPUnit recently. Although I've to admit (and heard similar feedback from fellow Drupalers) that atoum looked very attractive at first sight ;) (although I should note I didn't really dive in very deeply, merely glanced over the front page/readme on github) OTOH, PHPUnit's very extensive documentation and tutorials on authoring and designing tests is a very compelling counter-argument, which definitely helps a community like Drupal, since we need to teach thousands of contributors and users with (highly) varying levels of expertise in how to write tests. Also, when I first looked into PHPUnit, it looked "old" on the surface, but their latest development branch and "plugin" code is actually modern (or modernizing) inside.

Given that, overall, I'm not sure why atoum was created as a separate framework, instead of improving PHPUnit? ;) (yes, lemme play the provocative card here :P, but alas, that's a serious question which I'm absolutely sure of coming up, repetitively even)

Lastly, speaking of "Given", there's a great desire to leverage BDD/Behat style testing in Drupal tests, since probably ~30% of our existing web/functional/integration tests could be semi-easily rewritten into BDD/Behat's macro testing language — which means a huge potential for us, since writing such tests does not require any PHP knowledge, so it can be done by total newcomers/novices, which in turn is the opportunity (believe it or not) for getting completely new contributors involved, who may or may not turn out to be the next Drupal core maintainer 5 years later; i.e., affecting the very heart of the Drupal community.

HTH to give some high-level perspective. 1st paragraph still takes effect, but of course, any upstream testing framework info upfront helps to shape a better picture. :)

Thanks!

sun’s picture

Project:Drupal Platform» Drupal core
Issue summary:View changes

Updated issue summary.

bennetteson’s picture

Project:Drupal core» Drupal Platform

Atoum is more attractive and more easy to use.

gagarine’s picture

For a "web client" to do functional test http://mink.behat.org/index.html

mink is integrated out of the box with PHPUnit, Behat and Symfony2 but any integration is certainly easy http://mink.behat.org/phpunit.html

And for functional test in natural language, take a look at behat (it's used mainly for https://en.wikipedia.org/wiki/Behavior_Driven_Development ...).

EDIT: didn't see the issue summary evolve from the last time I read it.

gagarine’s picture

Project:Drupal Platform» Drupal core
Issue summary:View changes

Updated issue summary.

Sylvain Lecoy’s picture

Project:Drupal core» Drupal Platform

If you find this paper good Agile Unit Testing you can add it as a resource guidelines regarding unit testing.

Sylvain Lecoy’s picture

Project:Drupal Platform» Drupal core
Issue summary:View changes

Updated issue summary.

sun’s picture

Issue summary:View changes

Added discussion notes and battle plan.

sun’s picture

Title:Ditch and rewrite Simpletest, it is broken beyond repair» [meta] Replace the testing framework with PHPUnit + Ditch and rewrite Simpletest, it is broken beyond repair
Version:» 8.x-dev
Component:Code» simpletest.module
Category:bug» task

I've copied my notes from various discussions with various people into the issue summary, and revamped it entirely, including the envisioned battle plan.

Shapes the rough edges. I'm aware that it doesn't contain actual + actionable steps for getting a new testing system (code) infrastructure in place (i.e., base classes and so on). There's a tonne of things to figure out there. But frankly, I don't think anyone has time to discuss that until Dec 1st, so I'd prefer to defer that discussion until then, but of course wouldn't mind if anyone would jump on the gun and try out some initial PoC code, especially regarding Mink-based web tests.

Nevertheless, moving into core queue.

sun’s picture

Issue summary:View changes

Clarified and completed battle plan.

sun’s picture

Issue summary:View changes

More clarifications, moved PHPUnit docs links.

sun’s picture

Issue summary:View changes

Quote.

Sylvain Lecoy’s picture

Added a POC of a block unit test re-factored to use PHPUnit #1801176: Deploy a PHPUnit system with a bottom-up approach.

fgm’s picture

FWIW, there are already actual sites using PHPUnit tests in a Drupal setup: some customers of mine wouldn't bend over their testing and CI practice and chose to develop unit tests for their custom code using PHPunit only, no Simpletest runner.

fgm’s picture

Issue summary:View changes

Added Unit test.

Sylvain Lecoy’s picture

Issue summary:View changes

Added Issue #1801176 in the Unit test section.

Sylvain Lecoy’s picture

Issue summary:View changes

typo

fgm’s picture

More about this:

  • just as an experiment, I started adding PHPunit tests on object-based code in a recent project, and running it through Drush essentially means just having to make the tests inherit drush_unit_test_case and running drush_bootstrap at the beginning of each test (and cleaning every DB change at the end of the test. Mocks really help with object-based code, and sometimes just wrapping the basic core function call in an object method is all it takes to enable actual unit-testing via a mock
  • regarding Behat, Mink + Guzzle + Goutte is a nice and fast way to run integration tests, but they completely miss the JS testing, while more and more of our code becomes JS-dependent. Something else is also needed to support it (Selenium2, Sahi, Zombie...)
  • something I would very much like to see is PHPcov integration, which comes as a given with PHPunit and not (AFAIK) with our current simpletest: it would enable devs to see how far they are from C0 coverage when pushing code. While 100% C0 coverage is neither necessary nor sufficient, it is a useful direction
  • we really should have something to enable newbie devs to run and write tests without extra complication: the whole topic of testing is intimidating enought as it is
franskuipers’s picture

How can we take this initiative further?

I was very happy to see PHPunit sneaking into core with #1901670: Start using PHPUnit for unit tests! And because I want the learn and get experience with PHPunit I started to work on this in #1938184: Convert Drupal\Core\Utility\Color\ColorTest to PHPUnit and #1938068: Convert UnitTestBase to PHPUnit.

Looking around more into all PHPunit related issues I realised it is not a concerted effort yet.

Some ideas to bring this further:

I would like to help/carry on with this issue, but need some guidance as how to proceed.

RobLoach’s picture

Issue tags:+PHPUnit Blocker

The PHPUnit Blocker tag is moving a few parts of Core over to PHPUnit.

RobLoach’s picture

Issue summary:View changes

Added Issue #1872006 in the Unit test section.

larowlan’s picture

Version:8.x-dev» 9.x-dev

This is 9.x territory now, alas

larowlan’s picture

So I've started plugging on a bridge class LegacyWebTestBase that extends UnitTestCase (PHPUnit) and has 1:1 methods with SimpleTest WebTestBase
I've pushed this to a branch in my core sandbox
I have it to the point where you actually see the class make Curl requests (I have some scaffolding in for Mink/Goutte but leaving it with Curl/existing logic until stuff is at least running).
Commit summary (from sandbox)

ea9a7f6 Composer update, new vendor files
d7fb3dd Added php unit test listener
9f9f13e WIP on legacywebtestbase, up to 2175
c7014d4 [#die] Views fixes to make tests not need t()
afc8208 Changes to phpunit config
1ea406e Changes to LegacyWebTestBase
a3be798 Tour uses legacy test base
a3e9405 Disable verbose for now

To see where I'm at

cd core
./vendor/bin/phpunit

It will error out because no $_SERVER['TESTING_BASE_URL'] provided.
Edit your settings.php and add $_SERVER['TESTING_BASE_URL'] = 'http://d8.local'; using your local url.

./vendor/bin/phpunit

Sample output

PHPUnit 3.7.21 by Sebastian Bergmann.

Configuration read from /var/www/d82/core/phpunit.xml.dist

...............................................................  63 / 504 ( 12%)
............................................................... 126 / 504 ( 25%)
............................................................... 189 / 504 ( 37%)
............................................................... 252 / 504 ( 50%)
............................................................... 315 / 504 ( 62%)
............................................................... 378 / 504 ( 75%)
............................................................... 441 / 504 ( 87%)
..............................................................Starting 'testTourFunctionality'
F 504 / 504 (100%)
Test 'testTourFunctionality' failed.
Failed to set field name to e8hxk751
Failed asserting that false is true.mkdir() [<a href='function.mkdir'>function.mkdir</a>]: Permission deniedchmod() [<a href='function.chmod'>function.chmod</a>]: Permission deniedfile_put_contents(/var/www/d82/sites/default/files/php/service_container/.htaccess) [<a href='function.file-put-contents'>function.file-put-contents</a>]: failed to open stream: Permission deniedmkdir() [<a href='function.mkdir'>function.mkdir</a>]: Permission deniedchmod() [<a href='function.chmod'>function.chmod</a>]: No such file or directoryfile_put_contents(/var/www/d82/sites/default/files/simpletest/797998/php/twig/.htaccess) [<a href='function.file-put-contents'>function.file-put-contents</a>]: failed to open stream: No such file or directoryVerbose output from http://d82.taco/user:http://vendor/bin/sites/default/files/simpletest/verbose/-0.html

Time: 30 seconds, Memory: 66.25Mb

There was 1 failure:

1) Drupal\tour\WebTests\TourTest::testTourFunctionality
Failed to set field name to e8hxk751
Failed asserting that false is true.

/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:735
/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:2680
/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:1845
/var/www/d82/core/modules/tour/webtests/Drupal/tour/WebTests/TourTest.php:36
/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:832

FAILURES!
Tests: 504, Assertions: 1010, Failures: 1.

Note This will hose your existing site, we still have leakage between the containers/compiled twig

moshe weitzman’s picture

Note This will hose your existing site, we still have leakage between the containers/compiled twig

I still really think we should ditch simpletest and rely solely on phpunit. We have tried to plug these holes for *years* and been unsuccessful. The mental burden will be so much lower if we only have one Drupal, and rely on phpunit to be our test runner. I will personally write the (small) patches to help test bot run phpunit instead of simpletest.

Thanks for working on this.

catch’s picture

The test bot is already running PHPUnit and there's issues to convert existing core unit tests to use those. No bridge, just one-by-one rewriting the unit tests.

What we don't have is issues to convert functional tests away from PHPUnit - we're still going to have some functional tests, and PHPUnit isn't really designed for those. For that I think it'd make sense to start using Behat against core - which would get us functional JavaScript testing as well.

larowlan’s picture

@catch I wasn't trying to bridge unit test, but functional tests aka WebTestBase.
The idea being get something going with PhpUnit as the runner and then integrate Mink as the browser instead of SimpleTest.

jrockowitz’s picture

I am just starting to learn D8 and Symfony and was looking into how both handle functional testing. I found functional testing in Symfony very intuitive (http://symfony.com/doc/current/book/testing.html#functional-tests).

I started reading this issue and could not help but wonder why we wouldn't consider extending Symfony's WebTestCase and use it to replace SimpleTest. My guess is it might require Drupal 9 to include more components from Symfony, which seems to be the direction things are going for D9.

I am sure I am missing something ... but sometimes it is worth asking some simple questions when trying to solve big problems.

moshe weitzman’s picture

@jrockowitz - thats a great find! Symfony's webtestcase simulates an http client, but there is no actual http involved. thats a nice speedup. See http://symfony.com/doc/current/book/testing.html#working-with-the-test-c...

I'd be fine with Symfony's WebTestCase or Behat for our functional tests. Behat uses http which is unfortunate but writing tests is approachable to anyone.

fgm’s picture

Actually, you don't have to use HTTP at all to run behat, especially when using the Drupal extension: tests can run either using the API, using Drush, OR using and actual HTTP client (Blackbox API). It's simply a matter of tagging your scenarios so that the extension knows how to run them, and configuring the drivers in the test config.

http://www.dspeak.com/drupalextension/drivers.html

larowlan’s picture

I wonder if making this 1:1 compatible phpunit+mink replacement for simpletest means this could be in 8.1.0 in a semantic-version world?

Owen Barton’s picture

If I am reading it correctly, http://extensions.behat.org/symfony2/ provides a Behat driver that runs requests against the Symfony kernel directly - this would seem to give us the best of both worlds - write functional tests in Behat, but avoid the HTTP overhead.

Crell’s picture

Just a note, I'm not sure that Drupal's bootstrap is at the point yet that it could handle a non-true-HTTP functional test. A lot of progress has been made in that direction but especially things like the installer may not be able to handle it. I know Mark ran into lots of issues between simpletest and the installer when trying to separate out the bootstrap into better kernel layers.

That said, if we could pull off simulated-request-based testing that would a HUGE win, and I fully support it. Having taken that approach on a Silex project recently I much preferred it to what we do now in Simpletest.

Crell’s picture

Issue summary:View changes

Added issue 1901670.

chx’s picture

Title:[meta] Replace the testing framework with PHPUnit + Ditch and rewrite Simpletest, it is broken beyond repair» [meta] Replace the testing framework with PHPUnit + Ditch
Issue summary:View changes

simpletest is not broken beyond any repair. As I have mentioned in several issues: as long as you need to install a child Drupal to test and run it with Drupal you will have problems. That is irregardless whether we use our own browser or someone else's.

chx’s picture

Title:[meta] Replace the testing framework with PHPUnit + Ditch» [meta] Replace the testing framework with PHPUnit and possibly rewrite Simpletest

I am told Ditch is not a PHP project being considered. What do I know.

catch’s picture

Status:Active» Postponed

PHPUnit is already in.

If we're going to make any testing changes apart from individual conversions of some functional/integration tests to PHPUnit, I'd want to focus on js testing. JS testing will also individually convert some simpletests that are working around not being able to execute js, then there's a discussion to be had about what happens to the tests that don't belong to either browser testing or PHPUnit.

The issue for that is #237566: Automated JavaScript unit testing framework , so postponing this one.

jessebeach’s picture

sun’s picture

Component:simpletest.module» phpunit
Assigned:sun» Unassigned
larowlan’s picture

Spoke to omissis in irc who's been experimenting with this.
Encouraged him to create a new issue to track his approach - he created #2283257: [meta] Porting Drupal Integration and Functional TestCases to PHPUnit
Have added it as a child.