Problem/Motivation

I have started working on a symfony command line tool integrated with Drupal in order to play with functional and integration tests in phpunit as a way to dive in Drupal 8 development and possibly help moving away to simpletest if the experiment succeeds.

Proposed resolution

I have created a branch on github (https://github.com/omissis/drupal/tree/phpunitize-all-1234567890/core/sc...) to move this forward. At the moment there's a basic command that proxies and bootstraps phpunit and a very basic web test case (which should probably be renamed kernel test case).
The command can be invoked as such:

php core/scripts/console tests:run --verbose --phpunit-verbose --phpunit-colors --phpunit-configuration=/Users/omissis/Sites/experiments/drupal8-rules/core/phpunit.xml.dist path/to/phpunitTest.php.

Please note that all phpunit options are prefixed with --phpunit to avoid clashes with the console app options, which in turn is going to be developed to be as compatible as possible (command-line wise) to ./core/scripts/run-tests.sh.

Remaining tasks

* Kernel boot is still not complete and possibly not optimal;
* All the run-tests.sh options should be ported;
* Database setup/teardown is not there at all yet, as my primary focus at the moment is db-independent integration testing.

User interface changes

(New or changed features/functionality in the user interface, modules added or removed, changes to URL paths, changes to user interface text.)

API changes

The Symfony/Console dependency has been added as well as a console script needed to bootstrap the command line tool.

Original report by omissis

Comments

larowlan’s picture

larowlan
Location 🇦🇺🏝.au GMT+10
commented
Title: Porting Drupal Integration and Functional TestCases to PHPUnit » [meta] Porting Drupal Integration and Functional TestCases to PHPUnit

Adding meta to title

larowlan’s picture

larowlan
Location 🇦🇺🏝.au GMT+10
commented
Parent issue: » #1567500: [meta] Pave the way to replace the testing framework with PHPUnit and possibly rewrite Simpletest
robloach’s picture

robloach
he/him
commented

I like it! Does it have to be part of Drupal core?

+   "Drupal\\Core\\": ["core/lib/Drupal/Core", "core/scripts/Drupal/Core" ],

Since it's part of Core, I see no harm it in being in core/lib/Drupal/Core, or even core/lib/Drupal/Component/Script. It's just a namespace, don't let pre-existing directories fool you.

sun’s picture

sun
Primary language German
Location Karlsruhe
commented
Related issues: +#2248845: [run-tests.sh] simpletest_phpunit_xml_filepath() depends on public:// stream wrapper

Hm. It looks like the code does not really differentiate between the very discrete/different concepts of WebTestBase and KernelTestBase, but that hard differentiation should exist right from the start.

Attempting to convert WebTestBase involves a high risk of conflicting with parallel Behat/etc work and discussions around acceptance tests.

The stated list of goals are too far-reaching. There's no point in resembling the run-tests.sh command line options if the test runner is going to be phpunit — If it's phpunit, then it's phpunit. All of the Console related integration code is very distracting and out of scope (and duplicates 1-2 other issues).

I'd recommend to do the following:

  1. Focus. Narrow down the goals to just one test (type) base class. Strip away everything else + don't get distracted. Ignore legacy.

  2. Focus on the real meat and hard challenges first - e.g., the last point in the OP about (safe) test site directory + database setup must be the very first task. If these aspects do not work accurately and securely, then the entire effort is going to fail.

  3. Pick exactly one simple existing test of the converted type that is converted as part of the prototype as a proof of concept.

Challenges to be resolved:

  1. PHPUnit base working directory is ./core instead of ./ — cwd()'ing may work, but must not affect other tests in case a test prematurely ends.
  2. Setting up a test site directory should hopefully work using the existing functions, as they do not depend on a fully booted Drupal environment - but we might want to move/convert them into classes upfront (in a separate issue).
  3. Simpletest uses the database connection of the parent site by default (if no dburl is specified on the command line). It would be very wise to make this code NOT aware of any parent site and not inherit anything, and require to pass a dburl to phpunit instead. Perhaps through _ENV.
  4. This functionality will require write access to various filesystem locations, whereas the filesystem may not have been set up in any way yet. An explicit list of write access locations needs to planned, specified, and documented somewhere. The test runner needs to clean up after itself, even in case of unrecoverable fatal errors that interrupted a test.
  5. + much more I forgot to think of…

Those are the hard nuts to crack. I'd love to see, review, and help to work on an attempt to resolve them.

But I don't like to get distracted by off-topic/out-of-scope changes to port run-tests.sh to Console and whatnot… especially because there's no point in touching the legacy simpletest test runner of run-tests.sh at all, since the end result is going to be that these new tests are not going to be run with that script anymore. Likewise, PHPUnit is a fairly sophisticated test runner; it should not be wrapped into a custom.

omissis’s picture

omissis commented

@sun thanks for the answer.

I agree WebTestCase is not an appropriate name (and scope): the intent of this effort was geared toward integration tests that do not simulate a client as the Mink/Behat project will do.

Porting the current test runner to Console is just an idea which only purpose is to ease the porting of the current infrastructure to phpunit: I was not expecting that the whole infrastructure for, say, Drupal 9 is going to change overnight(I thought that if that was easy/quick it would have already been done for D8) and therefore maintaining command-line compatibility seemed at least cautious to me.
About rewriting the test runner: if the point above is no longer valid, it might still be needed to achieve parallel testing execution a-la-paratest(https://github.com/brianium/paratest): I am not sure is a strict requirement but if it's needed I don't think paratest will be a very reliable option.

All of that said, the console classes are separated from the bootstrapper and the testcase class, so getting rid of it and get back to vanilla phpunit should not be a big deal: I am not sure if that's a good idea anyway because in that case if you need to pass any custom option and implement some custom behaviour, you'll need to do that at the phpunit's bootstrap file level, which might bring us back to what Console does: unless you want to go the run-script.sh way and end up with 1300+ lines of (untested?) procedural code.

Anyway, for the proof-of-concept I just managed to port some Rules 8 tests to the new base class: now that I have some of them running I need to dive deeper into the filesystem and database setup and teardown procedures: I see they are quite complex and unfortunately from the little I have seen there is no way to decouple much from the database when it comes, for instance, to entities and the way they're stored. I need more time to understand how those systems work though.

jibran’s picture

jibran
he/him
Primary language Urdu
Location Toronto, Canada
commented
Version: 8.0.x-dev » 8.1.x-dev

Is this still relevant

dawehner’s picture

dawehner
Primary language German
commented

With #2304461: KernelTestBaseTNG™ and BrowserTestBase we kinda have the tools now which will solve the issue title. Just converting the script to console doesn't give us that much.
I would vote for won't fix.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Related issues: +#2624926: Refactor run-tests.sh for Console component.

Similar but different: #2624926: Refactor run-tests.sh for Console component.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dawehner’s picture

dawehner
Primary language German
commented
Status: Active » Closed (duplicate)

This feels really similar to #1567500: [meta] Pave the way to replace the testing framework with PHPUnit and possibly rewrite Simpletest ... note, that issue is basically done. I'll mark this issue as duplicate now