Fill in @defgroup/topic docs for Testing overview

Yay :) Here's some feedback...

1. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
   @@ -26,6 +26,8 @@
     * Test case for typical Drupal tests.
   + *
   + * @ingroup testing
     */
    abstract class WebTestBase extends TestBase {
   

   See below, you should also tag DrupalUnitTestBase.

2. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + * The Drupal project has embraced a philosophy of using automated regression
   + * tests. The basic idea is to have automated tests in place, both unit tests
   

   I don't think all tests are regression tests?

3. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + * - When making a patch to fix a bug, start by writing a test that illustrates
   + *   the bug and demonstrating that this test fails. Then fix the bug, and
   + *   demonstrate that the test passes.
   

   While that is the theory, it's not always that easy to write a test first, as you might not know how exactly it should behave until you fixed it :) Especially for non-unit tests.

   I'd leave out the test-first philosophy and instead just say that bugfixes need to come with a test to proof that the bug is fixed and stays fixed.

4. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + *
   + * @section running Running tests
   + * Both unit tests and functional tests are automatically run whenever you
   + * submit a patch for review to the Drupal Core project on https://drupal.org.
   + * Some contributed patches also have automated patch review set up.
   

   A lot here is about drupal.org and processes we have here. That's quite different to most other API documentations that we have. Not sure how much of that should really be in here.

5. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + *
   + * You can run tests yourself using the core/scripts/run-tests.sh script,
   + * using @link https://drupal.org/project/drush Drush @endlink, or from the
   + * Drupal user interface using the core Testing module.
   

   run-tests.sh needs the testing module to be installed too (at least unless you use it in the non-parent-installation mode).

   Also, we should probably also mention that you can run (phpunit) unit tess with phpunit, which is way easier to use and faster than run-tests.sh and I always recommend to use that directly when dealing with unit tests.

6. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + * unit test to test functionality of a class, classes, and/or functions, if
   

   We do not test functions with phpunit, as we do not load them, anything that relies on functions needs ugly workarounds right now...

7. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + * - The test class file must be named and placed under the yourmodule/tests
   + *   directory, according to the PSR-4 standard.
   

   According to PSR-4, they need to be in tests/src I think.

8. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + * - Methods in your test class whose names start with 'test', and which have
   + *   no arguments, are the actual test cases. Each one should test a logical
   + *   subset of the functionality.
   

   I would leave out the arguments part because phpunit does support arguments in combination with data providers. See \Drupal\Tests\Core\Access\AccessManagerTest for example. But I'd avoid explaining that here, just leave that part out.

9. +++ b/core/modules/system/core.api.php
   @@ -620,12 +620,88 @@
   + * - http://phpunit.de/manual/3.7/en/automating-tests.html for general
   + *   information on the PHPUnit framework.
   

   I *think* we're discussing to switch to 4.0 somewhere, so we should make sure this link is updated when we switch. Probably find and cross-post to the relevant issue.

10. +++ b/core/modules/system/core.api.php
    @@ -620,12 +620,88 @@
    + * - For most functional tests, define a class that extends
    + *   \Drupal\simpletest\WebTestBase, which contains an internal web browser and
    + *   defines many helpful test assertion methods that you can use in your tests.
    

    You should only use WebTestBase if you need to do web requests, otherwise use DrupalUnitTestBase, which is *much* faster and recommended to use whenever possible. WebTestBase does a full installation with the requested modules while DrupalUnitTestBase only created a in-memory pseudo-installation where you need manually create the tables you need and include all modules you need, including dependencies and required modules.

11. +++ b/core/modules/system/core.api.php
    @@ -620,12 +620,88 @@
    + * - You may also override the default setUp() method, which can set be used to
    + *   set up content types and similar procedures.
    

    I'd try to generalize this a bit and explain that setUp() can be used to set up the necessary environment that is common to all test methods in a given test class, for example content types (maybe use node types as we're in API documentation? content types is so confusing with nodes, content, entities, and * types)

    This might also be a good place to explain that each test method runs in a new, isolated test environment and can not rely on anything that it does not set up itself.

12. +++ b/core/modules/system/core.api.php
    @@ -620,12 +620,88 @@
    + * - You can define additional modules to be enabled by overriding the
    + *   $modules member variable. In some cases, you may need to make a test
    + *   module to support your test; put such modules under the
    + *   yourmodule/tests/modules directory.
    

    Additional in this case is anything that is not a required module, so defining the necessary modules is almost required (you need to enable yourself too). Maybe rewrite a bit to "You must define all required modules to run the test..." or so?

A very nice summary on the 3 main concepts for tests by @mongolito404 in #2232271-36: [Meta] Use Behat for validation testing:

Unit testing, integration testing and validation testing testing are three different beasts, looking for a single solution for all three of them is doomed to failure (or at least ugly hack and workarounds the limitations of the choosen tool in order to use it in a way it wasn't designed for).

QUnit is a unit testing framework for JavaScript, like PHPUnit is for PHP. So if what is needed are integration and validation tests, something more than QUnit is indeed needed. IMHO, for unit testing in JS, neither a browser or an HTML page are required, but you may need to mock some DOM objects (ie. you don't run your tests over an actual site with real pages). For instance, a properly designed Drupal behavior should be testable by calling it over a set of fixture DOM elements (ie. context of the behavior) and asserting the wanted changes of these elements or their children.

Being a BDD framework, Behat is a more a validation testing framework. It tests an complete web application as a blackbox, not its individual components. So it could be used to validate that the JavaScript on a standard Drupal install works as expected. As a bonus, Behat is also usable to test behaviors defined by both PHP and JavaScript at the same time. An example of validation testing would be testing that a specific Views admin scenario create the expected view.

That leave us with integration testing, which is hard to implement over code that is not designed as loosely coupled components. Unit testing has the same kind of issue, but while it is still possible to unit test individual functions and classes by mocking their (hidden) dependencies, integration testing requires a known logic to assemble components together in order to be able to tests the various integration scenarios.

…in D8, Drupal's JavaScript code is not designed as components (even tightly coupled), making integration testing nearly impossible to implement. So focus on both unit testing … and validation testing … will be more rewarding. Both will encourage more discipline and better quality in the JavaScript code, which should eventually make it possible to implement integration testing.

We can simply replace/remove the JavaScript parts, state PHPUnit as tool for unit testing, DUTB for integration testing, and WebTest for validation/acceptance testing, and the resulting summary will give a quick + concise introduction to the main concepts and tools we're using. :-)

+++ b/core/modules/system/core.api.php
@@ -676,32 +666,51 @@
+ * - For functional tests that do not test web output, define a class that
+ *   extends \Drupal\simpletest\UnitTestBase. This class is much faster than

DrupalUnitTestBase, not UnitTestBase (also the @ingroup), the second should really be deprecated in favor of PHPUnit/UnitTestCase. (There is an issue to convert all test that directly extend from it to phpunit, but it's... complicated (for example due to the not-testing-functions "rule", so things need to be converted to methods/classes first).

See https://drupal.org/node/1829160 for a pretty good change record about DrupalUnitTestBase and the difference to other tests.

Maybe we should have something like developing/api/testing that wraps both phpunit and simpletest documentations which would give us a place to explain the different base classes in more depth and link to detailed instructions for each, for example DrupalUnitTestBase could be based on that change record...

> "mocking" class

Instead: "mock" object.

> You can define additional modules to be enabled by overriding the $modules member variable

I would say by specifying or something like that -- while technically you are overriding the $modules the test framework uses reflection and gathers all the $modules so it's not like a test class extending say a base class will override which modules exist.

> DrupalUnitTestBase

Has been renamed to KernelTestBase

> The class name needs to end in the word Test.

Nope. It needs to be in the right directory/namespace and have a getInfo method. The "ends in the word Test" is just PHPunit making lives harder. Simpletest doesn't create minefields like that.

> * PHPUnit tests can also be run from the PHPUnit framework.

This needs a link somewhere because the only way to do this AFAIK is to change into the core directory and run phpunit from there (or perhaps there's some magic argument to show phpunit where the xml is)

Looks good to me.

Awesome.

Committed and pushed to 8.x. Thanks!