Problem

  1. Our coding standards have been blindly taken over for PHPUnit tests, and are rigorously applied now.

  2. Excessive comments do not help in any way, they harm readability of otherwise self-documenting test code:

    /**
     * Tests PHP environment helper class.
     *
     * @group Drupal
     * @group Utility
     * @coversDefaultClass \Drupal\Component\Utility\Environment
     */
    class EnvironmentTest extends UnitTestCase {
    
      /**
       * {@inheritdoc}
       */
      public static function getInfo() {
        return array(
          'name' => 'PHP environment utility helpers',
          'description' => '',
          'group' => 'Utility',
        );
      }
    
      /**
       * Tests \Drupal\Component\Utility\Environment::checkMemoryLimit().
       *
       * @param string $required
       *   The required memory argument for
       *   \Drupal\Component\Utility\Environment::checkMemoryLimit().
       * @param string $custom_memory_limit
       *   The custom memory limit argument for
       *   \Drupal\Component\Utility\Environment::checkMemoryLimit().
       * @param bool $expected
       *   The expected return value from
       *   \Drupal\Component\Utility\Environment::checkMemoryLimit().
       *
       * @dataProvider providerTestCheckMemoryLimit
       * @covers ::checkMemoryLimit
       */
      public function testCheckMemoryLimit($required, $custom_memory_limit, $expected) {
        $actual = Environment::checkMemoryLimit($required, $custom_memory_limit);
        $this->assertEquals($expected, $actual);
      }
    
      /**
       * Provides data for testCheckMemoryLimit().
       *
       * @return array
       *   An array of arrays, each containing the arguments for
       *   \Drupal\Component\Utility\Environment::checkMemoryLimit():
       *   required and memory_limit, and the expected return value.
       */
      public function providerTestCheckMemoryLimit() {
        $memory_limit = ini_get('memory_limit');
        $twice_avail_memory = ($memory_limit * 2) . 'MB';
    
        return array(
          // Minimal amount of memory should be available.
          array('30MB', NULL, TRUE),
          // Exceed a custom (unlimited) memory limit.
          array($twice_avail_memory, -1, TRUE),
          // Exceed a custom memory limit.
          array('30MB', '16MB', FALSE),
          // Available = required.
          array('30MB', '30MB', TRUE),
        );
      }
    
    }
    

    1. All of the extra phpDoc suggests that something special would be going, but this is just a regular test using a data provider.
    2. Insane amount of repetition of the namespace and class name of the class being tested.
    3. The connection between testCheckMemoryLimit()providerTestCheckMemoryLimit() is lost in a sea of useless comments.
  3. PHPUnit tests are most minimal units of test cases:

    1. Each unit test method maps to exactly one method on the test class:
      FooTest::testBar()Foo::bar()

    2. Multiple unit test methods can map to the same test class method:
      FooTest::testBarInvalid()Foo::bar()

    3. Data providers simplify argument permutation testing:
      FooTest::providerTestBar()FooTest::testBar()Foo::bar()

    4. PHPUnit's native @coversDefaultClass + @covers document the rest.

    None of this needs any additional documentation, because it's all regular PHPUnit test authoring practice.

  4. It is pointless and unhelpful to document anything else on top of self-descriptive + self-documenting unit test code.

Proposed solution

  1. Stop the coding standards nonsense for PHPUnit tests. Write self-descriptive + self-documenting unit tests instead.

    Identical test code as above, but readable this time:

    /**
     * Tests PHP environment helper class.
     *
     * @group Drupal
     * @group Utility
     * @coversDefaultClass \Drupal\Component\Utility\Environment
     */
    class EnvironmentTest extends UnitTestCase {
    
      public static function getInfo() {
        return array(
          'name' => 'PHP environment utility helpers',
          'description' => '',
          'group' => 'Utility',
        );
      }
    
      /**
       * @covers ::checkMemoryLimit
       * @dataProvider providerTestCheckMemoryLimit
       */
      public function testCheckMemoryLimit($required, $custom_memory_limit, $expected) {
        $actual = Environment::checkMemoryLimit($required, $custom_memory_limit);
        $this->assertEquals($expected, $actual);
      }
    
      public function providerTestCheckMemoryLimit() {
        $memory_limit = ini_get('memory_limit');
        $twice_avail_memory = ($memory_limit * 2) . 'MB';
    
        return array(
          // Minimal amount of memory should be available.
          array('30MB', NULL, TRUE),
          // Exceed a custom (unlimited) memory limit.
          array($twice_avail_memory, -1, TRUE),
          // Exceed a custom memory limit.
          array('30MB', '16MB', FALSE),
          // Available = required.
          array('30MB', '30MB', TRUE),
        );
      }
    
    }
    

Comments

sun’s picture

Issue summary: View changes
ParisLiakos’s picture

shouldnt this move to ^ ?
i also find the docs in test methods completely useless..but the @return for dataProviders methods might be useful when you revisit it later and want to check the correct order when adding new datasets, without having to scroll to the test method

sun’s picture

Hm, perhaps. That issue has a really large scope though, and only marginally touched the topic of phpDoc thus far...

Mile23’s picture

I think documenting @return for data providers is a good idea, because it's not always obvious, and it's very cumbersome to work out how to do inline comments in a readable/elegant way.

Saying something like:

/**
 * @return array
 *   Nested array of test data that looks like this:
 *    - Expected result.
 *    - Function's first argument.
 *    - Function's second argument.
 *    - etc.
 */

...can really help keep it sorted, and basically documents the input arguments to the test method. Also helps you write better tests.

For the rest, yah, general agreement, though no one should have to amend their patch to remove documentation. :-)

sun’s picture

The general problem space of data provider methods is that the arguments are typically noted in reversed order (compared to assertEquals()).

If we'd recommend to return $expected first + accept it first, then it would become obvious again (because there can only be 1 return value):

  public function testCheckMemoryLimit($expected, $required, $custom_memory_limit) {
    $actual = Environment::checkMemoryLimit($required, $custom_memory_limit);
    $this->assertEquals($expected, $actual);
  }
  public function providerTestCheckMemoryLimit() {
    return array(
      array(TRUE, '30MB', NULL),
      array(TRUE, $twice_avail_memory, -1),
      array(FALSE, '30MB', '16MB'),
      array(TRUE, '30MB', '30MB'),
    );
  }

In essence, it's about 1 return value and 1-n positional arguments. In abstract:

  public function testSomeMethod($expected, $arg1, $arg2, ...) {
    $this->assertEquals($expected, Class::someMethod($arg1, $arg2, ...));
  }
  public function providerSomeMethod() {
    return array(
      array(expected, arg1, arg2, ...),
    );
  }

By following this notation, additional @return phpDoc for the data provider inherently becomes superfluous.

Mile23’s picture

I get what you're saying, but there needs to be some record of the line of thinking beyond the PHP. Especially when the data doesn't necessarily represent a 1:1 relationship with the calling signature.

For instance: If I have to assemble a tree of mock objects in the test method, based on inputs from the dataprovider, the 1:1 relationship can get lost. We might say there should be a requirement for building that mock tree in the dataprovider, but then the dataprovider is unreadable because it's a huge block of getMock() calls. (Also takes more memory, since PHPUnit does a foreach() on the whole thing.)

Seems like at least documenting @returns from the dataprovider is a happy medium, and we have a decent way to mark up a list in doxy. We can also make it a rule of thumb that the dataprovider's data structure should be as close to the calling signature as possible, with $expected followed by $arg1, $arg2, etc.

jhodgdon’s picture

Title: Fix coding standards for PHPUnit tests » [policy] Fix coding standards for PHPUnit tests

People, when you file a coding standards discussion issue:

a) Please put [policy] in the issue title.
b) Use the "coding standards" tag (which was done here).

YesCT’s picture

this might have actually been intended to be the issue to *fix* existing phpunit tests... if so, it's not [policy] (needs title change back) but postponed on the policy issue.

jhodgdon’s picture

Title: [policy] Fix coding standards for PHPUnit tests » [policy, then patch] Fix coding standards for PHPUnit tests

If there is a separate policy issue, please link it here. Until then, we can make this an "agree on the policy and then make a patch" issue, as is often done.

sun’s picture

IMHO, #2057905: [policy, no patch] Discuss the standards for phpunit based tests tries to attack too many things at once — it looks it has agreement on structural/organizational/policy questions already, so we should simply go ahead with those and document them.

But if we continue to mix also coding standards into the other issue (whereas that topic was not really covered in the discussion at all yet), then it will take forever to complete.

This issue here is primarily meant as a policy issue so we do not continue this madness for new tests... — We could create a separate (novice) issue afterwards to change the existing tests accordingly, but that would be of very low priority.

jhodgdon’s picture

Project: Drupal core » Drupal Technical Working Group
Version: 8.0.x-dev »
Component: phpunit » Documentation

Coding standards decisions are now supposed to be made by the TWG

tizzo’s picture

Project: Drupal Technical Working Group » Coding Standards
Component: Documentation » Coding Standards
quietone’s picture

Title: [policy, then patch] Fix coding standards for PHPUnit tests » [policy, no patch] Fix coding standards for PHPUnit tests
Parent issue: » #2057905: [policy, no patch] Discuss the standards for phpunit based tests
Related issues: -#2057905: [policy, no patch] Discuss the standards for phpunit based tests

Changing the parent to the main meta about PHPUnit tests.