Problem/Motivation

There are several things phpUnit can view as "Risky". Generally these are things that are prone to failure, false positives, and/or false negatives. This patch fixes tests that fall into 2 of those categories.
1) Tests that don't assert anything
2) Tests that change global state.

Most of the missing assertions where tests that called ->fail directly on a failure condition instead of asserting the condition.
Most of the global state tests I just cleaned up after since they actually used them.

Pretty straight forward.

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Task because its just hardening/cleaning up tests
Issue priority Minor, tests should all probably work and continue working without these changes.
Unfrozen changes Tests
Disruption Should be none. Minor changes to tests and tests that are unlikely to be touched by other issues.

Comments

dawehner’s picture

Yeah, we don't have any risky tests :)

jibran’s picture

ROFL

neclimdul’s picture

StatusFileSize
new9.29 KB

lol, cone of shame

mile23’s picture

:-)

+1 for the concept. Don't have time for a proper review ATM.

mile23’s picture

Status: Needs review » Needs work

It looks like the way to repro this is to:

  • Apply the patch here: https://www.drupal.org/node/2105583#comment-10167708
  • Run PHPUnit cd core;./vendor/bin/phpunit -v. (Include -v to get PHPUnit to tell you which tests are risky.)
  • Then you apply the patch in #3, run PHPUnit again, and the risky notices go away.

Which is what happens, so yay, mission accomplished.

Some review stuff:

  1. +++ b/core/tests/Drupal/Tests/Core/EventSubscriber/SpecialAttributesRouteSubscriberTest.php
    @@ -84,6 +84,7 @@ public function testOnRouteBuildingValidVariables(Route $route) {
         $route_collection->add('test', $route);
         $event = new RouteBuildEvent($route_collection, 'test');
         $this->specialAttributesRouteSubscriber->onAlterRoutes($event);
    +    $this->assertTrue(TRUE, 'No errors where thrown');
    

    Not really a test of much, is it? You've swapped one kind of risky test for another. :-)

    Rather than assertTrue(TRUE), we should ignore $this->specialAttributesRouteSubscriber and mock our own. That way we can set expectations for all the stuff SpecialAttributesRouteSubscriber::onAlterRoutes() does.

  2. +++ b/core/tests/Drupal/Tests/Core/Render/RendererTestBase.php
    @@ -140,6 +145,10 @@ protected function setUp() {
    +  public function tearDown() {
    +    unset($this->currentUserRole);
    +  }
    +
    

    currentUserRole is a string... Why unset it? It's about to go out of scope anyway.

neclimdul’s picture

Status: Needs work » Needs review
StatusFileSize
new8.33 KB
new1.24 KB

1) It sort of does something because the phpunit error handler throws and exception but yeah its pretty weak. #2364467: SpecialAttributesRouteSubscriber::onAlterRoutes() doesn't return a value does it better so it just took it out.

2) I don't know what that's from but I re-factored that code several times so its probably useless tinkering. I can't see a point.

mile23’s picture

Status: Needs review » Reviewed & tested by the community

#2364467: SpecialAttributesRouteSubscriber::onAlterRoutes() doesn't return a value. Nice. An issue I'd forgotten I made, lost to the ravages of antiquity, reborn to new life though the use of the search function!

This looks good, and we can then argue about mocking in that other issue. :-)

wim leers’s picture

StatusFileSize
new7.8 KB
new1.32 KB

Fixed my remarks so this can stay at RTBC:

  1. +++ b/core/tests/Drupal/Tests/Core/Render/RendererPlaceholdersTest.php
    @@ -434,7 +434,8 @@ public function testScalarLazybuilderCallbackContext() {
    -    $this->renderer->renderRoot($element);
    +    $result = $this->renderer->renderRoot($element);
    +    $this->assertEquals('<p>This is a rendered placeholder!</p>', (string) $result);
    

    These changes don't belong here. That's not what the test is testing. Should be reverted.

  2. +++ b/core/tests/Drupal/Tests/Core/Render/RendererTestBase.php
    @@ -80,6 +80,11 @@ class RendererTestBase extends UnitTestCase {
       /**
    +   * @var string
    +   */
    +  protected $currentUserRole;
    

    Missing docs.

neclimdul’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new8.48 KB
new699 bytes

Appreciate the effort but that adds back in a the risky test we where fixing :(.

This seems more correct assertion though.

wim leers’s picture

@neclimdul: huh? That test only is the inverse of RendererPlaceholdersTest::testInvalidLazyBuilderArguments(): it asserts that no exceptions are thrown when only scalar arguments are specified for the #lazy_builder. It should not assert any of the output.

Where the risk there? There's no global involved.

mile23’s picture

Status: Needs review » Needs work

The risky part is that there are no assertions in that test. There is no expected behavior, since we can't specify that no exception was thrown.

One solution is to decide on some behavior to test, and having it match the documented return type and value is reasonable and it gives us information in the future if things change.

Another way to do it would be a try/catch block with the catch failing. This more explicitly says what you're testing, so it might be better.

Or, you know.. Combine the two. :-)

This patch fixes all the risky tests except for SpecialAttributesRouteSubscriberTest::testOnRouteBuildingValidVariables(), which is marked as risky after applying the patch at https://www.drupal.org/node/2105583#comment-10176962

neclimdul’s picture

Yeah from IS

1) Tests that don't assert anything.

Generally I think of this as needing a basic assertion to say "the code ran." Ideally "does not throw an exception" tests should be _very_ specific though and other unit tests will cover this 99% of the time.

I'm not sure from the last two comments what work needs to be done. If we need to separate this test from the other cleanups I'm ok with making another issue to clean it up like we did with the Subscriber though.

wim leers’s picture

Status: Needs work » Reviewed & tested by the community

Ahhh! That makes sense. Thanks :) Sorry for missing that!

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Test only change. Committed 4710b09 and pushed to 8.0.x. Thanks!

  • alexpott committed 4710b09 on 8.0.x
    Issue #2542486 by neclimdul, Wim Leers, Mile23: Fix risky phpunit tests
    

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.