Problem/Motivation

In order to make sure we're not breaking things that production needs, lets get a baseline of some 'end to end' tests that prove that the testrunner runs.

The purpose would be to exercise most of the paths that are essential for operation.

Potential Scenarios:

Note: This list has become a spreadsheet here: https://docs.google.com/spreadsheets/d/1EOXvbC2ir-J7VVxUrnR1xpWePArTr7Gu...

Theres probably more, but this is a start.

Proposed resolution

Add functional tests for the above scenarios.

Note that this issue is a meta. File a new child issue for each test.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

Mixologic created an issue. See original summary.

Mixologic’s picture

Issue summary: View changes
Mixologic’s picture

mile23’s picture

Issue summary: View changes
mile23’s picture

Beginning to make a framework for functional tests here: #2683013: Better functional test framework

Once that's in, we can move forward on these cases.

Mixologic’s picture

Issue summary: View changes
mile23’s picture

Issue summary: View changes
Mixologic’s picture

Issue summary: View changes

Passing test is in.

Mixologic’s picture

Issue summary: View changes
isntall’s picture

I've been doing some work on this and I think progress has been made.

  • isntall committed 817c4da on 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials
    Issue #2680565 by isntall: Create functional tests of run command output...
  • isntall committed e2b3c20 on 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials
    Issue #2680565 by isntall: Create functional tests of run command output...
jthorson’s picture

Issue summary: View changes

Writing some thoughts here while reviewing the code ... so there is some review, and some 'future work' items mixed in this comment. I recognize that some (or all?) of these are tangential to this issue, but wanted to make sure I captured them without needing to do a full context switch.

1. Usability quirk:

* This test base class will always use config:load to load the blank configset.
* You then specify DCI_* config values by overriding self::$dciConfig.

A side effect of this is that running the tests locally overrides your local configuration, which can lead to a bit of a 'wtf' moment the first time it's encountered. I think the test run should leave the system in it's original state once it's done running; so we should add a followup issue to enhance this to save/restore the existing configset. (TODO: Create issue)

2. Test configuration approach:

* DCI_* configuration for this test run.
* Override this array with your own config sets and settings.

Another option for this may be to create each of the functional test configurations as configsets of their own (rather than hard-coded fixtures within the class) and use the config:load command to pull up each combination for a test ... these configs could then serve a dual purpose; first for testing, and second as 'example' configs that devs can use as a base to clone/modify/extend for their own configs.
(TODO: Create issue)

3. Test Output: The test output (as displayed in PHPStorm, for example) could use more verbosity, especially for any 'intentional failure' test cases which bail early ... the output for these cases consists of two 'cloning statements' and nothing else. #2725495: Add display_errors to phpunit.xml

4. Scenario Coverage: I updated the list in the issue summary with the applicable test names.

5. Test modules: The tests use a combination of legit, published projects and user-namespaced sandboxes, both of which increase the risk of fragility ... not only due to the risk of code changes within the repo, but also because there is no way to know that those projects/users/etc. are being used in this way from the perspective when looking at those projects/users from a perspective outside of this project. One suggestion would be to create a namespaced user on drupal.org, and generate lightweight namespaced projects/sandboxes providing each of the dependent configurations as needed under that specific user account.

6. Git clones for every test: Ryan and I discussed this briefly at NOLA, and Mile23 expressed this as a concern as well in a comment on the "missing @groups" issue (#2680713-12: Handle missing @groups). This could use a big chunk of optimization.

7. Test description cut and paste issues:
SimpletestPatchFailTest contains the description for ContribNoTestsTest.
SimpletestPhpFatalTest contains the description for ContribNoTestsTest.
CoreNoGroupTest contains the description for CorePatchAppliedTest.

8. Duplication: I don't see a need for both 'PatchFailTest' and 'SimpletestPatchFailTest', which appear to do the same thing.

9. Run a failing core test against a single test class: Not related to the code, but more the issue summary. Ideally, it should be hard to find a commit hash where core has a failing test. ;) This could be done easier in contrib, especially if we adopt the approach suggested in #5.

Bouncing back to 'Needs Work' for #7 and #8 in this list ... otherwise, RTBC.

jthorson’s picture

Status: Active » Needs work

  • jthorson committed 9d29230 on 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials
    Issue #2680565 by jthorson: Removed duplicate test
    
jthorson’s picture

Status: Needs work » Reviewed & tested by the community

Fixed the comment changes and removed the duplicate test. Other than the test removal, no other functional changes made ... so I think this is good to go.

Mixologic’s picture

Issue summary: View changes
Mixologic’s picture

Component: DrupalCI console script » Testrunner Tests
mile23’s picture

Assigned: Unassigned » mile23
Status: Reviewed & tested by the community » Needs review

Reviewing 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials branch.

  • Mile23 committed a3c7352 on 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials
    Issue #2680565: Some cleanup
    
mile23’s picture

Status: Needs review » Reviewed & tested by the community

Some tests need depend on containers, updated TESTING.md to reflect this.

Removed out-of-scope changes to RunCommand.

Added @todo about timing code in JobResults.

Removed TaskInterface because that's the step after the tests are done. :-)

CoreD8PassingTest called setUp() for some reason I didn't understand, so I removed that.

CorePatchAppliedTest had a scaffold fwrite(), removed.

Marking this as RTBC so @Mixologic can push back as needed.

mile23’s picture

Assigned: mile23 » Unassigned
mile23’s picture

Status: Reviewed & tested by the community » Needs work

Pushed changes from 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials branch to dev.

Leaving the issue open so we can continue adding tests.

mile23’s picture

Issue tags: +Dublin2016
legovaer’s picture

At one point, the TESTING.md file states that you need to execute composer install. It seems like that composer has not been installed globally. Therefore we need to execute /home/vagrant/drupalci_testbot/composer.phar install instead.

mile23’s picture

Well spotted. :-)

Added to dev branch.

         // Re-run drupalci init to pick more images.
         $ cd /tmp
         $ git clone https://git.drupal.org/project/drupal.git
+        // Install dependencies for Drupal
         $ cd /tmp/drupal
-        $ composer install
+        $ ~/drupalci_testbot/composer.phar install
         $ cd ~/drupalci_testbot
         // Remove existing containers.
         $ ./drupalci docker-rm containers
mile23’s picture

legovaer’s picture

I've created a new sandbox project in order to test the syntax errors. You can find it here.

Mixologic’s picture

mile23’s picture

Title: Create functional tests of run command output to test production essentials » [meta] Create functional tests of run command output to test production essentials
Issue summary: View changes
Status: Needs work » Active

Converting this to a meta.

mile23’s picture

mile23’s picture

  • Mile23 committed 58b5651 on 2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials
    Issue #2680565: #24 fixed TESTING.md to show proper path to composer
    
Mixologic’s picture

Status: Active » Fixed

Im going to close this as the test coverage we have is adequate, doenst really need its own plan issue, and can be hashed out in individual issues.

Mixologic’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.