Problem/Motivation
In order to make sure we're not breaking things that production needs, lets get a baseline of some 'end to end' tests that prove that the testrunner runs.
The purpose would be to exercise most of the paths that are essential for operation.
Potential Scenarios:
Note: This list has become a spreadsheet here: https://docs.google.com/spreadsheets/d/1EOXvbC2ir-J7VVxUrnR1xpWePArTr7Gu...
Run a known passing core test against a single test class- Run a failing core test against a single test class
- Run a core test with a broken patch #2340939: Fail test if patch does not apply successfully ✓ (PatchFailTest)
- Run a core test with a valid patch ✓ (CorePatchAppliedTest)
- Run a d7 core test ✓ (PassingSimpletestLegacy7Test)
- Run a D8 contrib test with dependencies on other modules ✓ (ContribDependenciesTest)
- Run a D7 contrib test with dependencies on other modules ✓ (ContribDependenciesLegacy7Test)
- Run a core test with a patch that causes a phpunit segfault (https://www.drupal.org/pift-ci-job/369229)
- Run a core test with a patch that causes a simpletest segfault
- Run a core test with a patch that causes a php fatal error (example: https://dispatcher.drupalci.org/job/default/95621/consoleFull) ✓ (SimpletestPHPFatalTest)
- Run a contrib test against a repo with no tests. ✓ (ContribNoTestsTest)
- Run a contrib test that has composer dependencies
- Run a contrib test with patches
- Run a test that has syntax errors
- Run a matrix of tests for executable container + db container
- Run a test that does not have its @group set ✓ (CoreNoGroupsTest)
#2686531: Expose job data to runcommand
Theres probably more, but this is a start.
Proposed resolution
Add functional tests for the above scenarios.
Note that this issue is a meta. File a new child issue for each test.
Comments
Comment #2
MixologicComment #3
Mixologichttps://dispatcher.drupalci.org/job/default/92908/
Comment #4
mile23Comment #5
mile23Beginning to make a framework for functional tests here: #2683013: Better functional test framework
Once that's in, we can move forward on these cases.
Comment #6
MixologicComment #7
mile23#2680713: Handle missing @groups has a test now.
Comment #8
MixologicPassing test is in.
Comment #9
MixologicComment #10
isntall commentedI've been doing some work on this and I think progress has been made.
Comment #12
jthorson commentedWriting some thoughts here while reviewing the code ... so there is some review, and some 'future work' items mixed in this comment. I recognize that some (or all?) of these are tangential to this issue, but wanted to make sure I captured them without needing to do a full context switch.
1. Usability quirk:
A side effect of this is that running the tests locally overrides your local configuration, which can lead to a bit of a 'wtf' moment the first time it's encountered. I think the test run should leave the system in it's original state once it's done running; so we should add a followup issue to enhance this to save/restore the existing configset. (TODO: Create issue)
2. Test configuration approach:
Another option for this may be to create each of the functional test configurations as configsets of their own (rather than hard-coded fixtures within the class) and use the config:load command to pull up each combination for a test ... these configs could then serve a dual purpose; first for testing, and second as 'example' configs that devs can use as a base to clone/modify/extend for their own configs.
(TODO: Create issue)
3. Test Output: The test output (as displayed in PHPStorm, for example) could use more verbosity, especially for any 'intentional failure' test cases which bail early ... the output for these cases consists of two 'cloning statements' and nothing else. #2725495: Add display_errors to phpunit.xml
4. Scenario Coverage: I updated the list in the issue summary with the applicable test names.
5. Test modules: The tests use a combination of legit, published projects and user-namespaced sandboxes, both of which increase the risk of fragility ... not only due to the risk of code changes within the repo, but also because there is no way to know that those projects/users/etc. are being used in this way from the perspective when looking at those projects/users from a perspective outside of this project. One suggestion would be to create a namespaced user on drupal.org, and generate lightweight namespaced projects/sandboxes providing each of the dependent configurations as needed under that specific user account.
6. Git clones for every test: Ryan and I discussed this briefly at NOLA, and Mile23 expressed this as a concern as well in a comment on the "missing @groups" issue (#2680713-12: Handle missing @groups). This could use a big chunk of optimization.
7. Test description cut and paste issues:
SimpletestPatchFailTest contains the description for ContribNoTestsTest.
SimpletestPhpFatalTest contains the description for ContribNoTestsTest.
CoreNoGroupTest contains the description for CorePatchAppliedTest.
8. Duplication: I don't see a need for both 'PatchFailTest' and 'SimpletestPatchFailTest', which appear to do the same thing.
9. Run a failing core test against a single test class: Not related to the code, but more the issue summary. Ideally, it should be hard to find a commit hash where core has a failing test. ;) This could be done easier in contrib, especially if we adopt the approach suggested in #5.
Bouncing back to 'Needs Work' for #7 and #8 in this list ... otherwise, RTBC.
Comment #13
jthorson commentedComment #15
jthorson commentedFixed the comment changes and removed the duplicate test. Other than the test removal, no other functional changes made ... so I think this is good to go.
Comment #16
MixologicComment #17
MixologicComment #18
mile23Reviewing
2680565-Create-functional-tests-of-run-command-output-to-test-production-essentialsbranch.Comment #20
mile23Some tests need depend on containers, updated
TESTING.mdto reflect this.Removed out-of-scope changes to
RunCommand.Added
@todoabout timing code inJobResults.Removed
TaskInterfacebecause that's the step after the tests are done. :-)CoreD8PassingTestcalledsetUp()for some reason I didn't understand, so I removed that.CorePatchAppliedTesthad a scaffoldfwrite(), removed.Marking this as RTBC so @Mixologic can push back as needed.
Comment #21
mile23Comment #22
mile23Pushed changes from
2680565-Create-functional-tests-of-run-command-output-to-test-production-essentials branchto dev.Leaving the issue open so we can continue adding tests.
Comment #23
mile23Comment #24
legovaerAt one point, the TESTING.md file states that you need to execute
composer install. It seems like that composer has not been installed globally. Therefore we need to execute/home/vagrant/drupalci_testbot/composer.phar installinstead.Comment #25
mile23Well spotted. :-)
Added to dev branch.
Comment #26
mile23Made a spreadsheet of tests here:
https://docs.google.com/spreadsheets/d/1EOXvbC2ir-J7VVxUrnR1xpWePArTr7Gu...
Comment #27
legovaerI've created a new sandbox project in order to test the syntax errors. You can find it here.
Comment #28
MixologicI added a sub issue for the syntax error handling #2809565: Create a functional test that has a syntax error
Comment #29
mile23Converting this to a meta.
Comment #30
mile23Comment #31
mile23Comment #33
MixologicIm going to close this as the test coverage we have is adequate, doenst really need its own plan issue, and can be hashed out in individual issues.
Comment #34
Mixologic