Parent issue: #2866082: [Plan] Roadmap for Simpletest
Problem/Motivation
Our phpunit support within run-tests.sh is generally brittle and difficult to maintain, which is why we have experimented with other test runners such as paratest and Symfony's simple-phpunit.
This issue supports these goals:
- Isolate and remove the hard dependencies between run-tests.sh and the simpletest module.
- Refactor some parts of simpletest module into a testable class so it can be more easily maintained.
- Facilitate architectural changes to the testing system, whatever they might be.
There is some overlap between this issue and #2905007: Allow run-tests.sh to report skipped/risky/incomplete PHPUnit-based tests, where a JUnit helper class will be introduced once it's complete.
Proposed resolution
Refactor simpletest_run_phpunit_tests()
and its simpletest_phpunit_*()
friends to a class which doesn't depend on simpletest module.
Also refactor the JUnit conversion to a utility class to support , so the class doesn't need a require_once('simpletest.module')
.
Remaining tasks
- Add @trigger_error() to deprecated methods.
- Remove usages from run-tests.sh.
User interface changes
API changes
Data model changes
Comment | File | Size | Author |
---|---|---|---|
#116 | 2641632-116-interdiff.txt | 6.8 KB | kim.pepper |
#116 | 2641632-116.patch | 51.49 KB | kim.pepper |
#111 | interdiff.txt | 739 bytes | Mile23 |
#111 | 2641632_111.patch | 51.27 KB | Mile23 |
#107 | interdiff.txt | 23.28 KB | Mile23 |
Comments
Comment #2
Mile23Not rocket science. :-)
This just moves all the code from the functions to methods on a class. Add some dependency injection and we're done.
I left all the
simpletest_*()
functions in place for BC.@todo: Mark those functions as @deprecated.
@todo: Write some tests.
Comment #4
Mile23The failure relates to
SimpletestPhpunitRunCommandTest
, which I have to say is a super-awesome great test to have, but it's full of bugs, so I'm changing it here.It looks like this:
It sets the
file_system
service to deal with the temp file directory, which is good.Then it generates a temp file and uses its name as the test id, which is bad, because the test id is supposed to be an integer, and now we have an unneeded file in tmp.
It also uses the same id for two test runs, which means the files will overwrite themselves and thus we lose the benefit of expecting that two files will be generated.
I changed the test so that it uses a proper integer for the test id. I refactored the
include_once
rigamarole tosetUp()
.It now has near-identical tests of both the
simpletest_*()
function behavior, and also the newPhpUnitTestRunner
behavior, in order to prove backward compatibility.Minor changes to the test runner include service dependencies (so we only need the working path, and not the
file_system
service).Also a pure unit test of
runTests()
.Comment #5
Mile23Filed a separate issue for fixing the test: #2643624: Fix SimpletestPhpunitRunCommandTest
Comment #7
Mile23Setting to 8.1.x since this is a testing improvement.
Rerolling.
Comment #8
dawehnerRather than spending time here it would be better IMHO to get rid of the simpletest <-> phpunit integration and rather provide a phpunit wrapper script which fullfills the purposes of the testbot.
Comment #9
Mile23When someone tells me that it's not run-tests.sh and simpletest UI's job to run phpunit tests, then sure.
Until then: Please review. :-)
This issue is secondary to #2608532: Simpletest UI munges PHPUnit-based test output anyway.
Comment #10
dawehner@Mile23
Do you understand my point aka. the time invested into rewriting those bits is less effective than getting rid of it in the first place? No question though in general, the new code is much better, especially better tested.
Comment #11
Mile23Yes.
However, we have it and it sucks. We need it because PHPUnit doesn't handle fatal errors very well and so the testbot can't report it, so we have to use run-tests.sh. Which, btw: #2624926: Refactor run-tests.sh for Console component.
We *could* make our own PHPUnit test runner class which might do a better job with fatals than
\PHPUnit_Framework_TestSuite
under run-tests.sh, in which case this issue is the first reviewable step of refactoring.We also need to run PHPUnit tests from simpletest UI. I've only seen an issue which wants to hobble simpletest UI so you can't run PHPUnit tests, but I fixed that in #2608532: Simpletest UI munges PHPUnit-based test output so why would we break our own tools?
As I mentioned, I solved the problem in #2608532: Simpletest UI munges PHPUnit-based test output without this refactoring, but this refactoring would make it easier to solve future issues, including your concerns. So it's a nice-to-have. And I'd be into doing a bunch of the work I just outlined in this comment, but not if I have to call
simpletest_phpunit_drupal_core_devs_think_you_are_wasting_your_time()
. :-)Comment #12
dawehnerThanks a ton for trying to improve those things! A ton!
Well, what we actually just need is a small wrapper script for phpunit which does the handling of the fatals. https://github.com/brianium/paratest kinda provides that in some way already.
I'll review your work later, no worries.
Comment #13
joachim CreditAttribution: joachim at Torchbox commentedJust a couple of formatting nitpicks:
@file docblocks are deprecated for classes.
Stray whitespace.
Comment #14
Mile23Thanks.
Bumping to 8.2.x because it improves tests and that's allowed during beta.
Needed a re-roll. Also needed some changes to reflect #2575725: Run PHPunit tests one at a time via UI to avoid exception when selecting too many which I hope I did completely.
Comment #15
Mile23This is probably too disruptive for 8.2.x during RC/beta times. Moving to 8.3.x. The patch still applies but the deprecation messages will need to change. Re-running the tests.
Comment #16
dawehnerI really like separating the concerns here. It makes things much easier to follow.
Conceptually I'm wondering whether its better to make those methods non static, especially when you think about a converter as some actual existing thing you can initialize.
Do you mind open a follow up to discuss dropping this function? This is not actually used anywhere in core and is actually conceptually wrong :) We should pick of core/phpunit.xml as well
Let's keep the additional space :)
Nitpick: Is there a reason we use snake_case :)?
Comment #17
Mile23Thanks for the review.
#16.1: The reason
JUnitConverter
is a collection of static methods is so it can have BC with functions likesimpletest_phpunit_xml_to_rows()
. We move the functions to a class so that we can autoload the class instead of doing therequire_once
forsimpletest.module
. The functions don't keep state so we don't need the converter class to keep state. I wager that in the future, if we're abandoning simpletest, we won't need a way to convert between JUnit and {simpletest} schema, so let's not try too hard. If we discover that we still need {simpletest} then we can modifyJUnitConverter
to our whims.#16.2: Dead code in Simpletest. Who'da thunk? #2798273: simpletest_phpunit_configuration_filepath() is dead code. Deprecate it.
#16.3: Done.
#16.4: Fixed case of that variable. Also I realize I never set this issue as related to #2643624: Fix SimpletestPhpunitRunCommandTest
Comment #18
joachim CreditAttribution: joachim as a volunteer commentedA little nitpicking, but other than that looks good to me.
These are no longer used.
> + * Finds all test cases recursively from a test suite list.
In passing -- some of the docs here aren't very clear. But best to keep this a clean refactor and improve docs later.
All container-aware classes I've seen so far call it $container. For DX I think that should be kept consistent.
Comment #19
Mile23Agreed about the docs. They're not as great as they could be. That example, though, makes sense if you know how JUnit works. :-)
Here are the other items in a patch.
Comment #20
dawehnerThis makes certainly the code better to follow. While I personally think focusing on converting away from simpletest would be a better invest of time, and experiment with paratest for example, but for sure, this patch, as it is, is a nice thing!
Can we mark those classes explicit as internal? IMHO they are not meant to be used outside of core, or are they?
Nitpick, let's start with "\" if possible :)
Comment #21
joachim CreditAttribution: joachim as a volunteer commented> Can we mark those classes explicit as internal? IMHO they are not meant to be used outside of core, or are they?
If you mean with an @internal tag, then that's not actually yet part of our standard!
Comment #22
dawehner@joachim
Well, I'm more of a fan of structuring our standards around the idea what people actually use vs. the other way round.
The concept of
@internal
kinda exists already.Comment #23
joachim CreditAttribution: joachim as a volunteer commentedIndeed, but because it's not been set in our standards, API module doesn't support it yet...
Anyway, the boat's sailed on this, so I don't think adding more instances of this is a problem.
Comment #24
dawehner@joachim
Thank you for that :)
Comment #26
Mile23Probably needs an update after #2799021: Ensure a failing PHPUnit test shows enough information via run-tests.sh
Comment #27
Mile23Comment #28
Mile23Well, the thing is... They should be internal to the extent that they're almost guaranteed to change however we move forward with simpletest. We only have the JUnit converter in order to move data between phpunit test logs and {simpletest} schema. Which, btw, we assume we understand here without checking. :-) If we move towards PHPUnit only, then we likely won't need this conversion.
So yah, @internal for both classes.
Rerolled for changes linked above.
Comment #30
dawehnerCool, yeah there are areas which really simply shouldn't be part of an external API, but still, of course well documented.
Do you understand the three failures?
Comment #31
Mile23Yes, I understand the fails. Who put all those unit tests in there, anyway????
What I really don't understand is why we're now doing the same thing under two conditions:
But that's out of scope here.
Comment #32
Mile23Er, forgot the patch. :-)
Comment #33
dawehnerUbernitpick: 80 chars line, but annoying for commiters to fix it :(
Good observation. Well open an issue and carry on :) We solved an issue by introducing a smaller one
Comment #34
jhedstromAnything left to be done here? The cleanup/conversions look really great. +1 for RTBC here.
Comment #35
Mile23Addressed #33, plus a couple more coding standards things.
Comment #37
Mile23Fixed failing test.
Comment #38
Mile23Comment #39
klausiThis makes sense, the biggest problem I have with the patch is that it does not actually remove usages of the now deprecated simpletest_*() functions. I think we should do it in this issue right away, but we can also do it as follow-up.
Some random nitpicks:
should use the full qualified class name.
data type should be "string[]", right?
leading "\" missing
we don't really use this annotation and of course your test covers some code? I would remove it.
Comment #40
Mile23Thanks.
Let's do replacements as a follow-up since this is the testing system: #2808693: Remove usages of deprecated simpletest_phpunit*() and xml-related functions, turn PhpUnitTestRunner into a service
#39.4: The thing is... If you say
@covers ::simpletest_run_phpunit_tests
, the standards listener spits back thatsimpletest_run_phpunit_tests()
doesn't exist, because of some class loading weirdness. Improving the@covers
fortestSimpletestPhpUnitRunCommand()
, removing@coversNothing
fortestSimpletestPhpUnitRunCommandBackwardsCompatibility()
.Addressed other points as well.
Comment #41
klausiok, that works for me.
The number of assertions above matches what the branch results in 8.3.x returns, so our test system still works as it should.
Comment #42
Mile23Comment #44
Mile23Rerolled for modernity's sake.
Updated
PhpUnitTestRunner::runTests()
to match changes tosimpletest_run_phpunit_tests()
.Changed
PhpUnitTestRunnerTest
to use the newTestStatus
.Comment #45
klausiLooks good!
Comment #46
alexpottThese changes look sensible for 8.x whilst we have to maintain Simpletest. Just one thing...
Why not just simpletest.phpunit_runner to simpletest.services and use \Drupal in the procedural code?
Comment #47
Mile23I'm reticent to turn it into a service because that means we have to __construct() with service classes instead of just strings, which means adding a factory service class, which adds complexity. The service would only ever be used by procedural code which is deprecated, and simpletest UI which might be extinct soon.
Also PhpUnitTestRunner is not a service of simpletest (the point of this issue), and it shouldn't be a service of core (since it should only be available during testing), so therefore who defines it?
Let's talk about doing all that as a follow-up. We already have #2808693: Remove usages of deprecated simpletest_phpunit*() and xml-related functions, turn PhpUnitTestRunner into a service which I'll amend to include service-i-fying.
Baby steps...
Comment #48
klausiAgreed, turning this into a service can be done later. The constructors of the classes seem to be sufficiently decoupled from the container with the create() method. So there should not be API changes when we turn this into a service, where ever it might live.
Comment #50
Mile23Rerolled/updated to reflect changes to
simpletest_phpunit_run_command()
.Comment #52
Mile23The error is: "Slave went offline during build."
Re-running.
Comment #55
Mile23Missed
$output
. Ewps.Comment #56
Mile23OK. That's just weird. Fixed. Please ignore #55.
Comment #59
Mile23Re-running tests for the patch from #56.
Sorry about the bad file names.
Comment #61
klausiPatch does not apply anymore.
Comment #62
Mile23Rerolled for 8.4.x.
Comment #63
Mile23Can I not make a patch any more? Sheesh.
Pls ignore #62.
Comment #65
Mile23Reroll, updated deprecation messages.
Comment #67
Mile23Fixed the one typo that make 1754 tests fail.
Also fixed CS error reported by the testbot.
Comment #68
Munavijayalakshmi CreditAttribution: Munavijayalakshmi at Valuebound commentedLine exceeding 80 characters.
Fixed and attached new patch.
Comment #69
klausiLooks good, back to RTBC.
Comment #71
joelpittetTestbot hiccup
Comment #72
Mile23Comment #74
Mile23Needed a re-roll after #2850797: Prepare our phpunit tests to be BC compatible with phpunit 5.x/6.x
Comment #77
Mile23Rerolled for 8.5.x.
The follow-up will be to add @trigger_error() and remove usages from run-tests.sh.
Comment #78
Mile23Le patch.
Comment #79
Mile23Comment #81
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedHere's the error reported by Drupal CI for the patch in #78:
Comment #82
Mile23run-tests.sh
filters againstPHPUnit\Framework\TestCase
instead of\PHPUnit_Framework_TestCase
, so we can enforce changes that will make it easier to eventually use PHPUnit 5+.But that's OK because we're supposed to test
Core
namespace stuff againstDrupal\Tests\UnitTestCase
anyway.Comment #83
joelpittetThanks for resolving that, back to RTBC
Comment #84
Mile23Noticed that I pasted in the wrong issue number in the recent issue summary update. Fixed.
Comment #85
xjmNice patch. Just some nits so far.
Can we be a bit more specific? Edit: I mean about the "in a format". Usually we actually want to document what the actual keys are in an associative array. Since these are moved lines, followup would be best.
"Optional" should not be capitalized here. Does this exist in HEAD or?
Can has followup for "array of what".
Sounds like it's a
string[][]
? But actually we probably shouldn't be changing this hunk at all here; presumably the only changes should be gutting and deprecating the function, rather than improving its docs.Minor: I think this needs a leading
\
so's not to beDrupal\thing\Drupal\thing
. (In these hunks and elsewhere in the patch.)Make it die! ☠️
Comment #86
Mile23All those docblocks were from HEAD, but we just had this: #2722699: Fix Drupal.Commenting.FunctionComment.InvalidReturnNotVoid which seems to have improved things a little, creating a need for a reroll.
And that's why we should have a follow-up, too, since there are a ton of CS improvement issues that will probably touch simpletest.module.
Follow-up: #2909306: Improve docblocks for *_phpunit_* simpletest functions and/or PhpUnitTestRunner and JUnitConverter
Added leading \ to namespaces in deprecation docs.
Comment #87
borisson_We now have a followup for those docs improvements, so setting back to rtbc.
Comment #88
Mile23Comment #90
alexpottI think
\Drupal::service::...
can be replaced with$this->workingDirectory
Also I deliberated
findTestcases
vsfindTestCases
andxmlLogFilepath
vsxmlLogFilePath
currently in coreTestCases
andFilePath
are way more prevalent but the current patch is converting the snake case to camel case so the new names are in line with the old names. Not sure - but I think now might be a good time to bring the names into line with the rest of core. Unfortunately this also means replacing things like$testcases
with$test_cases
- however this is also totally in line with PHPUnit too.Comment #92
Mile23Rerolled for 8.6.x. Addressed case concerns from #90... The interdiff should tell that story.
Comment #93
Mile23Needs a change record, and the deprecation messages will need to point to it.
We should probably go ahead and add @trigger_error() too, since really only core should be calling these functions.
Comment #94
Mile23Added CR. Added @trigger_error(), fixed some missed instances.
Moved UiPhpUnitOutputTest to @group legacy. See #2932909: Convert web tests to browser tests for Simpletest module for more about that edge case.
Comment #95
Mile23This is a blocker for #2905007: Allow run-tests.sh to report skipped/risky/incomplete PHPUnit-based tests
Comment #96
Mile23Comment #99
Mile23Updated IS, added related issue. No longer a blocker for #2905007: Allow run-tests.sh to report skipped/risky/incomplete PHPUnit-based tests which includes a refactored JUnit helper class.
Comment #100
Mile23Some planning happening in this meta.
Comment #101
Mile23Rerolled.
Comment #102
kim.pepperLooking good.
Just a couple of nits:
Why do we need to support NULL? Why not just an empty array?
Should this implement \Drupal\Core\DependencyInjection\ContainerInjectionInterface ?
We should be using the standard deprecation format.
Comment #103
Mile23Thanks.
Addresses #102.
Comment #104
kim.pepperLGTM!
Comment #105
larowlanwe can return here and avoid the else
do we have an issue for this? is this still an open item for this patch?
should we also test for the phpunit.xml file which should take precedence? or is that an existing issue and not changed here?
same here re early return.
Comment #106
Mile23#105.1 & 4: Dun.
#105.2: Comes from this: #1963022: Problem with PHPUnit test results when using @dataProvider which was closed in 2014. So in 5 years no one has objected to using 'Other.' I think we're cool.
#105.3: You know what's funny? #2798273: simpletest_phpunit_configuration_filepath() is dead code. Deprecate it. We don't need this method on the
PhpUnitTestRunner
object. it must have come back in a flurry of copypasta.Closing #2808693: Remove usages of deprecated simpletest_phpunit*() and xml-related functions, turn PhpUnitTestRunner into a service because we should do the replacement here.
Slight uptick in test coverage.
We don't have much coverage of
JUnitConverter
, which tempts me to go ahead and refactor a little and/or just mark a lot of stuff@internal
. But really we have an issue here because simpletest-module-as-contrib will need a consistent API that uses all this for the batch runner, and that's a pain.Also, I realize that we're overlooking
simpletest_summarize_phpunit_result()
and that whole chain of results form building functions. I think in the interest of limiting scope creep that's a follow-up, or will just happen alongside moving simpletest to contrib.(run-tests.sh is a fake shell script that has a theming layer....)
Comment #107
Mile23Deprecated
simpletest_summarize_phpunit_result()
, folded it intoPhpUnitTestRunner
.Everything's
@internal
. We'll un-@internal some things for the eventual contrib simpletest module, or if someone shows up in issues to demand an API for their contrib module.Updated CR to point out that this is all
@internal
. :-)Added unit tests for stuff.
The follow-up I was thinking about in #106 turns out to exist already in #3074433: Decouple run-tests.sh from the simpletest module because run-tests.sh needs to be able to display a results form.
Comment #108
Mile23Comment #109
kim.pepperA micro-nit:
This can just be @inheritdoc now.
Comment #110
LendudeRan some tests using the Simpletest UI and using run-tests.sh with this patch applied and the results are identical with and without the patch (in my small sample anyway).
Comment #111
Mile23Addresses #109.
Comment #112
kim.pepperComment #113
Mile23Comment #114
larowlanComment #115
larowlanLooks like we have some phpcs regressions here
Comment #116
kim.pepperphpcs seems to get confused by having two
[[
together. This patch just splits them out onto new lines.Comment #117
Mile23Thanks, @kim.pepper. I think splitting them out like that is our CS anyway. Meaning phpcs isn't the confused party here.
Comment #118
Lendude#116 looks good, feedback has been addressed, back to RTBC.
Comment #120
larowlanCommitted 76f3729 and pushed to 8.8.x. Thanks!
🎉 published the change record