Contrib Testing Test discovery not working for d6/d7

Manually scanning for the files caused all sorts of conflicts on the old test runners ... every directory change and/or PSR standard change would break testing, and force us to go update the code on the test runners.

Because each testing framework is going to have it's own syntax, format, and default locations, I would prefer to leave 'test discovery' as a function of the individual testing frameworks, rather than building in this somewhat arbitrary discovery logic into DrupalCI, and then adjusting it for other testing frameworks that we try to integrate in the future.

So to address the test discovery problem identified here, my proposed solution would be that run-tests.sh be augmented to support a --directory argument, and have *it* perform it's own test discovery; rather than the hacky method used by PIFT/PIFR (which scanned for all test files, and then passed the complete list of discovered test files as individual command line arguments; resulting in a fifty-line command invocation).

+1

This would also solve the weird problem that testbot finds unported 7.x tests in 8.x modules and completely breaks on that. Would have to be backported to 7.x as well before it could be used and it needs to work on all operation systems no idea if recursive glob() or something like that works on windows?

Since the code has to be migrated anyway ... the 'right' thing to do would be to migrate it only once - into run-tests.sh instead of into DrupalCI.

Test discovery in PIFT consists of parsing over the appropriate directories, generating a string of all of the valid test classes which need to be passed onto the run-tests.sh command invocation, and then executing run-tests.sh with that complete list as individual arguments on the command line. It generates a ton of noise that will severely impact readability of the test results, and degrades the overall end user experience.

I would suggest that moving this over to run-tests.sh actually requires much less effort than porting it into DrupalCI ... as such, I am advocating for *not* trying to port this over into DrupalCI as an 'interim' step. Since run-tests.sh is actually a php script, the same directory parsing code can be put there instead; and as a net-new feature and testing unblocker, I'd argue it would be rather easy for us to fast-track it into the core codebase.

Related core issue (and patch) posted at #2551981: Add --directory option to run-tests.sh test discovery.

Fine ... I understand that we're in a hurry, but am concerned that approach means duplicated effort.

We don't have D6/7 drupalci testing yet, and the 'fast track' patch for D6/7 is a very minor tweak to the already existing D8 patch ... maybe 5 minutes work to port it to D6/7. The bigger question becomes whether the file scan is the best approach for D8.

Patch made it into D8 core in #2551981: Add --directory option to run-tests.sh test discovery - however we may still need our workaround for for D7/D6 depending on how long it takes to backport and get in.