Refactor run-tests.sh for Console component.

This is a first-pass at some of the work in phase 1 mentioned above. It's a WIP. Some things work, some don't.

I've used the file name run-tests.php because it's easier to have run-tests.sh in the next tab to refer to. Obviously we'd change the file name once it's in shape. Or would we? :-)

Because of the way Console works, you have to specify a run-tests command, like this:
php core/scripts/run-tests.php run-tests --list

Yah, thanks. I figured it out. I had to make a single-command application because otherwise there are parsing errors, since run-tests.sh has a lot of options that should be commands.

Still a WIP...

Exception handling is mostly baked. We can't use ConsoleEvents::EXCEPTION handler because that would add complexity in the wrong places. Add that to the @todo list.

Message coloring works with --color which is, sadly, off by default.

How to evaluate:

Apply patch.

Type: php ./core/scripts/run-tests.php --help

Ugly but useable... Still WIP.

Make sure there's an installed Drupal and do something like:

php ./core/scripts/run-tests.php --module node --color

Some tests fail for mysterious reasons, but many do not.

When you're done:

php ./core/scripts/run-tests.php --clean

Here it is patched as run-tests.sh. I'm also going to let the testbot run it, because there are some tests that fail locally in the UI.

The patch git generated is a confuzzled mess so I'll upload the new script as well.

Moved most of the code to \Drupal\Core\RunTests which allows us to start writing unit tests without a lot of require_once mumbo jumbo.

Starting to shim out the direct simpletest.module calls so we can mock them for testing.

See the issue update for more details.

Not entirely sure why #7 is failing. It'd be nice if CI began storing artifacts before the test ran, so we could see what's causing the problem. (There's no log of the exception/fatal.)

Anyway, it all works locally, so here is more lightweight refactoring with some unit tests.

Really not ready for prime-time yet.

Expanded some unit testing, refactored the Config class to facilitate testing.

CI errors still a mystery.

Local testbot says the error is:

$ more testgroups.txt 
exception 'Symfony\Component\DependencyInjection\Exception\InvalidArgumentExcept
ion' with message 'The service file "core/core.services.yml" is not valid.' in /
var/www/html/core/lib/Drupal/Core/DependencyInjection/YamlFileLoader.php:300
Stack trace:
#0 /var/www/html/core/lib/Drupal/Core/DependencyInjection/YamlFileLoader.php(68)
: Drupal\Core\DependencyInjection\YamlFileLoader->loadFile('core/core.servi...')
#1 /var/www/html/core/lib/Drupal/Core/DrupalKernel.php(1169): Drupal\Core\Depend
encyInjection\YamlFileLoader->load('core/core.servi...')
#2 /var/www/html/core/lib/Drupal/Core/DrupalKernel.php(824): Drupal\Core\DrupalK
ernel->compileContainer()
#3 /var/www/html/core/lib/Drupal/Core/DrupalKernel.php(441): Drupal\Core\DrupalK
ernel->initializeContainer()
#4 /var/www/html/core/lib/Drupal/Core/Test/TestRunnerKernel.php(71): Drupal\Core
\DrupalKernel->boot()
#5 /var/www/html/core/lib/Drupal/Core/DrupalKernel.php(686): Drupal\Core\Test\Te
stRunnerKernel->boot()
#6 /var/www/html/core/lib/Drupal/Core/RunTests/RunTestsCommand.php(209): Drupal\
Core\DrupalKernel->prepareLegacyRequest(Object(Symfony\Component\HttpFoundation\
Request))
#7 /var/www/html/vendor/symfony/console/Command/Command.php(256): Drupal\Core\Ru
nTests\RunTestsCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput
), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /var/www/html/vendor/symfony/console/Application.php(838): Symfony\Component\
Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), 
Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/html/vendor/symfony/console/Application.php(189): Symfony\Component\
Console\Application->doRunCommand(Object(Drupal\Core\RunTests\RunTestsCommand), 
Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Cons
ole\Output\ConsoleOutput))
#10 /var/www/html/vendor/symfony/console/Application.php(120): Symfony\Component
\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), O
bject(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/html/core/scripts/run-tests.sh(16): Symfony\Component\Console\Appli
cation->run()
#12 {main}

So the testrunner kernel's container is failing to initialize because there's something strange about how it tries to load core.services.yml.

If anyone has any ideas... :-)

Bug in drupalci. #2629700: Force run-tests.sh to run as www-data to generate the simpletest test list.

IMHO run-tests.sh should work outside of Drupal, given that this is what you want to test in the first place.

Ideally, yes, but I'm only trying to refactor it so it's testable and maintainable, not change any fundamental behavior. So far the patch has a number of unit tests of run-test.sh's behavior, which is the point of the exercise.

Decoupling can come later, along with whatever other changes we desire.

Speaking of which.... :-)

Here's a patch which adds some coding standards stuff, and moves PHP executable discovery away from getenv('_') towards PHP_BINARY. With tests. (The tests were reporting phpunit as the executable when using getenv('_').)

Once the sudo change happens at the testbot level, this will pass and will only need a change away from simpltest_script_underscore method names to camelCase to be acceptable, if not thoroughly tested.

But it will provide a framework by which new behaviors can be tested, and bugs tested against.

It might be that this gets shoved to 8.1.x and that's fine, too.

Bumped to 8.1.x.

Renamed all the under_score names to camelCase. Yanked Console's support for --help, so we only ever see the old help screen we're familiar with.

This is a nice stopping point. Let's call it a DRY normalization, such that the refactored repeated parts are now unit-testable. This is mostly related to test discovery, which is the part I was interested in.

Anyone is welcome to join in, of course. :-)

Next steps:

• Figure out why the old script can access core/core.services.yml just fine under the testbot, whereas this one can't unless you run it as sudo.
• Rename it to run-tests.php. Please.
• Promote e.g. --list and --clean to commands.

That's why I called it a stopping point.

The only real issue at this point is why core.services.yml isn't readable by the testbot. My local testbot builds keep flaking out after one run (don't dare vagrant halt or it's all over) so going is slow.

The wins here are:

• We can let the framework deal with parsing input, so that's code that doesn't need to be maintained.
• Any part of the test runner code can now be refactored more easily, and made more testable.
• More of the behavior of the test runner can be re-used by other components like simpletest module.

#2629700: Force run-tests.sh to run as www-data to generate the simpletest test list. is close to happening, so this refactor will be testable.

A lot of the code of run-tests.sh will have changed, so this won't just be a re-roll if/when work resumes.