run-tests.sh does not handle the error when invalid test groups/classes are specified

Should we show an informative message instead?

+1 for providing an actual helpful message. If you want to be super-fancy you could use
http://en.wikipedia.org/wiki/Levenshtein_distance to provide the probable wanted test.

There is also a fatal error when an unknown test class is specified:

$ sudo -u www php core/scripts/run-tests.sh   --url http://localhost/drupal8x --class "Drupal\block\Tests\BlockTests"

Drupal test run
---------------

Tests to be run:
  - Drupal\block\Tests\BlockTests

Test run started:
  Tuesday, August 12, 2014 - 08:28

Test summary
------------

PHP Fatal error:  Class 'Drupal\block\Tests\BlockTests' not found in /Library/WebServer/Documents/drupal8x/core/scripts/run-tests.sh on line 623
PHP Stack trace:
PHP   1. {main}() /Library/WebServer/Documents/drupal8x/core/scripts/run-tests.sh:0
PHP   2. simpletest_script_run_one_test() /Library/WebServer/Documents/drupal8x/core/scripts/run-tests.sh:36

Fatal error: Class 'Drupal\block\Tests\BlockTests' not found in /Library/WebServer/Documents/drupal8x/core/scripts/run-tests.sh on line 623

Call Stack:
    0.0018     396320   1. {main}() /Library/WebServer/Documents/drupal8x/core/scripts/run-tests.sh:0
    0.4106   14265056   2. simpletest_script_run_one_test() /Library/WebServer/Documents/drupal8x/core/scripts/run-tests.sh:36

FATAL Drupal\block\Tests\BlockTests: test runner returned a non-zero error code (255).

FATAL Drupal\block\Tests\BlockTests: Found no database prefix for test ID 3. (Check whether setUp() is invoked correctly.)
Test run duration: 0 sec

Changed the title appropriately. Retitle if we want this issue to cover unknown individual tests (test classes).

This will error and exit on invalid test groups or test classes and report which group or class is invalid. I love the Levenshtein distance idea -- "Did you mean ...?"

This patch also calculates likely matches based on the Levenshtein distance. Please credit @ednawig also on commit.

Test group example:

$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x fields
  ERROR: Test group not found: fields
  ERROR: Did you mean?
  ERROR: field

Test class example:

$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x --class "Drupal\block\Tests\BlockTests"
  ERROR: Test class not found: Drupal\block\Tests\BlockTests
  ERROR: Did you mean?
  ERROR: Drupal\block\Tests\BlockCacheTest
  ERROR: Drupal\block\Tests\BlockHtmlTest
  ERROR: Drupal\block\Tests\BlockLanguageTest
  ERROR: Drupal\block\Tests\BlockTest
  ERROR: Drupal\block\Tests\BlockTitleXSSTest
  ERROR: Drupal\block\Tests\BlockUiTest
  ERROR: Drupal\book\Tests\BookTest
  ERROR: Drupal\block\Tests\BlockBaseTest
  ERROR: Drupal\block\Tests\BlockFormTest

Wow, impressive!! I'm wondering if it would be better to show the suggestions in one line for not filling out with error messages.

This version prints the alternatives on one line. I made the match stricter so we don't get as many.

$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x --color --class "Drupal\block\Tests\BlockTests"
  ERROR: Test class not found: Drupal\block\Tests\BlockTests
  DID YOU MEAN? Drupal\block\Tests\BlockCacheTest, Drupal\block\Tests\BlockHtmlTest, Drupal\block\Tests\BlockTest, Drupal\block\Tests\BlockUiTest, Drupal\book\Tests\BookTest, Drupal\block\Tests\BlockBaseTest, Drupal\block\Tests\BlockFormTest

$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x --color view
  ERROR: Test group not found: view
  DID YOU MEAN? views

+++ b/core/scripts/run-tests.sh
@@ -1059,3 +1087,21 @@ function simpletest_script_color_code($status) {
+ * @param $string
+ * @param $array
+ * @return array

This needs work. param name's are not helpful and description is also missing.

• Fixed up the function parameter and return comments.
• Parameterized the string matching degree.
• Fixed DID YOU MEAN? foo, bar to read DID YOU MEAN: foo, bar?

(8.0.x *=)$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x --color --class "Drupal\block\Tests\BlockTests"
  ERROR: Test class not found: Drupal\block\Tests\BlockTests
  DID YOU MEAN: Drupal\block\Tests\BlockCacheTest, Drupal\block\Tests\BlockHtmlTest, Drupal\block\Tests\BlockTest, Drupal\block\Tests\BlockUiTest, Drupal\book\Tests\BookTest, Drupal\block\Tests\BlockFormTest?

(8.0.x *=)$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x --color view
  ERROR: Test group not found: view
  DID YOU MEAN: views?

1. +++ b/core/scripts/run-tests.sh
   @@ -767,8 +767,25 @@ function simpletest_script_get_test_list() {
        if ($args['class']) {
   -      foreach ($args['test_names'] as $class_name) {
   -        $test_list[] = $class_name;
   +      $groups = simpletest_test_get_all();
   

   Hm. This triggers a full (slow) test discovery even though you specified a concrete list of test classes to execute.

   Wondering whether we can loop over the given list first, but check whether class_exists()?

   That will only trigger a direct lookup + autoload of the specified classes. Only if one of them does not exist, we can enter an error code path, which performs the additional discovery in order to output suggestions. (?)

2. +++ b/core/scripts/run-tests.sh
   @@ -767,8 +767,25 @@ function simpletest_script_get_test_list() {
   +            simpletest_script_print("  DID YOU MEAN: ", SIMPLETEST_SCRIPT_COLOR_FAIL);
   
   @@ -803,7 +820,18 @@ function simpletest_script_get_test_list() {
   +            simpletest_script_print("  DID YOU MEAN: ", SIMPLETEST_SCRIPT_COLOR_FAIL);
   

   Not sure why the prefix/label is in all-uppercase?

   Also, listing one alternative per line as in earlier iterations was more readable. I'd recommend to decrease the possible amount of alternatives instead:

   $ run-tests.sh --class Drupal\block\Tests\BlockTests
     ERROR: Test class not found: Drupal\block\Tests\BlockTest
     Did you mean?
     - Drupal\block\Tests\BlockTest
     - Drupal\block\Tests\BlockUiTest
     - Drupal\book\Tests\BookTest
   

3. +++ b/core/scripts/run-tests.sh
   @@ -803,7 +820,18 @@ function simpletest_script_get_test_list() {
   +          exit;
   

   Please note #2189345: run-tests.sh should exit with a failure code if any tests failed

@sun Thank you for reviewing. For #1, you are absolutely right, class_exists is faster, so I moved the full discovery to the error path. For #2, done. For #3, do you want to postpone this until the exit() constants are in?

$ sudo -u www php core/scripts/run-tests.sh  --url http://localhost/drupal8x --color --class "Drupal\block\Tests\BlockTes"
  ERROR: Test class not found: Drupal\block\Tests\BlockTes
  Did you mean?
  - Drupal\block\Tests\BlockHtmlTest
  - Drupal\block\Tests\BlockTest
  - Drupal\block\Tests\BlockUiTest
  - Drupal\block\Tests\BlockFormTest

Thanks! Looks better now.

We don't necessarily need to wait for the other issue.

Can we find a better name for the simpletest_alternatives() function? (1) All functions in the script are prefixed with simpletest_script_. (2) Any particular reason for not moving the print/output code directly into it + rename it to simpletest_script_print_alternatives()?

btw, fixing issue title - a bug report should summarize the bug, not the feature. :-)

Renamed to simpletest_script_print_alternatives().

Changed the matching on test group names to be less strict than for classes in terms of the input length because the group names are shorter than class names.

Great work. Now it looks really clean.

We probably want to backport this to D7 for (1) parity/consistency, (2) UX, and (3) because D7 will be around for some more years.

Recommended commit message "Issue #2310415 by cilefen, ednawig TravisCarden: Fixed run-tests.sh does not handle the error when invalid test groups/classes are specified."

@ednawig helped.

Very nicely done!

Committed and pushed to 8.x. Moving back to 7.x for back port.

Here is the backport to D7.

(7.x *%=)$ sudo -u www php scripts/run-tests.sh  --url http://localhost/drupal7 --class "BlockMashTestCase" 
  ERROR: Test class not found: BlockMashTestCase
  Did you mean?
  - BlockCacheTestCase
  - BlockHashTestCase

(7.x *%=)$ sudo -u www php scripts/run-tests.sh  --url http://localhost/drupal7 Users
  ERROR: Test group not found: Users
  Did you mean?
  - User

Patch applies fine and works as expected.

Committed to 7.x - thanks!

I went ahead and added this to CHANGELOG.txt, since @cilefen pointed out that it's actually a new feature (in addition to a bugfix).