Allow --directory and @group to work together in run-tests.sh

I have pushed an initial commit to the MR. I'm utilising the existing --debug-discovery argument to give further debug output to show what it happening. Some of this debug could stay, other bits will be deleted.

I will use the diff url from this MR to download and apply the patch, to deminstrate it in contrib testing.

Interesting with the PHPUnit Build failed job, the command is
sudo -u www-data -E -H php ./core/scripts/run-tests.sh --phpunit-configuration $PHPUNIT_CONFIGURATION_FILE_PATH --debug-discovery --color --keep-results --types "$TESTSUITE" --concurrency "$CONCURRENCY" --repeat "1" --sqlite "./sites/default/files/tests.sqlite" --dburl $SIMPLETEST_DB --url $SIMPLETEST_BASE_URL --verbose --non-html --all --ci-parallel-node-index $CI_PARALLEL_NODE_INDEX --ci-parallel-node-total $CI_PARALLEL_NODE_TOTAL

but the test failed due to an unexpected value for test_names

    [test_names] => Array
        (
            [0] => 1
        )

This would normally be either empty, or a test group or a class or module, but not "1". Then we also see

[ci-parallel-node-index] => --ci-parallel-node-total
[ci-parallel-node-total] => 1

so this implies to me that $CI_PARALLEL_NODE_INDEX is unintentionally empty, thus shuffling the value along so it reads the string "--ci-parallel-node-total" as the value. This means we get left with a "1" which is taken to be a test group (positional parameter) and that causes the $group array to be filtered for the test called "1", so we end up with no tests found.

Thanks @mondrake for the link. I've followed that issue.

To allow testing in Gitlab pipelines here is a static patch file for core 11.2 as there have been some changes in 11.x (11.3) which clash when trying to apply using https://git.drupalcode.org/project/drupal/-/merge_requests/13363.diff

I have pushed a one-line fix for the empty CI_PARALLEL_NODE_INDEX, adding

# If CI_PARALLEL_NODE_INDEX is empty set it to 1 so that the argument parsing works correctly.
CI_PARALLEL_NODE_INDEX=${CI_PARALLEL_NODE_INDEX:-1}

and that has fixed all of the failing core pipeline tests :-)

This is ready for review and feedback. I have also used this patch in contrib jobs that were failing due to 3rd-party Test Discovery errors, and now they pass. I'll add links to them here.

Thanks. I've replied in the threads.

Here is a test on the Webform module, submitted via UI just trying to run with _PHPUNIT_TESTGROUPS = webform_access
https://git.drupalcode.org/issue/webform-3547627/-/pipelines/616970 - it fails as expected becuase of errors in the Feed module test files

Now the same UI pipeline, but using Gitlab Tempates MR410, which applies the patches from here and add --directory in the command
https://git.drupalcode.org/issue/webform-3547627/-/pipelines/616992 - phpunit runs just the small subset of tests and passes.

OK I'll open a new MR so I do not pollute your work, then

I have a static patch of the work done so far, so I can use that in other jobs, and for reference, so I'm fine with you using this MR, then we can see the further changes you do from this point. Core has enough redundant MRs already :-) But if you want to start again that's OK too.

Thank you. That's a better way to do it. Much cleaner to read. I will test in a real contrib pipeline, but as you said before this could also have a build test. I don't know where the tests for run-tests.sh are, but if you point me there I could look at expanding to add coverage for this. Or if you can do it easily and quickly then carry on and do so.

As per #16-18 this issue is postponed on #3536964: run-tests.sh - segregate command line parsing and use Symfony Console classes

But the error in run-tests.sh test discovery that is being solved here is now causing actual test failures in contrib, now that some 3rd-party modules have dropped using @group mymodule in favor of #[Group('mymodule')].

Do we have a plan for which run-tests.sh issues are being worked on, and in what order?

Adding meta parent

Rebased this MR, and it appears that the file .gitlab-ci/pipeline.yml is no longer in 'main' branch. Or more likely it's been moved or remaned. Hence the one-line fix I had to add before has got lost, and we get a recurence of the test failures as described in #4 above and fixed in #7
https://git.drupalcode.org/issue/drupal-3549601/-/jobs/8752436#L75

Yes, just found it. So I should probably re-do that single line change in the .gitlan-ci.yml file, which looks like is being used instead of the deleted one

@@ -163,13 +229,12 @@ default:
 # If any job in the stage fails, the pipeline will exit early.
 ################
 
-.default-stage: &default-stage
-  stage: 🗜️ Test
+.recursive-trigger: &recursive-trigger
+  stage: 🗜️ Additional tests
   trigger:
     # Rely on the status of the child pipeline.
     strategy: depend
-    include:
-      - local: .gitlab-ci/pipeline.yml
+    include: .gitlab-ci.yml
 
 .run-on-commit: &run-on-commit
   rules:

I replicated the change from before, but it still failed. Then I spoted that the variable names has also changed:

before
--ci-parallel-node-index $CI_PARALLEL_NODE_INDEX --ci-parallel-node-total $CI_PARALLEL_NODE_TOTAL

now
--ci-parallel-node-index $CI_NODE_INDEX --ci-parallel-node-total $CI_NODE_TOTAL

I have pushed a second change, but if the variable rename was accidental, let me know and we can fix it the other way.

Thanks for the rebases. I will re-visit the contrib testing I did in #9 to check that it works like before.