Drupalci.yml filtering for testgroups: 'scheduler,scheduler_api' does not run all the required tests

The tests which are no longer being run are

Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicProductTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicMediaTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerAdminSettingsTest

Previously, with no testgroups filtering, the tests were executed in reverse-alphabetical class name, and these three were the last three to be executed. Now, with the testgroups filter in place, the tests are being executed in forward-alphabetical class name order. I have no idea why the change or order, probably some side-effect of how the test list is selected or reduced.

However, is it a co-incidence that these three tests would now be the first three that should be run? Is it linked to the fact that three test classes are now correctly being filtered out due to not including @group scheduler_rules_integration and somehow another three are being dropped by accident?

First, to elimintate the change of identifier in the .yml syntax

OK, the three tests at the top of the list are still not being run. Now see what happens if I move one of API tests back into the main Scheduler group.

So! That's interesting. I moved EventsTest (which was running fine) out of @group scheduler_api into @group scheduler. The test still runs, but now we get one of the lost/discarded tests schedulerBasicProductTest back in the list of tests being run. This was the third test (in forwards-alphabetical order) that was being discarded.

The recovered test runs first, then we get the moved test, then the rest of the list in alphatbetical order. The API group has now been shifted to the end of the list. Something funny is going on.

See what happens if the order of the two test groups in the list are changed over.

Swapping the order of the two values in testgroups has had an impact. We have got back the two tests at the top of the list which had been lost, but have now not run the two tests in scheduler_api group. It is as if some kind of array counter bug means that the list of tests is not being merged correctly.

Here's the test comparison list that goes with #6

Now move the EventsTest back into scheduler_api group but keep the order of the two groups in the drupalci.yml parameter the same as in #8 above, i.e. testgroups: 'scheduler_api,scheduler'

Probably as expected, the test that is moved back into scheduler_api is not run.

Looking at the evidence of all the tests above, one theory could be that the number of tests getting run in the group being shown second is reduced by the number being run in the first group.

This run has only five tests in the scheduler group as I have moved all the others from that group into a new do_not_run group. All five tests were run and none of the three in scheduler_api were run.

Scheduler.Scheduler

Scheduler.Drupal\Tests\scheduler\Functional\SchedulerAdminSettingsTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicMediaTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicProductTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicTest
Scheduler.Drupal\Tests\scheduler\FunctionalJavascript\SchedulerJavascriptDefaultTimeTest

Now going to move those last two tests into do_not_run

There are 3 tests in scheduler group and 3 tests in scheduler_api group but only the three in scheduler get run:

Scheduler.Scheduler

Scheduler.Drupal\Tests\scheduler\Functional\SchedulerAdminSettingsTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicMediaTest
Scheduler.Drupal\Tests\scheduler\Functional\SchedulerBasicProductTest

The next change will be to move one of the running tests from scheduler into do_not_run group

So now that BasicMedia is in the do_not_run group, we have 2 tests defined in scheduler and 3 tests defined in scheduler_api. Dropping one of the running tests has allowed one of the api tests to show up. This is the "third" api test that should be run.

In the commit above I defined BasicProduct to the do_not_run group. This has allowed another of the already defined scheduler_api tests to be included and run. There is now just one test defined in the scheduler group and this is causing just one test from the scheduler_api group to be ignored.

This is a core bug in core/scripts/run-tests.sh and I have raised #3251817: Make run-tests.sh select all the required tests when testgroups: has more than one value to request the fix.

That's good. When my patch for run-tests.sh is applied we get the expected set of tests being run. In this cut-down example we get the four tests compared to the three in #15 above.

Commited the change to drupalci.yml which applies the patch to core/scripts/run-tests.sh. This can be removed when #3251817: Make run-tests.sh select all the required tests when testgroups: has more than one value is committed.

Also moved the javascript test from @group scheduler into @group scheduler_js

Don't want to re-open this issue, and cannot run patch tests when it is Closed.
See #3049585-7: Check drupal.org testing infrastructure for the test.