Speed up the tests on DrupalCI by not running slow tests last

Maybe we could reuse the phpunit @large annotation and set the timeout to something long, like 20 minutes? Not sure if that's using that annotation appropriately, or not.

+++ b/core/modules/settings_tray/tests/src/FunctionalJavascript/SettingsTrayBlockFormTest.php
@@ -9,6 +9,7 @@
+ * @group #slow
  * @group settings_tray

So when you say 'orders by alphabet' that's a bit of a problem, because TestDiscovery doesn't know how to deal with multiple groups. So if you make the first group '#slow' then 'settings_tray' is ignored by run-tests.sh. Anyone using run-tests.sh to run that group then loses those tests.

But hey, guess what. We have this: #2296615: Accurately support multiple @groups per test class

We could also introduce our own group annotation scheme like @early or something, which would then be a sort for run-tests.sh. Then establish some metrics with, for instance, speedtrap: https://github.com/johnkary/phpunit-speedtrap

I think Using @large is a bit problematic

If the PHP_Invoker package is installed and strict mode is enabled, a large test will fail if it takes longer than 60 seconds to execute. This timeout is configurable via the timeoutForLargeTests attribute in the XML configuration file.

see https://phpunit.de/manual/6.5/en/appendixes.annotations.html#appendixes....

60 seconds :) yes we can change it but we're amusingly different from that.

I think using groups is a good idea but it does sound like we need to fix #2296615: Accurately support multiple @groups per test class first.

Here's a first pass at @early.

Using a group is fine, too. I just wanted to avoid magical naming conventions that work because other things don't work.

It's primitive, but it works.

I appreciate the primitive simplicity. ;)

Do we document test groups anywhere?

- If so, can we update about this? What's the criteria for being #slow? How will people know when to use this group or not? Is the expectation that core committers will Just Know(tm) and get it right from here on out? Or will @Lendude keep a watchful eye on all tests added to core? ;) Neither seems particularly sustainable.

- If not, can we add some? ;) Not sure where. Perhaps in core/tests/README.md and/or core/core.api.php? Both mention that test groups exist, but neither really gives any guidance about how to define them.

Otherwise, +1 for RTBC on this. Getting the slow operations started first is a classic scheduling algorithm in concurrent systems.

Cheers,
-Derek

Doc pages: https://www.drupal.org/docs/8/phpunit

We don't have much about groups, other than mentioning you need at least one in this document: https://www.drupal.org/docs/8/phpunit/phpunit-file-structure-namespace-a...

As far as metrics for which tests get the treatment: I'd say we should include a timing tool that fails your test run if an unmarked test takes too long. I'd say something like https://github.com/johnkary/phpunit-speedtrap but that requires PHP 7.1.

It's not a very complicated listener to put together, but does require doing. And then bikeshedding. :-)

Note that one benefit here is that we can say:

./vendor/bin/phpunit -c core --exclude-group #slow

Yeah, I'm actually a maintainer of that doc guide. ;) I was thinking it'd be good to document this stuff directly in core, not just in a handbook page.

Meanwhile, https://www.drupal.org/docs/8/phpunit/phpunit-file-structure-namespace-a... does mention @group, but only to say:

Every test class MUST specify at least one (first) @group that matches the originating module short name (or Drupal core component name).

Not particularly helpful...

So yes, we could add something about #slow to that page. But I'm wondering if we can also put it into some of the included-with-core-in-Git docs, and if so, if there's a preference about where and what.

Speedtrap sounds nice, but that's obviously a separate issue. For now, I'd like some clarity (I guess from @Lendude) on the criteria used to generate this patch so we have something to start with.

Thanks,
-Derek

#3033037: DrupalCI should track and display job run times is about the time of the entire build run (1 hr 10 min etc).
#2943340: Process PHPUnit junit reports separately is the drupalci issue that would have us use the phpunit junit.xml, which already has the test timing data in it, vs the junit that run-tests.sh generates.

Here's your follow-up: #3040694: Use the phpunit result cache to order tests by slowest first in gitlab jobs

How about including all the update path tests in the #slow group? They're all really slow "by nature" :)

@amateescu re: #23: At #3040694: Use the phpunit result cache to order tests by slowest first in gitlab jobs we've got a patch that puts all tests that take longer than 75 seconds in the #slow group, and a mechanism to fail new tests that are added which take more than 90 seconds and aren't #slow. There's a lot from core/modules/system/tests/src/Functional/Update in that patch, but definitely not everything. ;) If we put too many things in #slow, it starts to undo the effectiveness of the scheduling algorithm.

Anyway, I'm really in favor of #3040694 as a quantifiable way to decide what belongs in #slow or not.

I just tried applying both patches. Much to my surprise, a few of the tests marked #slow in here didn't hit the 75 second mark over there. Of course that patch catches a lot more tests than this did. Looking at the interdiff, I also noticed one of the ones in here has a typo (@gruop #slow).

So, not entirely sure how to proceed now. Honestly, I think #3040694-11: Use the phpunit result cache to order tests by slowest first in gitlab jobs is the actually RTBC patch now. But in fairness, this was @Lendude's cool idea, and @Mile23 and I just built upon it. In terms of issue history, commit credits, etc, I think we should commit something here first, with @Lendude as the primary author, then we can re-roll #3040694 to remove whatever the conflicts are, and commit @Mile23's cool test listener and the rest of the tests that should also be #slow.

How does that sound to everyone?

NW for the typo, and removing the #slow tests from here that aren't 75 seconds slow.

#24 sounds great to me :)

Let's keep both of the Kernel tests from #14.

Committed dcd66e1 and pushed to 8.8.x. Thanks!

c/p to 8.7.x