Problem/Motivation
Follow-up from #3449743: Try to optimize test ordering when run-tests.sh is used with a mixture of test types which wasn't enough.
When gitlab runs contrib tests, we want to run the slowest tests first, currently, slowest tests are run last.
Steps to reproduce
Before:
php ./core/scripts/run-tests.sh --sqlite 'sites/default/files/.sqlite' --directory modules/contrib/paragraphs
Drupal test run
---------------
Tests to be run:
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsClientsideButtonsClaroTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsStableEditPerspectivesUiTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsAddWidgetTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsClientsideButtonsTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsEntityReferenceWarningTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsWidgetElementsTest
- Drupal\Tests\paragraphs\Kernel\migrate\FieldCollectionTypeSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsItemRevisionSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\FieldCollectionItemSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsFieldMigrationTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsItemSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\FieldCollectionItemRevisionSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsTypeSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphContentMigrationTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsTypeMigrationTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsCompositeRelationshipTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsBehaviorPluginsTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsTypeHasEnabledBehaviorPluginTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsCollapsedSummaryTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsIsChangedTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsLangcodeChangeTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsEntityMethodsTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsAccessTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsReplicateTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsSummaryFormatterTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsTypesTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsTranslationTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsPreviewTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsFieldGroupTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsWidgetButtonsTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsConfigTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsInlineEntityFormTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsUiTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsAdministrationTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsAddModesTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsContactTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsAccessTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsEditModesTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsLegacyContentModerationTranslationsTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsEntityTranslationWithNonTranslatableParagraphs
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsSummaryFormatterTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsTypesTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsTranslationTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsPreviewTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsTranslationsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsFieldGroupTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsWidgetButtonsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsConfigTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsDragAndDropModeTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsInlineEntityFormTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsUiTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAdministrationTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAddModesTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsContentModerationTranslationsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAlterByTypeTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsContactTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAccessTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsReplicateEnableTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsBehaviorsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsEditModesTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsDuplicateFeatureTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsEntityTranslationWithNonTranslatableParagraphs
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsHeaderActionsTest
- Drupal\Tests\paragraphs\Functional\Migrate\MigrateUiParagraphsTest
- Drupal\Tests\paragraphs\Functional\ParagraphsWidgetButtonsTest
- Drupal\Tests\paragraphs\Functional\ParagraphsUiTest
- Drupal\Tests\paragraphs\Functional\ParagraphsBehaviorsTest
- Drupal\Tests\paragraphs\Functional\ParagraphsUninstallTest
- Drupal\Tests\paragraphs\Unit\migrate\FieldCollectionsFieldInstanceSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\ParagraphsFieldInstanceSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\ParagraphsProcessOnValueTest
- Drupal\Tests\paragraphs\Unit\migrate\ParagraphsFieldSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\MigrationPluginsAltererTest
- Drupal\Tests\paragraphs\Unit\migrate\FieldCollectionFieldSettingsTest
- Drupal\Tests\paragraphs_demo\Functional\ParagraphsDemoTest
- Drupal\Tests\paragraphs_type_permissions\Functional\ParagraphsTypePermissionsTest
- Drupal\Tests\paragraphs_library\FunctionalJavascript\ParagraphsContentModerationTest
- Drupal\Tests\paragraphs_library\FunctionalJavascript\ParagraphsLibraryItemEntityBrowserTest
- Drupal\Tests\paragraphs_library\Functional\ParagraphsLibraryTest
- Drupal\Tests\paragraphs_library\Functional\ParagraphsLibraryItemTranslationTest
- Drupal\Tests\paragraphs_library\Functional\ParagraphsLibraryItemTest
- Drupal\Tests\paragraphs_library\Functional\MultilingualBehaviorTest
After:
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsAddWidgetTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsClientsideButtonsClaroTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsClientsideButtonsTest
- Drupal\Tests\paragraphs_library\FunctionalJavascript\ParagraphsContentModerationTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsEntityReferenceWarningTest
- Drupal\Tests\paragraphs_library\FunctionalJavascript\ParagraphsLibraryItemEntityBrowserTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsStableEditPerspectivesUiTest
- Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsWidgetElementsTest
- Drupal\Tests\paragraphs\Functional\Migrate\MigrateUiParagraphsTest
- Drupal\Tests\paragraphs_library\Functional\MultilingualBehaviorTest
- Drupal\Tests\paragraphs\Functional\ParagraphsBehaviorsTest
- Drupal\Tests\paragraphs_demo\Functional\ParagraphsDemoTest
- Drupal\Tests\paragraphs_library\Functional\ParagraphsLibraryItemTest
- Drupal\Tests\paragraphs_library\Functional\ParagraphsLibraryItemTranslationTest
- Drupal\Tests\paragraphs_library\Functional\ParagraphsLibraryTest
- Drupal\Tests\paragraphs_type_permissions\Functional\ParagraphsTypePermissionsTest
- Drupal\Tests\paragraphs\Functional\ParagraphsUiTest
- Drupal\Tests\paragraphs\Functional\ParagraphsUninstallTest
- Drupal\Tests\paragraphs\Functional\ParagraphsWidgetButtonsTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsAccessTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsAddModesTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsAdministrationTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsConfigTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsContactTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsEditModesTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsEntityTranslationWithNonTranslatableParagraphs
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsFieldGroupTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsInlineEntityFormTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsLegacyContentModerationTranslationsTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsPreviewTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsSummaryFormatterTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsTranslationTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsTypesTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsUiTest
- Drupal\Tests\paragraphs\Functional\WidgetLegacy\ParagraphsWidgetButtonsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAccessTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAddModesTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAdministrationTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsAlterByTypeTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsBehaviorsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsConfigTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsContactTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsContentModerationTranslationsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsDragAndDropModeTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsDuplicateFeatureTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsEditModesTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsEntityTranslationWithNonTranslatableParagraphs
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsFieldGroupTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsHeaderActionsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsInlineEntityFormTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsPreviewTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsReplicateEnableTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsSummaryFormatterTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsTranslationTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsTranslationsTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsTypesTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsUiTest
- Drupal\Tests\paragraphs\Functional\WidgetStable\ParagraphsWidgetButtonsTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsAccessTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsBehaviorPluginsTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsCollapsedSummaryTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsCompositeRelationshipTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsEntityMethodsTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsIsChangedTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsLangcodeChangeTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsReplicateTest
- Drupal\Tests\paragraphs\Kernel\ParagraphsTypeHasEnabledBehaviorPluginTest
- Drupal\Tests\paragraphs\Kernel\migrate\FieldCollectionItemRevisionSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\FieldCollectionItemSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\FieldCollectionTypeSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphContentMigrationTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsFieldMigrationTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsItemRevisionSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsItemSourceTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsTypeMigrationTest
- Drupal\Tests\paragraphs\Kernel\migrate\ParagraphsTypeSourceTest
- Drupal\Tests\paragraphs\Unit\migrate\FieldCollectionFieldSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\FieldCollectionsFieldInstanceSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\MigrationPluginsAltererTest
- Drupal\Tests\paragraphs\Unit\migrate\ParagraphsFieldInstanceSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\ParagraphsFieldSettingsTest
- Drupal\Tests\paragraphs\Unit\migrate\ParagraphsProcessOnValueTest
Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes
Release notes snippet
Issue fork drupal-3450616
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
catchWhat this does:
Consolidates test discovery when --directory is passed as an argument, so that tests are discovered via the same mechanism. When directory is passed, we get all the available test classes then filter ones out that aren't in the passed directory. This was the least intrusive way I could think of to unify things.
When --types isn't passed, we order tests alphabetically, stripping off the module namespace. What this means in practice is that tests are ordered Build / Functional JavaScript / Functional / Kernel / Unit which coincidentally roughly corresponds to how long each test type takes. After this all @group #slow tests are prepended to the beginning of the list so they run first.
This means that @group #slow will be respected for contrib, whereas it currently isn't, and whether it's set or not, slower test types will run first, maximising the benefits of concurrent test runs.
Comment #4
catchComment #5
smustgrave commentedRecommended way to test?
Comment #6
catchThe way I tested this:
1. Have 10.3.x locally.
2. composer require drupal/paragraphs drupal/entity_browser
3. Start running paragraph tests with
php ./core/scripts/run-tests.sh --sqlite 'sites/default/files/.sqlite' --directory modules/contrib/paragraphs. All we need is the run-tests.sh output about which tests are going to be run, don't actually need to run the tests, - I just crtl-C'd as soon as I had that.4. Apply the MR diff to 10.3, then run the same run-tests command, and make a note of the order (then ctrl-C again). It should be unit tests at the end, and sub-module tests should be in order of test type, not in order of module.
5. Extra bonus testing, edit a paragraphs test that appears towards the end (like a unit test) and add @group #slow to it. This should bring it to the start of the run-tests.sh output.
As with the previous issue (even though that didn't have the desired effect), it's going to be a lot easier to see any performance changes by committing it to core and then watching a contrib pipeline with concurrency, especially since we probably also need to tweak the run-tests concurrency per the linked issue on top of this. But the steps above should confirm that the issue actually does what it's supposed to.
Comment #7
smustgrave commentedApplied MR
Added group #slow to a unit test
Does as advertised
Comment #8
catch#7 shows I introduced a bug here with the most recent commit. I was trying to avoid ordering alphabetically when $args['types'] was passed in, but $types_processed is set to TRUE when module is passed too, so that doesn't work. Had another look and also realised it's valid to pass multiple types to $args['types'] so I think we should just order alphabetically all the time.
Last bit of the output from #7:
Last bit of the output after reverting that last change;
Leaving RTBC since it's just removing an if statement that regressed from the previous state of the patch.
Comment #9
quietone commentedI read the issue summary, comments and the MR. This look very straightforward. I then did the manual testing as described in #6 and used the same Unit test as smustgrave in #7. I reproduced the same results as #7 not #8. I don't think the revert stated in #8 was pushed.
Comment #10
catchOof. Pushed the revert this time.
Comment #11
quietone commented@catch, thanks.
I now get the expected order of tests. However, I also get a deprecation
This is on 10.3.x and PHP 8.3.
Comment #12
catchThanks need to check my ddev is reporting deprecation errors properly. Added a cast before returning.
Comment #13
quietone commentedNo deprecations but the sorting still wasn't as expected. I re-read the PHP man page for usort and spotted a problem with the return values from the callback. I left a suggestion in the MR about that.
Comment #14
catchGood point, applied that suggestion.
Comment #15
quietone commentedI retested on 10.3.x and the end of the list is now sorted by test type. The end of the list is now as expected.
So, this is now good to go.
Comment #16
alexpottCommitted and pushed af8831671c to 11.x and bef045d92f to 11.0.x and 9699eff69f to 10.4.x and a47b41772e to 10.3.x. Thanks!
Backported to 10.3.x as a test only change
Comment #17
alexpott