Problem/Motivation

To be more effecient and fast in developing for Drupal we should find failures in the testsuites as quickly as possible. Currently if i work on jsonapi i need to wait ages to get to jsonapi tests. What if we prioritize the running of tests of changed modules/parts of core and fail on that.

"Fix the system you are working in first, then test integration in the rest of Drupal"

  1. Check what files have changes
  2. Identify the subsystem/core/vendor that have changes
  3. Run tests for those modules first (and stop on failure)
  4. Then run the rest of the tests

This would surface errors earlier, and make sure we get feedback as fast as possible. This also saves testing minutes since we start with tests that are most likely to break. The chance a change in migrate breaks menu's is rather small.

Source: https://drupal.slack.com/archives/CGKLP028K/p1694092091417259

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

bbrala created an issue. See original summary.

bbrala’s picture

bbrala
Location Netherlands
commented
Issue summary: View changes
fjgarlin’s picture

fjgarlin commented
Project: GitLab Templates » GitlabCI Testbed for Drupal core
Component: gitlab-ci » Code
bbrala’s picture

bbrala
Location Netherlands
commented
Issue summary: View changes
fjgarlin’s picture

fjgarlin commented

Since #3386076: GitLab CI integration for core is merged, shouldn't we move this issue to core? Or do you prefer to keep it in this project as "playground"?
I'm going through the issues and closing the ones I can in favour of core follow-ups.

bbrala’s picture

bbrala
Location Netherlands
commented

Agree, but not sure what meta to hang it under

fjgarlin’s picture

fjgarlin commented
Project: GitlabCI Testbed for Drupal core » Drupal core
Version: » 11.x-dev
Component: Code » phpunit
Parent issue: » #3387107: [meta] GitLab CI feature parity with DrupalCI

Added it under #3387107: [meta] GitLab CI feature parity with DrupalCI.

larowlan’s picture

larowlan
Location 🇦🇺🏝.au GMT+10
commented

In theory we could align labels in MRs with @group annotations on the tests and run those first

catch’s picture

catch
he/him
Primary language English
commented

I think --directory might be broken in run-tests.sh, or at least it seemed to be when I tried to use it, but we could probably figure out core + module directories from the git diff and run those when they're changed using --directory. However that wouldn't help for composer updates so would need to account for that kind of change too.

bbrala’s picture

bbrala
Location Netherlands
commented

Directory could work, there is also module argument it seems.

But a filter on directory is easy enough for the testlist. We'd also need an negative filter though.

--skip "list of dirs/modules" and
--only "list of dirs/modules"

catch’s picture

catch
he/him
Primary language English
commented

Only concern I have about this is that after #3386474: [omnibus] Speed up gitlab ci runs we'll have 12-15 minute feedback on all-green tests, if we add an extra stage, green MR runs will take longer to report back and run more jobs, even if failures come back quicker.

catch’s picture

catch
he/him
Primary language English
commented

We might want to look at https://gitlab.com/gitlab-org/gitlab/-/issues/23605 too (not implemented but lots of demand). With #3387706: Don't make other tests depend on PHPUnit we'd run unit tests parallel to other tests (so all tests run earlier instead of waiting), but if we were able to interrupt and cancel jobs when any of the test fails, that feedback would go onto the overall pipeline status/issue a lot quicker -

i.e. a unit test failure cancels the other tests and reports back. Then you fix the unit tests and the others will go until one type fails and it reports back, and etc.

bbrala’s picture

bbrala
Location Netherlands
commented

It's a tradeoff. Testing changed stuff first would help getting to green faster from a contributer standpoint. Just if all is fine it might take longer.

Sometimes the suite ran is pretty small if only changes are run but sometimes is might not be. Perhaps a step early, like phpunit, IF the changed testlist is smaller than x tests (let's say 100) is a way to go.

catch’s picture

catch
he/him
Primary language English
commented

That gives me an idea.

We could put this in the main/parent pipeline with the lint steps, that way the smaller subset of tests would run parallel to the linting jobs. This would address three things - you'd be able to see some test results even if linting fails, you'd see them at the earliest possible point, and because it would run in parallel with linting it's not adding an extra blocking stage to the whole run. Also the pods used for this step would then get re-used by the child/full test runs.

bbrala’s picture

bbrala
Location Netherlands
commented

Yapp that's what I was thinking. Only possible hard part;

Are changes in core/lib one set or do we segregate there also ;) that might be a challenge.

catch’s picture

catch
he/him
Primary language English
commented

Maybe for core/lib we could run only unit + kernel from core/tests and not functional/functional javascript? That'd be quick enough, then if you're unlucky enough that you only get a failure in a functional test, you just have to wait for the full run.

Otherwise it's either the MR tags idea or completely re-organising core components to ship their own test coverage etc., which... maybe in five years or so?

bbrala’s picture

bbrala
Location Netherlands
commented

So basically:

Create a list based on module, matching /modules/module/ in the path, or core with core/lib in the path. Unique that list and either filter the testlist or inverse filter the testlist.

That would be an easy poc.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.