Problem/Motivation

No reason to fetch a whole drupal history, it will help
- to transfer less data from Gitlab
- speeds-up each CI job (fetching 50 commits takes ~10seconds comparing to 1minute for a whole repo)

Steps to reproduce

See https://git.drupalcode.org/project/drupal/-/jobs/82685

Getting source from Git repository 01:05

Proposed resolution

- add GIT_DEPTH=5to fetch only limited amount of of commits from MR branch
- fix 📔 Spell-checking job - it should fetch only branch to compare with

The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines.

Remaining tasks

- review/comit

User interface changes

API changes

Data model changes

Release notes snippet

Issue fork drupal-3387400

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

andypost created an issue. See original summary.

andypost’s picture

Status: Active » Needs review
andypost’s picture

catch’s picture

We might be able to reduce the depth further once we've done #3387058: [PP-1] Add caching to cspell job in GitLabCI assuming that can take advantage of cloud caching.

This looks good to me and think we should try it.

andypost’s picture

andypost’s picture

Cloud caching has downsides
- more data transfered from/to runner
- zipping cache before sending takes time and brings high load to caching server

Meantime I see #3386566: Add support for 'test only' changes to gitlab CI which also gonna diff changed files so this functionality can be extracted to before script for such jobs

Moreover the job should fail when it unable to find a lit of changed files

andypost’s picture

Assigned: andypost » Unassigned
Issue summary: View changes

Added to IS about ${CI_COMMIT_BRANCH} which probably should not be used in scripts

andypost’s picture

andypost’s picture

Issue summary: View changes

Changed to fetch less to compare and fixed IS as TARGET_BRANCH=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}${CI_COMMIT_BRANCH} which works both for MRs and commits

catch’s picture

Status: Needs review » Reviewed & tested by the community

I think we should see how this affects job performance, RTBCing and will fish for a +1 in slack.

This Unit test run finished in 1m45s https://git.drupalcode.org/issue/drupal-3387400/-/jobs/89382 vs 2m48s for HEAD: https://git.drupalcode.org/project/drupal/-/jobs/90216

So that is consisent with the ~50s that andypost reported above.

longwave’s picture

Status: Reviewed & tested by the community » Needs review
catch’s picture

Status: Needs review » Needs work

Discussed with @longwave in slack - let's move the cspell change to its own issue since it's harder, and we might just try full runs at least on branch tests anyway. We'll get the benefit from the rest of the changes here without that, especially in terms of load on the git hosting.

andypost’s picture

Status: Needs work » Needs review

The job fixes can't be moved to separate issue because it will fail

I did test it in https://git.drupalcode.org/issue/drupal-3252386/-/jobs/88908

andypost’s picture

Sorry the valid link is https://git.drupalcode.org/issue/drupal-3252386/-/jobs/88870

Job does not fail but doing nothing, it smells a new issue

andypost’s picture

Status: Needs review » Needs work

it needs work for CI_MERGE_REQUEST_PROJECT_URL which is missing for commit/tag runs

longwave’s picture

Status: Needs work » Needs review

Added a suggestion that might work.

andypost’s picture

Looks failure was random, on restart it passed

    Testing Drupal\Tests\Component\Utility\RandomTest
    ......E.....                                                      12 / 12
    (100%)
    
    Time: 00:00.032, Memory: 4.00 MB
    
    There was 1 error:
    
    1)
    Drupal\Tests\Component\Utility\RandomTest::testRandomMachineNamesUniqueness
    RuntimeException: Unable to generate a unique random machine name
    
    /builds/issue/drupal-3387400/core/lib/Drupal/Component/Utility/Random.php:169
    /builds/issue/drupal-3387400/core/tests/Drupal/Tests/Component/Utility/RandomTest.php:127
    /builds/issue/drupal-3387400/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
    /builds/issue/drupal-3387400/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
    /builds/issue/drupal-3387400/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:651
    /builds/issue/drupal-3387400/vendor/phpunit/phpunit/src/TextUI/Command.php:144
    /builds/issue/drupal-3387400/vendor/phpunit/phpunit/src/TextUI/Command.php:97
    
    ERRORS!
    Tests: 12, Assertions: 97, Errors: 1.
[31mFail      run-tests. Unknown              0 Unknown                            
[0m    FATAL Drupal\Tests\Component\Utility\RandomTest: test runner returned a
    non-zero error code (2).
andypost’s picture

longwave’s picture

Status: Needs review » Reviewed & tested by the community

Spellcheck job passed, code looks OK, the only way to branch test is to merge it and see.

andypost’s picture

+1 RTBC and using the variable in job, suggested the same to backport to D7 #3387052: [D7] GitLab CI integration for core

Some tests still running but what we fixing here is https://git.drupalcode.org/project/drupal/-/jobs/91306

$ export TARGET_BRANCH=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}${CI_COMMIT_BRANCH}
$ git fetch -vn --depth=$GIT_DEPTH "${CI_MERGE_REQUEST_PROJECT_URL:-origin}" "+refs/heads/$TARGET_BRANCH:refs/heads/$TARGET_BRANCH"
warning: redirecting to https://git.drupalcode.org/-/push_from_secondary/9/project/drupal.git/
POST git-upload-pack (334 bytes)
POST git-upload-pack (524 bytes)
From https://git.drupalcode.org/project/drupal
 * [new branch]        11.x       -> 11.x
$ export MODIFIED=`git diff --name-only refs/heads/$TARGET_BRANCH|while read r;do echo "$CI_PROJECT_DIR/$r";done|tr "\n" " "`
$ echo $MODIFIED | tr ' ' '\n' | yarn --cwd=./core run -s spellcheck:core --no-must-find-files --file-list stdin
1/1 /builds/project/drupal/.gitlab-ci.yml
 437.27ms
2/2 /builds/project/drupal/.gitlab-ci/pipeline.yml 88.84ms
CSpell: Files checked: 2, Issues found: 0 in 0 files

  • catch committed 53dbbdc6 on 10.1.x
    Issue #3387400 by andypost, longwave, catch: GitlabCI should fetch less...

  • catch committed e0eb2312 on 11.x
    Issue #3387400 by andypost, longwave, catch: GitlabCI should fetch less...
catch’s picture

Version: 11.x-dev » 10.1.x-dev
Status: Reviewed & tested by the community » Fixed

Committed/pushed to 11.x and cherry-picked to 10.1.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.