[META] Define a default .gitlab-ci.yml template that projects can inherit

I am part of the testing phase with upgrade_status. Started today and this CI/CD option showed up, woot:

Pipelines offers me lots of options, but none seem to be a match.

So I went to start with a template, but this is VERY generic template that only lets me know what stages I can use and suggests a simple template where the steps echo things out.

I guess I could take that template just for fun to see how this work (I did not use Gitlab CI/CD before). But I did not find the next step. I was expecting this issue to have SOME initial CI suggestions, but it seems to be a skeleton, so not sure how to actually get started.

@moshe suggested we take a look at https://git.drupalcode.org/project/keycdn/-/blob/8.x-1.x/.gitlab-ci.yml which includes https://gitlab.com/drupalspoons/composer-plugin/-/raw/master/templates/.... but that has lots of drupalspoons specific things which I don't think will/should apply here.

I think it'd be great if somebody who knows how this is supposed to work could pair with at least one Drupal contrib module maintainer, so that we can get over hurdles quickly, learn quickly, establish best practices quickly. If we don't do that, adoption is going to suffer.

Pairing with somebody who knows the intricacies and assumptions of the GitLab CI infrastructure is critical if this is to work at all (I'm speaking from experience — albeit with a different CI system than GitLab — pairing was the only way to get unblocked).

That being said … @moshe seems to successfully reuse Drupal Spoons's GitLab CI: https://git.drupalcode.org/project/keycdn/-/blob/8.x-1.x/.gitlab-ci.yml → https://git.drupalcode.org/project/keycdn/-/pipelines/4229/builds works! And that led me to https://gitlab.com/drupalspoons/composer-plugin, which is explicitly saying This project needs a new name and a new home (ideally managed by the Drupal Association). This plugin has 2 goals: and that first goal is CI template. Provide a golden path (see Spotify's definition of this term) that Drupal contrib projects can use with Gitlab CI..

Seems like @moshe has already succeeded here, and we can pretty much blindly follow his lead? 😊 Looks like he already made it super simple! 🤩🥳

Let's see. Some of the questions I have:

• Ok I can blindly copy https://gitlab.com/drupalspoons/composer-plugin/-/raw/master/templates/.... but why?!
• Am I right to use the wodby PHP container(?)
• Does it have the services I need?
• Are the artifacts spoons-named because other parts of the code require that?
• The composer plugin maybe?
• Looking at that is an entirely built out thing https://gitlab.com/drupalspoons/composer-plugin/-/raw/master/bin/setup does it assume certain workflows about drupalspoons? It names itself DrupalCI, which is not true. Its not DrupalCI?!
• I want to reproduce multiple test environments, yet this CI file only ever defines one, should I define them as separate pipelines or as separate tests in the same stage?
• Compared to the very restricted predefined enviroments of DrupalCI this offers practically unlimited freedom where I can run whatever code I want on the CI system. Is this intended?
• What if I use a PHP build that does crypto mining on the side or serves as a spam relay?
• How do we ensure interoperability of Drupal packages in the future, if compared to the DrupalCI platform now, they will be tested with arbitrary PHP containers that may or may not be compatible with each other?

I think various of these questions hide rabbit holes I can go down for days without doing anything useful.

I worked with @moshe on the composer plugin linked by @Wim Leers and what that project also wants to achieve is that developers can run the same tests, that we now start to make available for GitLab CI, also in their local environment with the same config. That way we can encourage developers to test before push and offload some of the work from d.o infrastructure.

Having said that, GitLab has added a lot of new functionality to their framework recently, and I'm just about to e.g. try their Code Quality template based on Code Climate. This is nicely documented at https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html

Another template I've started is to periodically build webpack applications for projects if upstream components had available updates.

Both experiments are currently under way in a tiny BPMN.iO project at https://git.drupalcode.org/project/bpmn_io

As I have 4-5 years experience with GitLab CI on our self-hosted GitLab instance, I'm happy to continue helping here to build a solid and configurable pipeline infrastructure for d.o contributors. And yes, continuing that on top of what's already present with DrupalSpoons seems reasonable.

I'm an alpha tester... WIP notes here:

1. I have used GitLab CI for GovCMS (Australian) projects so I am familiar with it but I wanted to explore as if I wasn’t familiar with it. I intentionally didn’t read the gitlab Drupal Slack discussions fully so I could go in mostly unaware of what I should do.
   
2. Looked around and did not see anything on the drupal.org project page or when editing the project that indicates GitLab CI is enabled.
3. Clicked on GitLab activity link on lower right of project page.
4. See GitLab CI/CD pages available: Pipelines, Editor, Jobs, Schedules.
5. Clicked on Pipelines and see there is a Use template button. Search the page for Drupal and there is no option… I already knew this but just trying to think like a new person would think ;)
6. Clicked on Use template button and see it has a simple .gitlab-ci.yml file I can start with. But I know that I can use examples from other alpha testers based on recent chats so I don’t want to do this.
7. Clicked on Editor and see Create new CI/CD pipeline button so I click on that. It takes me to the same thing as the Use template button via Pipelines.
8. Clicked on Jobs and see Create CI/CD configuration file button so I click on that and it takes me to the Editor page. Not the best UX IMO but it’s open source! :)
9. Clicked on Schedules and see New schedule button so I click on that and it has a Schedule a new pipeline form which looks like cron stuff but I don’t know what I should do with this so don’t fill out the form.
10. Decided I should find the documentation for this so Google GitLab CI and get GitLab CI/CD | GitLab. Lots of good stuff here.
11. Watch the intro ~5 minute video shown on the main docs page which is a decent intro but assumes people understands basics concepts of CI/CD and DevOps. It shows the workflow of the pipeline which is good and goes into the gitlab-ci.yml file. Pet peeve: all 3 of the personas are male.
12. Decided I want to focus on figuring out what to put in the gitlab-ci.yml file so search for docs specifically for that and find Keyword reference for the `.gitlab-ci.yml` file | GitLab linked from the main but it’s overwhelming so go to the quick start guide: Get started with GitLab CI/CD | GitLab
13. The get started guide says: “To use GitLab CI/CD:
    • Ensure you have runners available to run your jobs. If you don’t have a runner, install GitLab Runner and register a runner for your instance, project, or group.
    • Create a .gitlab-ci.yml file at the root of your repository. This file is where you define your CI/CD jobs.”

14. Looked at Settings > CI/CD > Runners and see that Enable shared runners for this project is enabled.
15. Going back to the .gitlab-ci.yml file, it has a good section on what this is, with the example code available, and where you can look after you add it: Get started with GitLab CI/CD | GitLab. I recommend this section for anyone who’s never used GitLab CI/CD before.
16. While I could commit the test .gitlab-ci.yml file, I know that there has been discussions in the Drupal Slack channel about the .gitlab-ci.yml file so I go back to Slack and find these.
17. Irina links to a meta issue specifically about creating a starter .gitlab-ci.yml file for people to use (this issue!): #3265092: [META] Define a default .gitlab-ci.yml template that projects can inherit
18. I read this issue to see that we have some examples we can leverage such as from keycdn and bpmn_io so my next step will be to look at these examples to see what I want to add to my .gitlab-ci.yml file. Stay tuned :)

I did just blindly copy https://git.drupalcode.org/project/keycdn/-/blob/8.x-1.x/.gitlab-ci.yml to be able to work around DrupalCI not allowing any module to be tested that has a direct or indirect dependency on a composer plugin (see @alexpott's comment at #3334914-10: Testing is broken because simplesamlphp/composer-module-installer contains a Composer plugin which is blocked).

And … it sort of works. It still is installing PHP 7.4 by default, but a simple addition fixes that:

variables:
  PHP_TAG:
    value: "8.1"

(which I found thanks to https://gitlab.com/drupalspoons/composer-plugin)

but unfortunately it does seem to run phpcs on the entire codebase, including all of vendor, rather than just the Drupal module being tested: https://git.drupalcode.org/project/cdn/-/jobs/36021. I'm 99% certain this is not happening for keycdn but is happening for cdn because the latter has a drupal/core entry in the require section.

This means adopting GitLab CI will require a significant number of existing projects to change their composer.json (validation for those files to ensure compatibility with the facade would not be a bad thing IMHO…), or the GitLab CI template that we currently have should introduce additional measures to avoid running phpcs on vendor. Although we probably need both.

Per https://www.drupal.org/docs/creating-custom-modules/add-a-composerjson-f..., it's better to effectively create a partial composer.json and let d.o's composer facade generate a drupal/core requiement based on the *.info.yml … or I guess alternatively one could use conflict?

Although even removing that seems to still run into this problem … 🤔😳 At this point I have to give up and move on to other responsibilities…

But it's definitely looking tentatively promising! See the MR at #3334243: CDN 4.x requires PHP >=8.1 but composer does not respect it due to bug in d.o composer facade.

@Wim: You can either just disable the phpcs check with DCI_SKIP_PHPCS: 1 or you can override and customize that job definition from the template and do whatever else you want there to skip the vendor folder. And see also the comment in my issue.

That said, I'm not quite sure why the drupalspoons composer plugin would bother to run composer install within the module at all or why you would end up with a vendor folder inside the module.

I've been working on getting tests working for redis based on select2 and made some progress (have a working double-matrix for d9/d10 and different redis clients), but currently stuck on the fact that apparently php assertions are disabled on the used docker php image/configuration which is causing some test fails and a bunch of other things, seems like Wim has the same problem in his module too.

Drupal 9 on PHP 8.1 is also causing issues due to guzzle, tried to switch over to drupal/core and update guzzle but didn't really manage that yet properly and feels ugly.

Also, the main issue is that if I want to test the Relay integration which is a non-standard extension, it seems like I would have to build, release and maintain my own Docker image that includes that, or maybe use an image that allows me to add apt repositores and packages, which would slow things down I guess.

The example for redis is here: #3340972: Set up GitlabCI.

That said, in the current state, without any d.o issue queue integration and no official patterns/templates, this seems to be in a more experimental/early phase than I assumed and you probably don't want to use it unless you really have to, like the redis case with requires extra extensions and services or that composer-plugin problem, which seems fixable with a one-liner in DrupalCI if we just allow any plugins for now.

I figured out a solution for the assertion problem, it's not very pretty though: https://git.drupalcode.org/project/redis/-/commit/83dd8a6b04bbb91eb33439...

I also solved the guzzle problem, with a combined D9 + PHP 7.4 and D10 + PHP 8.1 matrix.

Working with the template does feel quite challenging, it's basically 3 layers of the jobs, a setup script that's external than then again builds on a composer plugin, if any of what those things are doing aren't what you need (see the composer plugin problem above), then your only option is to copy & paste the whole thing. At least for the script. My understanding is that composer plugin was written for manual/local testing first, hence the composer aliases. It should probably be at least only a script.

It's also not quite clear to me how to manage some of the convenience we have right now in setting up multiple jobs, with easy to manage core + php versions + database backend (if I understood correctly, postgresql/sqlite support is kinda there in the script but actually disabled/commented out right now), recurring test and so on. Test Matrixes and jobs depending on specific combinations is a powerful feature but quite hard to understand and set up.

Being able to run tests in similar concurrency as we have now is also going to be critical for larger contrib projects and obviously core.

The latest template for GitLab-ci has been created here: drupal.org/project/gitlab_templates

https://git.drupalcode.org/project/gitlab_templates/-/blob/1.0.x/gitlab-ci/template.gitlab-ci.yml

When users in the GitLab WebIDE go to add a CI file it is presented as one of the template options.

Need to update the issue summary with instructions/screenshots

We've continued to make a lot of progress on this template in:
https://git.drupalcode.org/project/gitlab_templates/-/blob/1.0.x/gitlab-...

I would say this issue is in 'needs review' status.

• Our next steps are to provide general availability for all projects- knowing that the testing is primarily for Modern Drupal contrib.
• We'll continue to be working with @larowlan and team on a core testing template.
• And then will look to legacy D7 support.

I'm going through the children issues here. Most of it can be called done now.
GitLab CI adoption is going well and we have +1700 modules on it already (this number is from last month).

All children issues are fixed now.
Documentation for contrib is available here: https://www.drupal.org/docs/develop/git/using-gitlab-to-contribute-to-dr...

I am marking this one as fixed.