Pipeline: access the job log

Hello bluegeek9

Where exactly do you get that message? Is it on the scheduled daily pipeline or in a MR pipeline?

Also, deleting the _SHOW_ENVIRONMENT_VARIABLES: 1 will have no effect on making the logs viewable. In fact, we do not want to delete that line, as the environment variables are useful.

I'm not a maintainer, but setting this to "needs more info" anyway, as that is the situation here.

Thanks. Yes I also cannot see that log. That one was five months ago, so just to check here is a recent pipeline and I can't see the log for any job https://git.drupalcode.org/project/address/-/pipelines/773467

Yes, it could be those things. We can investigate here. At least the MR pipeline shows the same problem.

Pretty sure CI_DEBUG_SERVICES has nothing to do with it (although I'd be okay removing that now, the thing it was trying to help debug has long since been resolved).

Can y'all view job logs for this pipeline? https://git.drupalcode.org/issue/address-2995680/-/pipelines/774078

If not, once you click “Request access” at https://www.drupal.org/project/address/issues/2995680 can you do so?

If you’re able to view the logs once you have access to that issue fork, I 100% understand the problem here, but have no idea how to fix it. 😢

https://git.drupalcode.org/project/address/-/pipelines/773467 was triggered by BojanZ, a maintainer of this project. Therefore, the pipeline ran in the parent project, and only other maintainers (like me) can view the logs, manually trigger jobs, etc.

https://git.drupalcode.org/project/address/-/pipelines/630447 is the pipeline for the job linked in #6, and was triggered by a commit I pushed. Again, since I’m a maintainer, that pipeline ran in the parent project.

In both cases, the pipeline URL starts with https://git.drupalcode.org/project/address.... If you see pipeline URLs like that, you're basically doomed.

Compare to the pipeline I linked above (https://git.drupalcode.org/issue/address...). That was triggered by a non-maintainer, so it ran in the issue fork project, not in the parent project. Once you click “request access”, GitLab gives your user an elevated role in the issue fork project, and that gives you permission to view jobs and manually trigger them.

This situation has gotten dramatically worse in the core issue queue from what appears to be an upstream regression in GitLab itself. See #3576458: [regression] Subsystem and Topics maintainers require access to re-run, trigger, or view tests for the gory details. But for now, us subsystem maintainers (who have a ‘developer’ role in the Drupal core project), are triggering pipelines for our core contributions to run in the parent project, not the issue fork, so we can’t manually trigger jobs, and in some cases, can’t view the logs. It’s a disaster.

Anyway, I’m sadly not sure there’s much we can do to prevent maintainers from triggering pipelines to run in the parent project. There are apparently some flags in GitLab that might control this behavior, but even as a project maintainer, I can’t see them for this project. I believe #3547431: Change global settings for all GitLab projects: fork pipelines and merge trains is trying to explore this, but the last check there revealed that ci_allow_fork_pipelines_to_run_in_parent_project was set to FALSE (at least for Drupal core). Not sure if/how to set that for other projects like Address. I’ll respond there again to ask.

TL;DR: If a commit is pushed by a project maintainer, for now at least, the resulting MR pipeline will run against the parent project. If a non-maintainer pushes a commit, it should run in the issue fork project. If the pipeline runs in the parent project, only maintainers can really do much with the pipeline. Although it’s a waste of resources, in the meanwhile, if you’re working on an issue that a maintainer pushed commits for, you might have to push an empty commit to the MR branch to trigger a new pipeline you can actually work with.

Alas,
-Derek

Well, great! I was wrong. According to #3547431-17: Change global settings for all GitLab projects: fork pipelines and merge trains CI_DEBUG_SERVICES could be to blame. Let’s try removing that

This will be a useful test, since I'm a maintainer and the pipeline is running in the parent project:

https://git.drupalcode.org/project/address/-/pipelines/774335

Yes I can see the logs in https://git.drupalcode.org/project/address/-/pipelines/774335

So we've learned that CI_DEBUG_SERVICES: 1 should be used temporarily, to invesigate problems, or maybe set via the UI for a manual pipeline, but it should not be added permanently to the .gitlab-ci.yml config if anyone other than maintainers want to see the logs.

I say "maintainer" in the general sense, as I'm not sure of the specifc new role labels in Gitlab which were allowing or denying access in this case.

Merged. Thanks, y'all!

Some more context on CI_DEBUG_SERVICES:

It is intended for debugging services. Services are additional containers booted to provide resources for the main test container (for Drupal tests the most common service is the Database container).

The setting enables the container logs to be captured. For most development these are not going to be necessary. Especially if you use the stock Drupal gitlab_templates system.

Some situations where these additional logs are useful:

• Suspect a service container is not starting correctly (modified ENTRYPOINT of a docker image or concern the container is not properly configured)
• Suspect services not fully initialized before tests start (Note: IIRC GitLab proceeds as soon as the ports response in each service container, if you know the container takes time to boot after the port is opened you may need a wait in your before_script section)
• Suspect a service is crashing (segfault) mid-test.

There are other scenarios where these can be useful, however the vast majority of the time the service debug logs are not likely what you want to enable to diagnose a fault (and if they are needed from the main testing container that is a log that should be directly captured as a job artifact or standard output during the test run).

For those who are familiar with ddev, CI_DEBUG_SERVICES is about the same as ddev logs -s db (any container except web) or for those who work directly with docker, docker logs any_container_except_your_php_or_web_container. If you wouldn't run those style of commands to debug the test failure you likely do not need CI_DEBUG_SERVICES.

Thanks for the additional context.

We had added it to help infra try to track down https://www.drupal.org/node/3414252 and IIRC, we did actually gather a little bit of useful evidence for them in the process. However, I promptly forgot all about it, moved on to way too many other things, and it was sitting in this .gitlab-ci.yml ever since. 😅 I didn't mean to leave it here all this time, it just happened. But thankfully, this whole episode reminded me about it and now it's gone. 🎉 As it should have been months ago. 😂

Cheers,
-Derek