Enable Live Previews for projects with issues on GitLab

Just for information, do you know if the Tugboat preview is still built OK when the issue is on Gitlab? Is it just that the link is needed?

I assume that we'll need to communicate with the tugboat api to retrieve the URL.

From the above, it seems to be MR-based branch-based, so it's not a predictable link and therefore cannot be put in the first automated comment.

Maybe the automatic comment might be better suited in the MR rather than the issue, or in the fork-management page, where we list branches and MRs.

In D7, the information is obtained here https://git.drupalcode.org/project/drupalorg/-/blob/7.x-3.x/drupalorg/in... and then passed to a JS script to render the link.

Also, the building of the preview doesn't seem to be automatic based on this https://git.drupalcode.org/project/drupalorg/-/blob/7.x-3.x/drupalorg/in...

Maybe the automatic comment might be better suited in the MR rather than the issue, or in the fork-management page, where we list branches and MRs.

Yes, either of those places sound more reasonable than in the issue. I think there would be a Tugboat build per MR, so having the links in the issue would be confusing if there was more than one MR.

Tugboat itself might do more of the automated comments/etc with everything on GitLab. We may still have special requirements that separate the Drupal.org use case from the usual Tugboat customer. In particular, not every Drupal.org user has access to the Tugboat dashboard, so that’s why we needed our own rebuild/logs UI. We should take a step back and work with Tugboat to figure out what the integration should look like today.

Asked in the #tugboat-support public slack channel: https://drupal.slack.com/archives/C01FA5V1CG0/p1781540422316569

Hi everyone! Exciting stuff here. Let me lay out how the integration works today, and where we'd like to take it, because I think it answers some of the questions above.

How it works today

The integration is driven entirely from the Drupal.org side (the drupalorg D7 module), talking to our hosted service at api.tugboatqa.com:

1. GitLab MR webhook → Drupal.org. When a merge request is opened or reopened, GitLab fires a webhook to a Drupal.org endpoint (authenticated with a shared X-Gitlab-Token). Drupal.org records the MR/branch and queues the preview build.
2. Config resolution differs for core vs. contrib:
   • Contrib projects that ship a .tugboat/config.yml are picked up automatically. We read that file straight from the MR's source branch and let Tugboat use the project's own config.
   • Drupal core has no .tugboat/config.yml, so Drupal.org fetches a config we maintain in a separate GitHub repo (TugboatQA/drupalorg), keyed by branch, and passes it to Tugboat at build time.
3. Repository auto-registration. The first time an MR comes in for a project we haven't seen, Drupal.org registers it as a Tugboat Repository on the fly (briefly adding our service user to the GitLab project to authorize the clone, then removing it). After that, the preview is built (type=pullrequest), tied to the MR.
4. Surfacing results. Drupal.org calls our API to fetch preview URLs, build state, and logs, and renders them on the issue fork UI, along with "rebuild" and "view log" actions.

So to answer the question raised above: previews are being built for projects with a valid config. What's missing for GitLab-issues projects is the link surfacing, because that rendering is wired into the Drupal.org issue node UI, not into GitLab. That's the crux of why this needs a rethink rather than a patch.

Where we'd like to take it

As issues move to GitLab, we'd like to move this integration onto an On-Premise Tugboat instance wired directly to Drupal's GitLab, rather than keep driving it through webhooks and API proxying. The big win: with GitLab OAuth as the login provider, GitLab users get direct access to the Tugboat dashboard (preview URLs, build logs, and rebuilds, as well as new features such as Google Lighthouse reports and Visual Regression Testing) without Drupal.org / GitLab having to proxy any of it.

On the "where should the links live?" question: Tugboat already has a built-in setting to post a comment back on the merge request with a link to the testing environment (it keeps that comment updated as the preview rebuilds). It's actually disabled in our current Drupal.org integration, but enabling it means the preview link lands right on the MR with no custom logic on anyone's side, which also sidesteps the "multiple MRs per issue" confusion from posting links on the issue.

Access control comes along for free too. Tugboat already has a Sync Users option in Repository Settings: GitLab users with the Maintainer role (or higher) on a project automatically get access to that repo in Tugboat, and if their role is revoked in GitLab it's revoked in Tugboat too. So permissions stay entirely governed by GitLab, there's no separate Tugboat user list to manage.

To get there, three things need solving:

1. Commit .tugboat/config.yml to Drupal core branches. This removes the special-case core path entirely — core would be picked up exactly like contrib, with no externally-maintained config repo and no custom build plumbing. There's already an issue for this (#3280730: Add Tugboat configuration to core) that stalled on a concern about the config relying on Drush; worth reviving and settling on an install approach (e.g. core's native install script) so this can land.
2. Auto-add Repositories for contrib when a config.yml is present. In the On-Premise model we'd want Tugboat to register a project directly off GitLab (e.g. via a group/system hook or a scan for .tugboat/config.yml), instead of relying on the Drupal.org webhook as the trigger. End result is the same "drop in a config.yml and it just works," but we will need some other home for the automation.
3. Stand up the On-Premise instance. This needs (A) a wildcard domain (e.g. *.tugboat.drupalcode.org or similar) pointing at a host we run, and (B) a GitLab OAuth application (client key/secret) from the self-hosted GitLab for dashboard login and the Sync Users access described above. @drumm and I spoke briefly about this at DrupalCon, and we would need access to a dev environment to kick the tires on for this.

Happy to jump on a call to scope the On-Premise setup and the GitLab OAuth / DNS specifics whenever the team wants to dig in.

instead of relying on the Drupal.org webhook as the trigger

This could be the same general webhook handoff, with the implementation moved to new.drupal.org. That will likely be the most maintainable long-term, since we have the webhook deliveries set up there already.

Currently D7 is storing in the database a few things tugboat related, does this set up means that we wouldn't need to? We can either setup the webhooks to tugboat or trigger tugboat calls from new.drupal.org (via the webhook receiver).

It seems from the above, that tugboat can handle the GitLab integration side of things, which would be great, and and would mean that the new.drupal.org integration would be much lighter.

I recall a lot of what we store in D7 is to avoid making API requests to Tugboat while rendering issue pages for people to view. If Drupal handling webhooks is purely queue processing, that can be stateless and make API requests for everything.

If we end up with a Drupal UI for anything, the initial page load should not have any API requests, the front-end can make them as/if needed. We might not need that with how GitLab & Tugboat will work together.

I think there might have been some heavier API requests, paging through a list of everything to find one item, that’d be best to avoid altogether.

For transitioning, projects will be migrating over the next few weeks. When the new Tugboat integrations are in a good place, we can bulk migrate all remaining contributed projects with Tugboat. That way we have less to worry about for a transition period.

However, core will almost certainly be the last project to migrate and may be delayed quite awhile behind contrib, see #3589620: [meta] Requirements for migrating DrupalCore to GitLab issues