Discussion: Install major providers in AI module kernel tests to catch WSOD regressions

I see the motivation source #3584344: The handleApiException signature change from Exception to Throwable breaks downstream providers (e.g. Anthropic) that override this method. exactly the kind of silent cross-module breakage that is easy to miss in review and painful to diagnose. That said, I think the proposed remedy introduces a coupling risk that outweighs the safety gain, and in my understanding maybe there are lighter-weight alternatives worth considering before going this route.

My assistant AI finding and I have a bunch of thoughts.

The core problem with installing provider modules in AI module CI

If AI module kernel tests install any of these modules (OpenAI, Anthropic, Ollama) then a broken release in any one of those four modules blocks AI module merges entirely + even for completely unrelated patches. We need to list the modules that are necessary to monitor. Additionally, we should onboard the maintainer to actively assist us in unblocking us.

The "advisory / allowed to fail" variant mentioned in the issue reduces that coupling, but also reduces the value: once it is non-blocking, developers learn to ignore it, and the signal degrades quickly.

And when we talk about version pinning I feel it helps but shifts the problem: someone now owns the job of tracking 4 provider releases and updating pins. In a community-maintained project that work tends to go undone, and stale pins quietly make the tests meaningless.

Some of my brainstormed ideas with the help of Claude code.

Alternative 1: a custom PHPStan rule in the AI module (catches at the source)

The specific regression in #3584344: The handleApiException signature change from Exception to Throwable breaks downstream providers (e.g. Anthropic) that override this method. widening \Exception to \Throwable on an overridable method is detectable statically without installing any provider module or running any PHP.

PHPStan at level 6+ checks Liskov Substitution Principle compatibility: if a subclass overrides a method with a narrower parameter type than the parent, it flags an error. The AI module currently runs PHPStan at level 1, which does not catch this class of error. But yes I understand that jumping the whole codebase to level 6 is not realistic, it would surface a large number of pre-existing errors that would take significant time to work through. That is not the right trade-off for a targeted problem.

A targeted fix would be a custom PHPStan rule (in theory, I will yet need to work on the script) that specifically detects parameter type widening on protected or public methods in abstract classes (or classes tagged @api). The rule would fail the AI module's own CI build before the breaking change is merged, with zero coupling to provider module health.

Concrete outcome: had this rule existed, the diff in #3584344: The handleApiException signature change from Exception to Throwable breaks downstream providers (e.g. Anthropic) that override this method. would have produced something like:

  OpenAiBasedProviderClientBase::handleApiException()
  Widening parameter type on an overridable method is a breaking change
  for all subclasses. Use a new method name instead.

No provider modules need to be installed. No version pins to maintain. The rule benefits all current and future providers automatically, with no action needed from provider maintainers.

Alternative 2: providers run PHPStan level 6 against AI contracts (inverse dependency)

Provider modules could add drupal/ai as a require-dev dependency and run PHPStan at level 6 in their own CI(but again of course, the provider maintainer won't accept this. LV6 checks add too much effort to maintain). PHPStan would then check each provider's subclass signatures against the AI module's base class declarations. When AI widens a parameter type, the provider's CI catches the incompatibility on the provider side:

  Parameter #1 $e of AnthropicProvider::handleApiException()
  should be compatible with parent method \Throwable

The limitation is timing: the provider's CI only runs when someone pushes to the provider repo, not when AI module changes. So there is still a gap between when the breaking change lands in AI and when the provider discovers it.

Hybrid path

These two approaches complement each other well:

1. Immediate: add a custom PHPStan rule to the AI module that flags parameter type widening on overridable methods. This blocks the breaking change before it merges, with no coupling.
2. Long-term: publish a phpstan-extension package (or document a phpstan.neon baseline) that provider modules can adopt. It would include the custom rule plus a level 6 config tuned for AI provider subclasses, making it easy for provider maintainers to opt in.
3. Optional: a nightly cross-repo PHPStan job that checks out each major provider at its latest release tag and runs static analysis with the current AI dev branch as a dependency. This is much lighter than the original proposal: no Drupal installation, no kernel boot, no WSOD risk and does not block AI module CI(I'm not sure if this is feasible or not, so it can be added benefit if we can do it).

I will try to share a draft the custom PHPStan rule if the approach gets traction.

Note: As I mentioned earlier, this comment and the research work were drafted with help of AI assistance and reviewed by me.