This project is not covered by Drupal’s security advisory policy.
Drupal AI Benchmark is a community-driven, open benchmark for comparing how different AI models and AI coding harnesses perform on real Drupal work. It gives the community one place to see, in charts and hard numbers, which models are best suited to each kind of task, and whether cheaper or locally-hosted models can hold their own against expensive commercial APIs on Drupal-specific work.
This is not a Drupal module. It's a container-based (DDEV) test harness that runs automated benchmarks against a real Drupal codebase and reports reproducible scores.
The benchmarks
The project currently defines four benchmarks:
- Issue resolution. We take real Drupal issues that have already been fixed and generate a reverse patch that re-introduces the original bug. Each model then has to fix it. Success isn't judged by opinion: we run the project's own test suite and check whether the tests pass. This is a SWE-bench-style benchmark built specifically around the Drupal codebase.
- Code quality and best practices. We start from code that we know is bad: it breaks community coding standards and contains problems that the Audit module can detect. The Audit module's score becomes an objective yardstick. We record the score before and after each model edits the code, which tells us whether a model is capable of bringing code in line with Drupal best practices even when there's no neatly documented issue pointing at what's wrong.
- Tool and agent use. Several community modules expose Drupal to AI agents over MCP or an API. This benchmark measures whether a model is actually capable of driving the site through those tools and completing concrete tasks against content or configuration.
- Translation quality. We take the hardest strings to translate from Drupal core and ask each model to translate the same text into several languages. The existing community translation is treated as the gold standard, and each model's output is checked against it. As with the other benchmarks, we run this both ways: the harness out of the box with no prompting, and then again with translation-specific skills and prompts based on Drupal's community translation practices. This shows how well models translate on their own, and how much a good Drupal-aware prompt improves the result.
Two runs per model: out-of-the-box vs. Drupal-tuned
Every benchmark runs twice for each model and harness combination. First with the harness (Claude Code, OpenCode) exactly as it ships, no custom prompts. Then again with Drupal-specific skills, agents and prompts. Comparing the two runs shows how much good Drupal context actually buys you, and answers a practical question: is a cheap or local model with solid Drupal prompts good enough, or do you still need a far more expensive model to get there?
Models and cost
The comparison spans paid commercial APIs, cheap or free external APIs, and local models you can run on your own hardware. Because of that, the results speak to cost as much as to capability: not just which model is smartest, but which one makes sense for the money.
Tracking how AI-in-Drupal evolves
The benchmarks are re-run over time, so the project doubles as a quality control on the whole ecosystem. When a new model appears, or the community updates its prompts and skills, we can see whether results actually improved or whether something quietly regressed.
Contributing
Open an issue to request a new model or harness, or submit a merge request with the results of a run on your own hardware and GPU. Accepted runs are added to the published charts, so the community can compare models across different setups.
⚠️ Work in progress. This project is under active development and is not finished yet. The benchmarks, methodology and results are still being built out and will change. Expect breaking changes, incomplete tasks and scores that shouldn't be treated as final. Feedback and contributions are welcome, but don't rely on this yet for any real decision-making.
Project information
- Ecosystem: AI (Artificial Intelligence)
- Created by trebormc on , updated
This project is not covered by the security advisory policy.
Use at your own risk! It may have publicly disclosed vulnerabilities.
