Steps to reproduce
Currently there is no way to automatically run empathy evaluations on a schedule. Site admins must manually navigate to the "Run Evaluation" form each time they want to benchmark their AI provider. This means model regressions (e.g., after a provider update) can go undetected until a user notices degraded quality.
Dries' roadmap specifically calls out "autonomous AI responding to triggers/schedules" as a key 2026 capability. Scheduled empathy evaluation is a natural fit.
Proposed resolution
Add a cron-based scheduled evaluation system:
1. New service `EmpathyScheduledEvaluator` that runs evaluations on a configurable frequency (6h / 12h / 24h / 48h / weekly).
2. `hook_cron()` implementation that checks the last run timestamp via `StateInterface` and triggers evaluations when due.
3. Settings form additions under a "Scheduled Evaluation" section with enable/disable toggle, frequency selector, scenario selection, provider/model selector, and alert email address.
4. Email alerts via `hook_mail()` when any metric drops below the configured threshold.
5. Model switch detection: When the site's AI provider config changes, automatically trigger an evaluation suite and notify admins of score changes.
6. Lightweight execution: 1 run per scenario per cron cycle to avoid long-running cron.
7. Config schema updates and `update_10001()` hook for existing installs.
Remaining tasks
- [ ] Create `EmpathyScheduledEvaluator` service
- [ ] Add `hook_cron()` and `hook_mail()` to module file
- [ ] Update config schema and install defaults
- [ ] Add scheduled evaluation section to settings form
- [ ] Add update hook for existing installs
- [ ] Write tests for the scheduled evaluator service
- [ ] Manual testing: enable scheduled evaluation, run cron, verify dblog and email alerts
Issue fork ai_empathy-3583741
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
joshua1234511Comment #3
joshua1234511Comment #4
joshua1234511Comment #5
joshua1234511Comment #8
joshua1234511Comment #10
joshua1234511