Steps to reproduce

Currently there is no way to automatically run empathy evaluations on a schedule. Site admins must manually navigate to the "Run Evaluation" form each time they want to benchmark their AI provider. This means model regressions (e.g., after a provider update) can go undetected until a user notices degraded quality.

Dries' roadmap specifically calls out "autonomous AI responding to triggers/schedules" as a key 2026 capability. Scheduled empathy evaluation is a natural fit.

Proposed resolution

Add a cron-based scheduled evaluation system:

1. New service `EmpathyScheduledEvaluator` that runs evaluations on a configurable frequency (6h / 12h / 24h / 48h / weekly).
2. `hook_cron()` implementation that checks the last run timestamp via `StateInterface` and triggers evaluations when due.
3. Settings form additions under a "Scheduled Evaluation" section with enable/disable toggle, frequency selector, scenario selection, provider/model selector, and alert email address.
4. Email alerts via `hook_mail()` when any metric drops below the configured threshold.
5. Model switch detection: When the site's AI provider config changes, automatically trigger an evaluation suite and notify admins of score changes.
6. Lightweight execution: 1 run per scenario per cron cycle to avoid long-running cron.
7. Config schema updates and `update_10001()` hook for existing installs.

Remaining tasks

- [ ] Create `EmpathyScheduledEvaluator` service
- [ ] Add `hook_cron()` and `hook_mail()` to module file
- [ ] Update config schema and install defaults
- [ ] Add scheduled evaluation section to settings form
- [ ] Add update hook for existing installs
- [ ] Write tests for the scheduled evaluator service
- [ ] Manual testing: enable scheduled evaluation, run cron, verify dblog and email alerts

Issue fork ai_empathy-3583741

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

joshua1234511 created an issue. See original summary.

joshua1234511’s picture

joshua1234511’s picture

Version: 1.0.2 » 1.0.3
joshua1234511’s picture

Assigned: Unassigned » joshua1234511

  • 7fd7da22 committed on 1.0.x
    Issue #3583741: Add scheduled cron-based empathy evaluation with...
joshua1234511’s picture

Status: Active » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

joshua1234511’s picture

Status: Fixed » Closed (fixed)