Steps to reproduce
The current human rating system allows users to rate AI responses on empathy (1-5) and explanation quality (1-5) with optional notes. However, there is no way to measure whether multiple raters agree with each other (inter-rater reliability), no way to compare two AI responses side-by-side without bias, and no feedback mechanism to help raters calibrate their scoring consistency.
Proposed resolution
Enhance the human rating system for more rigorous evaluation:
1. Inter-rater reliability score: When multiple humans rate the same response, calculate Cohen's kappa (for 2 raters) or Fleiss' kappa (for 3+ raters) as an agreement metric. Display this on the result view page alongside individual ratings.
2. Blind comparison mode: New page/form that presents two AI responses (from different models) to the same scenario side-by-side. The rater scores both without knowing which model generated which. After submission, reveal the models and show comparative results.
3. Rating calibration dashboard: Show each rater how their scores compare to the consensus average. Highlight raters who consistently score higher or lower than peers. This helps teams develop consistent scoring standards.
4. Rating statistics: On the dashboard, show aggregate rating stats — total ratings, average human vs AI score divergence, most-rated scenarios.
Remaining tasks
- [ ] Implement inter-rater reliability calculation (Cohen's/Fleiss' kappa)
- [ ] Display reliability score on result view pages
- [ ] Create blind comparison form and route
- [ ] Build rating calibration dashboard section
- [ ] Add aggregate rating statistics to main dashboard
- [ ] Write tests for kappa calculation
- [ ] Manual testing with multiple raters
Issue fork ai_empathy-3583745
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
joshua1234511Comment #3
joshua1234511Comment #4
joshua1234511Comment #6
joshua1234511Comment #8
joshua1234511Comment #10
joshua1234511Comment #11
joshua1234511