Steps to reproduce

The current human rating system allows users to rate AI responses on empathy (1-5) and explanation quality (1-5) with optional notes. However, there is no way to measure whether multiple raters agree with each other (inter-rater reliability), no way to compare two AI responses side-by-side without bias, and no feedback mechanism to help raters calibrate their scoring consistency.

Proposed resolution

Enhance the human rating system for more rigorous evaluation:

1. Inter-rater reliability score: When multiple humans rate the same response, calculate Cohen's kappa (for 2 raters) or Fleiss' kappa (for 3+ raters) as an agreement metric. Display this on the result view page alongside individual ratings.
2. Blind comparison mode: New page/form that presents two AI responses (from different models) to the same scenario side-by-side. The rater scores both without knowing which model generated which. After submission, reveal the models and show comparative results.
3. Rating calibration dashboard: Show each rater how their scores compare to the consensus average. Highlight raters who consistently score higher or lower than peers. This helps teams develop consistent scoring standards.
4. Rating statistics: On the dashboard, show aggregate rating stats — total ratings, average human vs AI score divergence, most-rated scenarios.

Remaining tasks

- [ ] Implement inter-rater reliability calculation (Cohen's/Fleiss' kappa)
- [ ] Display reliability score on result view pages
- [ ] Create blind comparison form and route
- [ ] Build rating calibration dashboard section
- [ ] Add aggregate rating statistics to main dashboard
- [ ] Write tests for kappa calculation
- [ ] Manual testing with multiple raters

Issue fork ai_empathy-3583745

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

joshua1234511 created an issue. See original summary.

joshua1234511’s picture

joshua1234511’s picture

Version: 1.0.2 » 1.0.3

joshua1234511’s picture

Status: Active » Needs review

  • joshua1234511 committed f0d501c5 on 1.0.x
    Issue #3583745: Add inter-rater reliability, blind comparison and rating...
joshua1234511’s picture

Status: Needs review » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

joshua1234511’s picture

Assigned: Unassigned » joshua1234511
joshua1234511’s picture

Status: Fixed » Closed (fixed)