LLM evaluation · judge calibration
LLM-as-Judge Evaluation Harness
Agreement: judge vs humans, against the human ceiling
Quadratic-weighted kappa. A judge can't be expected to beat human-to-human agreement.
Does the judge name the right problem?
Recall of the judge's primary issue against the planted issue.
Length bias check
Each dot is a response. Above zero means the judge rated helpfulness higher than the humans did.
Human review queue
Items routed to a human: a 2+ point gap with humans, a safety disagreement, or low judge confidence.