LLM evaluation · judge calibration

LLM-as-Judge Evaluation Harness

Agreement: judge vs humans, against the human ceiling

Quadratic-weighted kappa. A judge can't be expected to beat human-to-human agreement.

Does the judge name the right problem?

Recall of the judge's primary issue against the planted issue.

Length bias check

Each dot is a response. Above zero means the judge rated helpfulness higher than the humans did.

Human review queue

Items routed to a human: a 2+ point gap with humans, a safety disagreement, or low judge confidence.