AI data quality ยท weekly QA

Annotation QA Analyzer

Findings and actions

Sorted by severity. Every finding comes with the action a QA lead would take this week.

Root causes of disagreement

Each label that disagrees with the other raters, assigned to one cause.

Agreement by content type

Fleiss' kappa. Low agreement across the whole team points to the guideline, not the people.

Weekly agreement: team vs drifting annotators

Share of labels matching the other raters' majority. Drift starts in week 5.

Annotator scorecard

Lowest agreement first.