AI data quality ยท weekly QA
Annotation QA Analyzer
Findings and actions
Sorted by severity. Every finding comes with the action a QA lead would take this week.
Root causes of disagreement
Each label that disagrees with the other raters, assigned to one cause.
Agreement by content type
Fleiss' kappa. Low agreement across the whole team points to the guideline, not the people.
Weekly agreement: team vs drifting annotators
Share of labels matching the other raters' majority. Drift starts in week 5.
Annotator scorecard
Lowest agreement first.