Inter-annotator agreement (IAA) is the statistical measure of how consistently multiple human annotators apply the same labels to the same data. It is the primary quality signal for any annotation project. But IAA is systematically harder to achieve — and systematically less well-understood — for low-resource languages like Arabic and Hebrew than for English.
Why IAA Is Harder for Arabic and Hebrew
Orthographic ambiguity. Unvowelised Arabic and Hebrew allow multiple valid readings of the same written word. Two annotators labelling the same token may assign different labels — both correctly — because the written form is genuinely ambiguous without context.
Dialect variation. A label applied in MSA may not transfer to Egyptian Arabic. Guidelines written for one dialect produce systematic disagreement when applied by annotators from another dialect background.
Thin calibration corpora. English annotation projects can draw on large, publicly available calibration datasets. For Arabic dialects and Hebrew, such datasets are sparse — calibration must be built from scratch for each project.
Cohen's Kappa vs Krippendorff's Alpha
Cohen's κ measures pairwise agreement between exactly two annotators on a categorical task. It is the right metric for binary classification and simple multi-class tasks with exactly two annotators. Target: κ ≥ 0.80 for production annotation.
Krippendorff's α is the right metric when you have more than two annotators, ordinal or interval scales (like 1–5 rating tasks in RLHF), or tasks with missing data. Target: α ≥ 0.75 for production annotation. A common mistake is using Cohen's κ on an RLHF preference task with a 1–5 scale — the ordinal relationship is discarded, producing an inflated and misleading score.
Practical Steps to Improve IAA
1. Build a dialect-specific calibration set of 50–100 items with gold-standard labels before live annotation begins. 2. Write dialect-explicit guidelines with examples from that specific dialect. 3. Run a 200-item mini-pilot and measure IAA before committing to full volume. 4. Log disagreement patterns per label, not just the aggregate score — a systematic disagreement on one label is more actionable than a clean average. 5. Budget for adjudication: approximately 15–20% of items will need resolution by a senior linguist on a well-run project.
What to Ask a Vendor
The four questions that reveal genuine quality practice: What IAA metric do you use, and why? What is your calibration process before live annotation starts? Do you provide per-batch IAA reports, or only aggregate scores? What happens to a batch that does not reach the agreed IAA threshold?
AXOVATE provides per-batch IAA reports with every delivery and re-annotates batches that fall below threshold at no charge.