Reinforcement Learning from Human Feedback (RLHF) has become the standard technique for aligning large language models with human preferences. But the quality of the preference data — the pairs of model responses labelled "chosen" vs "rejected" by human annotators — determines everything. Bad preference pairs produce misaligned models. And nowhere is the risk of bad preference pairs higher than in Arabic.
What a Preference Pair Actually Is
A preference pair consists of a prompt (the input to the model), a chosen response (the one the human annotator prefers), a rejected response (the one the annotator disprefers), and an optional rationale explaining why the chosen response is better. These pairs train the reward model in PPO-based RLHF, or serve directly as training data in Direct Preference Optimisation (DPO).
Where Arabic Preference Annotation Goes Wrong
Formality confusion. MSA and colloquial Arabic are both correct in their registers. An annotator from a formal academic background may consistently prefer MSA responses even when the prompt is in Egyptian dialect — producing a reward signal that pushes the model toward inappropriate formality in conversational contexts.
Religious sensitivity blind spots. Arabic responses frequently invoke religious expressions (inshallah, alhamdulillah) that are culturally neutral in context but may be incorrectly flagged by non-native annotators applying generic safety guidelines.
Code-switching handling. Many real Arabic prompts contain English technical terms. A "chosen" response that correctly preserves the English term may be incorrectly downgraded by an annotator who prefers full Arabic translation — even when translation would be unnatural.
The AXOVATE Framework
Our RLHF annotation guidelines for Arabic cover dialect-register matching (responses should match the register of the prompt), cultural neutrality calibration through shared calibration sets before live annotation begins, structured rationale format coded along dimensions (accuracy, naturalness, cultural fit, safety), and double annotation with adjudication by a senior linguist for conflicts.
IAA on RLHF preference tasks in Arabic reaches Krippendorff's α ≥ 0.76 with this framework — above the published average for English preference annotation datasets.