AI & LLM Evals / Quiz
The Judge's Hidden Biases
Your LLM judge prefers longer answers, agrees with whoever asked, and rates by format. If you don't correct for it, you're scoring its prejudices.
- Difficulty
- Hard
- Format
- Quiz
- Points
- 200
- Estimate
- 11 min
// MISSION BRIEF
Your Mission
Beyond position and self-preference, LLM judges carry subtler biases: verbosity, sycophancy, and format preference. This quiz covers the biases that quietly corrupt judge scores and how to counter them.
Scoring: heavier timer, equal weight per question.
// FIRST CONTACT
Battle teaser
A judge consistently rates longer answers higher, even when a short answer is equally correct. This is:
- AVerbosity/length bias, mitigate by controlling for length or instructing the rubric to ignore it
- BA parse error
- CPosition bias
- DCorrect; longer is better
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
ai-evalsllm-as-judgebiassycophancyverbosity-bias