AI & LLM Evals / Quiz
LLM-as-Judge Reliability
Position bias, self-preference, calibration, and when to trust a model grading another model. Meta-evaluation for AI testers.
- Difficulty
- Hard
- Format
- Quiz
- Points
- 200
- Estimate
- 9 min
// MISSION BRIEF
Your Mission
Using an LLM to grade LLM outputs (G-Eval, judge models) scales evaluation, but judges have their own biases. If you don't validate the judge, you're measuring noise. This quiz covers making LLM-as-judge trustworthy.
// FIRST CONTACT
Battle teaser
When a judge model compares answer A vs answer B, it tends to favor whichever is presented first. This is:
- AA network issue
- BToken limit
- CPosition bias, mitigate by swapping order and averaging both orderings
- DTemperature drift
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
ai-evalsllm-as-judgeg-evalbiascalibration