Agentic AI Testing / Quiz
Designing Agent Benchmarks
A benchmark that leaks into training or runs once is a vanity metric. Real agent benchmarks are contamination-aware, reliability-aware, and hard.
- Difficulty
- Medium
- Format
- Quiz
- Points
- 150
- Estimate
- 10 min
// MISSION BRIEF
Your Mission
Agent benchmarks (tau-bench, SWE-bench, WebArena, GAIA, and internal ones) try to measure real capability. Design them badly and the score is theater. This quiz covers the properties that make a benchmark trustworthy.
Scoring: equal weight per question.
// FIRST CONTACT
Battle teaser
Benchmarks like SWE-bench and WebArena are notable because they:
- AMeasure typing speed
- BTest agents on realistic multi-step tasks (fixing real repo issues, completing web tasks) with verifiable outcomes
- COnly ask single trivia questions
- DGrade the agent's prose style
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
agenticbenchmarksevaluationcontaminationreliability