Agentic AI Testing / Scenario
The 95% Benchmark
A vendor claims 95% on their shiny agent benchmark. Read the fine print and decide how much of that number you should believe.
- Difficulty
- Hard
- Format
- Scenario
- Points
- 200
- Estimate
- 15 min
// MISSION BRIEF
Your Mission
A vendor pitches their coding agent: "95% success on our AgentBench-Pro suite." Your team is about to buy based on that number. Before you do, read how the benchmark was actually run.
Separate the real signal from the marketing.
// FIRST CONTACT
Battle teaser
First artifact
The vendor's benchmark methodology
Which methodology flaws inflate the vendor's 95%? (Select all that apply.)
- ASingle run with best-of-retries reported (hides non-determinism)
- BTiny toy tasks unlike production (weak construct validity)
- CSuccess judged by the agent's OWN generated tests (reward hacking / self-grading)
- DTasks published 14 months ago (likely training contamination)
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
agenticbenchmarkscontaminationreliabilityevaluation