Agentic AI Testing / Scenario
Detecting Planning Failures at Scale
42,000 runs a week. Nobody reads traces at that volume. Your job: turn loop, stall, and drift pathologies into automated detectors with alerts.
- Difficulty
- Hard
- Format
- Scenario
- Points
- 200
- Estimate
- 14 min
// MISSION BRIEF
Your Mission
Your company's research agent handles 42,000 runs weekly. Support says users complain it "takes forever then gives up", cost is up 22%, and the only monitoring is a service-uptime ping.
You cannot read 42,000 traces. Design the detectors that find planning failures automatically, and decide what happens when they fire.
// FIRST CONTACT
Battle teaser
First artifact
Weekly ops digest
Which automated detectors should run over production traces? (Select all that apply.)
- AForced-stop rate as a first-class failure metric with an alert threshold
- BHire a reviewer to read every trace manually
- CStep-count outlier detector: runs far above the baseline for their task category
- DGoal-relevance heuristic: flag tool calls unrelated to the request category (get_weather in a supplier search)
- ENear-duplicate call detector: N consecutive calls to the same tool with trivially rephrased arguments and overlapping results
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
agenticplanningmonitoringproductiondetection