AI Agent Frameworks / Scenario
Boss: The Agent Release Gate
New model, new prompts, Friday deadline. LangGraph app, LangSmith evals, CrewAI content crew. You own the gate. Averages look fine. Look closer.
- Difficulty
- Boss
- Format
- Scenario
- Points
- 300
- Estimate
- 22 min
// MISSION BRIEF
Your Mission
You are the QA lead for Atlas, a customer-support agent: a LangGraph app with a human-review interrupt for big refunds, tools for orders and payments, LangSmith tracing plus eval pipelines, and a CrewAI crew that drafts KB articles. Release candidate v2.4 swaps the base model and rewrites two prompts.
The release decision is yours. You have the gate checklist, the CI setup, the experiment comparison, and a trajectory diff. This is the boss fight: read the evidence like a professional, weight what matters, and make the call.
// FIRST CONTACT
Battle teaser
First artifact
.github/workflows/agent-ci.yml (excerpt)
The release manager points at overall correctness (0.86 to 0.88). What does the slice table actually say?
- AEverything improved; the table confirms the average
- BLatency improved, which offsets the refund slice
- CThe slices are too small to ever matter
- DThe average hides a collapse: refund-policy correctness fell 0.95 to 0.71 while bigger slices improved. Averages across slices are not a gate; per-slice thresholds are
// SKILL TAGS