AI Agent Frameworks / Scenario
The Crew That Ships Different Answers
A nightly CrewAI pipeline produces a different compliance report every run, and the exact-match test flakes. Diagnose what is actually wrong.
- Difficulty
- Hard
- Format
- Scenario
- Points
- 200
- Estimate
- 14 min
// MISSION BRIEF
Your Mission
A fintech runs a two-agent CrewAI pipeline nightly to draft a compliance summary. Three consecutive runs produced three different reports; one was missing the "Data Retention" section entirely. The only test asserts byte-equality against last month's output, and it fails roughly every night.
Read the crew, the run diffs, and the test. Decide what is a bug, what is physics, and what a real test suite looks like here.
// FIRST CONTACT
Battle teaser
First artifact
crew.py
Why do the three runs differ? (Select all that apply.)
- ANo structured output is enforced (no output_pydantic / output_json), so nothing guarantees the section list
- BThe {week} input changed between the three runs
- CLLM sampling is non-deterministic, identical inputs legitimately produce different phrasing and ordering
- DThe expected_output fields are vague ('useful notes', 'a good report'), so the model has no structural contract to hold
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
crewaideterminismstructured-outputflakinessai-agents