AI & LLM Evals / Quiz
Regression Evals in CI
Ship an LLM change, run the evals, gate the merge. Except the outputs are non-deterministic, so a naive gate flaps. Here's how to make it hold.
- Difficulty
- Medium
- Format
- Quiz
- Points
- 150
- Estimate
- 10 min
// MISSION BRIEF
Your Mission
LLM apps need regression protection like any software, but non-determinism breaks the usual pass/fail gate. This quiz covers building an eval gate in CI that catches real regressions without flapping on noise.
Scoring: equal weight per question.
// FIRST CONTACT
Battle teaser
The core idea of a regression eval gate in CI is:
- AShip first, evaluate quarterly
- BOnly test after users complain
- CManually eyeball a few outputs before merge
- DRun the app against a golden set on each change and fail the build if key metrics drop below agreed thresholds
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
ai-evalsci-cdregression-testingnon-determinism