Agentic AI Testing / Quiz
LangSmith Evals as CI Gates
evaluate() in a GitHub Action is easy. Making the gate fast, affordable, and non-flaky enough that the team respects a red build, that is the craft.
- Difficulty
- Medium
- Format
- Quiz
- Points
- 150
- Estimate
- 10 min
// MISSION BRIEF
Your Mission
An eval that only runs on a laptop protects nothing. This quiz covers wiring LangSmith evaluations into CI: secrets, dataset versioning, thresholds, baselines, cost control, and the variance problem that makes naive gates flap.
Scoring: equal weight per question.
// FIRST CONTACT
Battle teaser
The minimum viable LangSmith CI gate is:
- AA cron job that emails a PDF nobody opens
- BPrinting the score without acting on it
- CLANGSMITH_API_KEY provided via CI secrets, evaluate() run against a pinned dataset, aggregate scores read from the results, and a nonzero exit when a threshold is breached
- DA README instructing developers to run evals locally
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
agenticlangsmithcieval-gates