AI Agent Frameworks / Quiz
Feedback Loops and Regression on Traces
Thumbs-down events, annotation queues, online evaluators. Production traffic is the best eval dataset you are not using yet.
- Difficulty
- Hard
- Format
- Quiz
- Points
- 200
- Estimate
- 10 min
// MISSION BRIEF
Your Mission
Pre-release evals test what you thought of. Production tells you what you missed, IF you wire feedback to runs, review samples systematically, and convert bad traces into permanent regression cases. This quiz covers closing that loop in LangSmith.
Scoring: equal weight per question.
// FIRST CONTACT
Battle teaser
A user clicks thumbs-down in your app. The right LangSmith wiring:
- AEmail the complaint to the team alias
- Bclient.create_feedback(run_id, key="user_score", score=0), attaching the signal to the exact run that produced the answer
- CFeedback cannot be recorded programmatically
- DIncrement a counter in Redis, unlinked to anything
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
langsmithfeedbackregressionmonitoringai-agents