AI & LLM Evals / Quiz
OpenAI Evals: Registry, Samples, oaieval
The open-source evals framework: YAML registry entries, JSONL samples with input and ideal, basic match classes, and when to reach for model-graded evals.
- Difficulty
- Easy
- Format
- Quiz
- Points
- 100
- Estimate
- 8 min
// MISSION BRIEF
Your Mission
OpenAI's open-source evals framework popularized a pattern the whole industry copied: evals as registry entries (YAML) pointing at datasets (JSONL) and eval classes, run from one CLI. Knowing its shape makes every other eval tool feel familiar.
Six questions on the registry, sample format, match classes, and model-graded evals. Speed bonus active.
// FIRST CONTACT
Battle teaser
In the evals framework, a new eval is registered by:
- AA YAML entry in the registry mapping the eval name to an eval class (e.g. evals.elsuite.basic.match:Match) plus args like the samples_jsonl path
- BUploading a spreadsheet through the API dashboard
- CEmailing the dataset to OpenAI for approval
- DA Python decorator on your model code
Answers, scoring, hints, and the full battle stay sealed.
// SKILL TAGS
openai-evalseval-registrymodel-gradedllm-evals