Online vs Offline LLM Evaluation Interview Questions

Senior AI QA work requires two different measurement systems to cooperate. Offline evaluation supplies controlled comparisons before release. Online evaluation reveals behavior under real requests, dependencies, users, and product incentives. An interview answer should never collapse them into the same score or frame one as the mature replacement for the other.

These 19 questions test whether a candidate can choose the right evidence for a decision, connect proxy metrics to outcomes, control production exposure, and feed newly observed failures back into a reproducible suite. The expected answer includes data lineage, slice analysis, privacy, and a response when the signals disagree.

Define the Two Evidence Environments

LangSmith's evaluation concepts distinguish offline work on datasets and examples from online work on production runs and threads. The useful interview move is to name what each environment knows: offline cases may have reference outputs and controlled replay, while live traces reveal actual distribution and product context but often lack immediate ground truth.

The loop is only credible if identifiers, versions, eligibility rules, and delayed outcomes can be joined without pretending that sampled traces represent all traffic.

Choose Evidence Before Release

1. Why does a new prompt need offline evaluation before an online experiment?

Offline cases can expose known safety, correctness, formatting, and tool-use regressions without placing users in the candidate path. I would run the baseline and candidate on the same versioned examples, inspect critical slices, and require deterministic veto checks before planning limited exposure. Offline success does not predict every live outcome, but it removes failures already represented in the team's evidence. The tradeoff is slower iteration versus avoiding experiments that had no reasonable case to reach production.

2. How would you build an offline set for a feature with little historical traffic?

Start from the product specification, threat model, domain sources, and adjacent workflow incidents. Create observed-like cases and carefully marked synthetic boundary cases, then have domain reviewers validate expected behavior. Separate development cases from a blind holdout and record source provenance. The set will have uncertain prevalence, so report risk coverage rather than claiming production representativeness. Early online sampling should be designed to discover missing intents, not merely to confirm the handcrafted set.

3. What makes an offline benchmark unsuitable as a release gate?

A benchmark is unsuitable when its cases are exposed to tuning, labels are disputed, coverage no longer matches product behavior, the evaluator cannot distinguish critical failures, or results cannot be reproduced from pinned inputs and configuration. I would audit lineage and review failed and passed examples around the proposed boundary. Popularity or historical use is not sufficient evidence. A narrower governed set can support a release decision better than a large benchmark whose relationship to risk is unknown.

Design Online Evaluation Safely

4. How would you introduce online evaluation for a high-impact assistant?

Begin with the existing production output and run reference-free monitors, trace validation, and human review without changing user behavior. Next evaluate a shadow candidate on eligible traffic with side effects disabled. Only after offline gates and shadow review would I allow a small, reversible cohort under explicit stop conditions. This progression trades speed for bounded harm. The answer should identify who can stop exposure and how affected requests are found, not just name an A/B platform.

5. Why is random online sampling insufficient for monitoring quality?

Random sampling estimates common behavior but can miss rare languages, tools, intents, account states, and severe failures. I would combine a representative sample with risk-triggered strata based on policy flags, unusual trajectories, errors, and low-confidence proxies. Reports keep the strata separate or apply known weights. Targeted sampling increases detection but distorts prevalence, so it must not be summarized as if every retained run had equal inclusion probability.

6. How would you evaluate a live run when no reference answer exists?

Use layered evidence: deterministic policy and schema checks, reference-free rubric graders, trajectory invariants, user feedback, and delayed product outcomes. Escalate uncertain or high-risk cases to human review. I would calibrate automated judgments on a labeled production sample and preserve reasons or component results, not only a scalar. The limitation is explicit: some correctness questions cannot be resolved at request time, so the monitor should route evidence rather than manufacture confidence.

Connect Proxies to Product Outcomes

7. What would you do if helpfulness scores rise but task completion falls?

First verify the join, cohort, and task-completion definition. Then inspect whether the grader rewards fluent explanation while the workflow needs decisive tool execution, and compare trajectories for abandoned or repeated steps. I would relabel a stratified sample against the product outcome and revise the rubric or add a completion evaluator. The live outcome should dominate the flattering proxy when instrumentation is sound, though confounding product changes must be ruled out before blaming the model.

8. How would you handle outcomes that arrive days after the LLM response?

Emit a stable, privacy-safe interaction identifier and record candidate, prompt, evaluator, and cohort versions at serving time. Join later events through a governed pipeline with a declared attribution window and rules for multiple interactions. Track missing joins and censoring by cohort. I would avoid converting every absent event into failure because the observation window may be incomplete. Delayed labels improve realism, but they slow rollback evidence and can be confounded by downstream experiences.

9. Why should online evaluator drift be monitored independently from model drift?

An evaluator's provider, prompt, rubric, parsing, or input fields can change while the application stays constant. Preserve a frozen calibration set of production examples and re-run it whenever the evaluator changes. Monitor parse failures, score distributions by slice, and disagreement with human review. Otherwise a dashboard shift can be misdiagnosed as product regression. Operating another measurement check adds cost, but an uncalibrated monitor can trigger both false alarms and unsafe silence.

Resolve Conflicting Signals

10. How would you investigate an offline win and an online loss?

Confirm candidate identity and instrumentation first. Compare traffic slices with dataset slices, inspect evaluator differences, and replay losing production traces offline after privacy review. Look for dependency behavior, conversation history, latency, side effects, and intents absent from the static set. Then add adjudicated failures to a new dataset version and test the proposed fix. I would contain the candidate cohort until the loss is understood; averaging online and offline numbers would erase the disagreement rather than explain it.

11. What if offline quality declines slightly while an online product outcome improves?

Inspect which offline slices declined and whether they represent safety or mandatory correctness. A critical veto remains binding even if a broad business outcome rises. If declines are low-risk and the online result is credible, use human review and a controlled exception with monitoring while revising outdated proxies. This is a risk decision, not a rule that production always wins. The candidate should preserve both pieces of evidence and state whose approval accepts the residual risk.

12. How can Simpson's paradox mislead an online comparison?

An aggregate can favor a candidate because its cohort contains easier traffic even when it loses within important slices. I would check allocation and outcome by intent, locale, tool path, customer segment, and time, using predeclared slices where possible. Reweighting or a regression analysis may clarify the comparison, but sparse cells need uncertainty notes. The interviewer is listening for population reasoning, not a promise that randomization automatically fixed every imbalance or post-treatment effect.

Operate Gates, Thresholds, and Rollbacks

13. How would you define an offline release gate without overfitting to one score?

Use critical vetoes for policy and contract failures, minimum results for named risk slices, and a paired comparison for broader quality. Require dataset and evaluator versions plus failed-case review. Any numeric boundary is organization-specific; for example, a document might label illustrative_threshold values that are derived later from baseline variance and risk tolerance. Multiple gates complicate reporting, but they prevent a large easy slice from compensating for a dangerous regression.

{
  "policy": "illustrative_release_policy",
  "critical_vetoes": ["unauthorized_side_effect", "sensitive_data_exposure"],
  "slice_thresholds": "derive_from_validated_baseline_and_risk",
  "paired_quality_rule": "candidate_must_not_regress_beyond_approved_margin",
  "online_plan": "shadow_then_reversible_cohort"
}

14. What online stop condition would you require before exposure begins?

Declare observable conditions tied to harm: policy violations, tool-side-effect anomalies, error or abandonment changes, severe latency, and reviewer-confirmed failure clusters. Define evaluation windows, minimum evidence, owners, and whether the response is automatic rollback, traffic freeze, or investigation. Illustrative thresholds must be labeled as such until calibrated. A stop rule invented after seeing results invites negotiation exactly when the team is under pressure.

15. Why should rollback verification include both online and offline checks?

Traffic routing can report a rollback while caches, prompts, tools, or long-lived conversations still use candidate state. Verify serving metadata on live traces and confirm the harmful online signal returns to expected behavior. Re-run the incident case and core offline suite against the restored configuration. The dual proof covers deployment reality and regression behavior. It costs additional response time, but a control-plane status alone does not establish that users are back on the safe path.

Feed Production Learning Back Into Tests

16. How would you promote a production failure into the offline suite?

Preserve the trace in a restricted store, confirm consent and redaction, group it with related incidents, and establish expected behavior through domain review. Create a minimized reproducible case plus relevant context, then assign provenance, risk slice, and split without leaking it to tuning workflows. Keep the original incident link for audit. One noisy transcript should not become a golden verbatim; the goal is a stable test of the failure mechanism.

17. Why should production-derived cases be deduplicated by failure family?

Repeated popular incidents can overwhelm an offline score and make one fix look like broad improvement. Cluster exact and semantic relatives, review the family boundaries, and retain representative variants for meaningful conditions such as locale or tool state. Report family-level and row-level results separately when volume matters. Deduplication may hide real frequency, so prevalence stays in online metrics while the offline suite focuses on behavioral coverage and reproducibility.

18. How would you detect that the offline suite has stopped predicting online quality?

Track candidate deltas across repeated releases and compare them with later, properly joined online outcomes by stable slice. Investigate changes in sign, rank, or error-family coverage rather than demanding one universal correlation number. Audit dataset age, evaluator drift, and product changes. With few releases, evidence will be weak, so supplement trend analysis with production-failure coverage review. The correct response is to revise the measurement model, not tune thresholds until historical decisions look consistent.

Assess the Senior Strategy

19. How would you review an evaluation plan that reports one combined online-offline score?

Reject the unexplained aggregate. Ask which populations, labels, evaluators, and decision times were combined; a curated reference-based result and a sampled production proxy are not interchangeable units. Require a decision table showing offline gates, online exposure evidence, product outcomes, uncertainties, and conflict resolution. A summary status can exist after those rules are explicit. Senior strategy keeps evidence types visible so stakeholders know what was proven and what remains a monitored bet.

Close the Measurement Loop Deliberately

The strongest candidate does not choose a side in an online-versus-offline debate. They build a controlled progression: governed offline evidence, a reversible release decision, representative and risk-targeted live observation, delayed outcome joining, and an audited path back into the suite.

The decisive action is to stop when the two systems disagree and learn why. Shipping because one dashboard is green, or blocking because one stale benchmark moved, both surrender judgment to a proxy. A senior QA lead preserves the disagreement, finds the population or measurement boundary behind it, and changes the next release plan with that evidence.