Langfuse Tutorial: Traces, Scores, Datasets, and Evals

A support assistant told a customer that a damaged product was outside the return window. The final answer looked fluent and cited an internal policy page. The trace showed the retriever had selected a policy for business accounts, while the session belonged to a consumer account. Without the retrieval span, QA could only report a wrong answer. With it, the team could fix the tenant filter and add a regression example.

Langfuse is most useful when traces become test evidence, not when it is installed only to produce attractive dashboards. This workflow moves from one instrumented request to a reviewed dataset and a defensible release comparison.

Decide what a useful trace must prove

Map the application before instrumenting it. For a retrieval assistant, one user turn may include input normalization, authentication context, retrieval, reranking, prompt assembly, a model generation, tool calls, and response formatting. Each observation should have a stable name and a clear input-output boundary.

Define required metadata without sending secrets:

Do not send raw access tokens, payment data, or unnecessary personal information. Decide redaction before production tracing and test the redaction itself.

Send one controlled trace

Use a dedicated project or environment and synthetic input. Credentials belong in environment variables, with the base URL matching the chosen Langfuse deployment. The current Python SDK supports explicit observations through a context manager:

from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(
    as_type="span",
    name="answer-support-question",
) as request_span:
    request_span.update(
        input={"case_id": "RET-CONSUMER-017", "question": "Can I return a damaged kettle?"},
        metadata={"environment": "qa", "release": "support-2026.07.10-rc2"},
    )

    with langfuse.start_as_current_observation(
        as_type="generation",
        name="compose-answer",
        model="configured-chat-model",
    ) as generation:
        # Call the application's model client here.
        generation.update(output="Contact support so we can assess the damaged item.")

    request_span.update(output={"case_id": "RET-CONSUMER-017", "status": "answered"})

langfuse.flush()

The python block should create a parent span with a nested generation. flush() matters for a short-lived script because trace delivery is buffered. In an application, instrument real boundaries and follow the SDK’s lifecycle guidance instead of flushing every request.

After execution, verify trace count, hierarchy, timestamps, input and output, model metadata, environment, and request correlation. Missing children or empty inputs are instrumentation defects, not harmless dashboard gaps.

Trace retrieval and tools as testable steps

The final response alone cannot reveal whether a correct answer came from the right evidence. Record retrieval query, safe document identifiers, rank or score, filters, index version, and selected excerpts according to privacy policy. For tools, record tool name, validated arguments, outcome, retry count, and sanitized error.

A review table for case RET-CONSUMER-017 might be:

Trace names should remain stable across releases so comparisons are possible. Put high-cardinality request values in attributes or metadata, not in observation names.

Manage prompts as versioned release inputs

Treat a prompt change like code. Record author, reason, variables, expected input schema, review status, and deployment label or environment. Never edit the only production prompt and then try to reconstruct the previous behavior from memory.

For each trace, link or record the exact prompt version used. A label such as production is useful for deployment, but the immutable version is what makes a failure reproducible. Test missing variables, malicious retrieved text, excessive context, empty history, and formatting instructions. Prompt compilation should fail clearly when a required variable is absent rather than sending a malformed request.

Compare versions on the same dataset with the same application path. Changing prompt, model, retrieval index, and temperature together may improve a score, but it destroys causal evidence.

Curate a dataset from real failure modes

Build dataset items around decisions, not a pile of easy questions. Sources can include reviewed production failures, support escalations, policy boundaries, adversarial cases, and deliberately constructed edge cases. Remove or mask personal data before promotion.

Each item should contain input, expected facts or outcome, tags, risk, and provenance. For example:

{
  "case_id": "RET-CONSUMER-017",
  "input": {
    "segment": "consumer",
    "question": "Can I return a damaged kettle after opening the box?"
  },
  "expected": {
    "must_use_policy": "consumer-damaged-item",
    "must_not_claim": ["automatic refusal", "business account terms"],
    "escalate_if": "damage details are insufficient"
  },
  "tags": ["returns", "retrieval-filter", "high-impact"]
}

The json block is a test design example, not a claim about an import schema. Reviewers should observe that the oracle allows wording variation while fixing the required policy decision. Keep a separate holdout set for final comparison so prompt tuning does not overfit every known case.

Add scores with documented rubrics

Scores need names, value ranges, direction, owners, and rubrics. A value of 0.8 is meaningless without knowing whether it measures groundedness, tone, or task success. Separate hard failures from gradual quality dimensions.

For the support assistant, useful scores include policy_selection as boolean, unsupported_claims as a count where zero is best, resolution_quality on a human rubric, and latency_ms as an observed operational value. Keep human labels for a calibration subset. Compare automated evaluator disagreements against those labels before trusting a gate.

When an evaluator uses another model, store its version, rubric, input, and reasoning evidence permitted by policy. Do not let the same vague judge both create and validate the expected answer. Test evaluator sensitivity with clearly good, borderline, and clearly bad responses.

Run an experiment that supports comparison

Select a baseline prompt version and one candidate. Hold model configuration, retrieval index, tool implementations, and dataset version constant unless one of those is the intended variable. Run every item and inspect both aggregate scores and case-level regressions.

Use a decision table rather than one average:

A candidate that gains on routine questions but fails one damaged-item policy case should not pass through an average. Record exceptions explicitly with owner and rationale.

Debug failures from the inside out

Start at the earliest observation that diverges from expectation. If identity metadata is wrong, later prompt work is irrelevant. If retrieval is correct but generation invents a rule, inspect prompt instructions and context ordering. If generation is correct but the delivered response is wrong, inspect post-processing, caching, and session association.

Compare a failing trace with a passing trace using the same dataset item. Check release, prompt version, index, filters, model settings, tool arguments, retries, latency timeline, and scores. Beware of sampling gaps: absence of a trace does not prove the request never occurred. Correlate with application logs and ingestion health.

Operate Langfuse as part of QA evidence

Monitor whether expected trace volume, required fields, and observation hierarchy remain present after deployment. Restrict project access, set retention according to data policy, rotate credentials, and verify redaction with sentinel secrets that must never appear. A tracing outage should alert separately from an application-quality regression.

Sampling needs its own test. If only part of traffic is traced, verify that the sampling rule does not systematically remove failures, slow requests, a customer segment, or one worker type. Keep request-level correlation outside Langfuse so an untraced incident can still be diagnosed. For multi-turn products, confirm that session identifiers group the intended conversation without joining different users or splitting one conversation after a retry.

For release review, retain the dataset version, experiment identifiers, compared prompt versions, model and retrieval configuration, rubric definitions, aggregate and per-case results, failed traces, and accepted exceptions. Langfuse does not decide whether an assistant is safe to ship. It makes the chain of evidence inspectable enough for the team to decide.