Audit Langflow Requirement Agent Source Provenance

Langflow requirement agent provenance testing compares every generated criterion and open question with an immutable source line or source ID. The audit rejects contradicted or invented claims, flags omitted ambiguities, verifies citation targets, and retains the flow output plus trace evidence needed to locate where source meaning changed.

A valid JSON record is not proof of a valid requirement. Shape validation can confirm that criteria is a list and that each item has a source field, yet a cited line may say the opposite of the generated claim. Provenance testing joins each claim back to frozen evidence and classifies that relationship explicitly.

The Langflow API guide documents programmatic flow execution, while the official Langflow traces guide documents flow and component evidence. Together they provide execution and diagnostic surfaces. The semantic oracle still comes from the approved requirement and a reviewed set of expected facts and open decisions.

What is Langflow requirement agent provenance testing?

Provenance testing asks whether the generated record can show where each statement came from and whether that source actually supports it. For a requirement agent, outputs often include acceptance criteria, assumptions, risks, test ideas, and questions. Each category needs a clear epistemic status. Confirmed behavior must be grounded; unknown behavior must remain a question; unsupported guesses must fail.

The repository scenario w2-requirement-agent-audit in AI_FOR_QA_WAVE2_CHALLENGES, at src/db/seed-data/challenges/ai-for-qa-wave2.ts, supplies a concise example. Source R-17 says remembered-session duration is undecided, revocation is supported, and maximum device count is unspecified. The sample agent invents a 30-day duration and a five-device limit while correctly grounding revocation. It also returns no open questions.

That fixture reveals several independent errors. The duration claim is contradicted by its cited line. The device-limit claim is unsupported by a line that says the limit is absent. The revocation claim is supported. The empty question list omits known ambiguities. A parser can accept every field while the record remains unfit for test design.

This scope differs from generic Langflow component testing, which checks component boundaries and flow inputs. It also differs from Langflow component exception testing, which verifies error and status behavior. Provenance testing evaluates successful-looking semantic output against source evidence.

Use deterministic claim-level checks before model grading. Source IDs, citation existence, duplicate claims, missing fields, and expected ambiguity coverage can be exact. Entailment may need a reviewed rule, human adjudication, or evaluator, but its input and output should remain inspectable.

Design the audit so a reviewer can reproduce it outside Langflow. Store the frozen source, structured candidate output, and verdict record as ordinary test artifacts. Langflow is the execution environment, but provenance is a data relationship. This separation allows a parser or evaluator defect to be replayed without rerunning a model and helps compare flow revisions against the same evidence.

Establish ownership for each failure class. Source resolution failures belong to ingestion or fixture management. Unsupported claims may belong to generation or prompting. Missing expected items may originate in retrieval, context limits, or output instructions. Malformed records belong to schema production or parsing. The trace helps localize the first divergence, but the claim verdict remains stable while the root cause is investigated.

What source contract should each generated record preserve?

Freeze a source version before running the agent. Assign stable document and line or span IDs, store the text used by the flow, and record a content digest or version identifier. A citation to R-17 line 3 is meaningful only if every test resolves it against the same R-17 revision.

Define separate output collections for confirmed criteria and open questions. A confirmed criterion needs a claim ID, normalized text, one or more source references, and optionally a confidence or review status if the product uses one. An open question needs the unresolved decision, its source reference, and the owner or workflow state when known. Do not put guessed answers in the confirmed collection.

The contract should distinguish at least these states:

Do not let a confidence score replace classification. A highly confident unsupported claim is still unsupported. Likewise, a low-confidence grounded fact does not become an open product decision. The contract should preserve source relationship separately from model confidence.

Evaluating LLM outputs for faithfulness and groundedness provides the broader evaluation context. Requirement provenance adds a product-specific rule: unknown source decisions must survive as unknowns rather than being completed from common practice.

How do you build claim-to-source audit fixtures?

Start with small source excerpts whose expected relationships can be reviewed line by line. Include direct support, explicit negation, explicit uncertainty, missing values, paraphrase, multi-line support, and distractor lines with related vocabulary. Keep the fixture source separate from the generated candidate so the audit cannot treat the candidate as evidence for itself.

Represent the source and expected coverage with typed data. The first code block models R-17, resolves citations, and classifies exact fixture claims. A production classifier can be more sophisticated, but deterministic rules remain useful for explicit unknowns and prohibited inventions.

from dataclasses import dataclass
from enum import StrEnum


class Verdict(StrEnum):
    SUPPORTED = "supported"
    CONTRADICTED = "contradicted"
    UNSUPPORTED = "unsupported"
    BAD_CITATION = "bad_citation"


@dataclass(frozen=True)
class SourceLine:
    ref: str
    text: str


@dataclass(frozen=True)
class Criterion:
    text: str
    source: str


SOURCE = {
    "R-17:1": SourceLine("R-17:1", 'A signed-in user can select "Remember this device".'),
    "R-17:2": SourceLine("R-17:2", "The remembered-session duration is undecided."),
    "R-17:3": SourceLine("R-17:3", "A user can revoke remembered devices in Security settings."),
    "R-17:4": SourceLine("R-17:4", "Error copy and maximum device count are unspecified."),
}


def audit_known_fixture(criterion: Criterion) -> Verdict:
    line = SOURCE.get(criterion.source)
    if line is None:
        return Verdict.BAD_CITATION

    claim = criterion.text.lower()
    if "revoke" in claim and criterion.source == "R-17:3":
        return Verdict.SUPPORTED
    if "30 days" in claim and criterion.source == "R-17:2":
        return Verdict.CONTRADICTED
    if "five devices" in claim and criterion.source == "R-17:4":
        return Verdict.UNSUPPORTED
    return Verdict.UNSUPPORTED

The function is intentionally narrow. It provides an exact oracle for this regression fixture rather than pretending keyword rules solve general entailment. For broader datasets, pair reviewed labels with a deterministic citation resolver and an evaluator whose reasons are stored for adjudication.

Give every fixture a case ID, source version, expected claim verdicts, expected open questions, and prohibited inventions. The same source can produce several candidate records so the audit tests missing, duplicate, and conflicting output independently.

How do you detect contradicted and invented criteria?

Resolve the citation before judging the claim. A missing document or line is a bad citation, not an unsupported claim. Once resolved, compare the complete claim with enough surrounding source context to preserve qualifiers, negation, scope, actor, condition, and units. A fragment can appear supportive while the full sentence contradicts it.

Contradiction fixtures should include explicit unknowns. Statements such as "not decided," "not specified," and "pending owner approval" cannot support a concrete value. Encode these markers as deterministic checks where possible. They create strong release gates because the source itself says an answer is unavailable.

Invented criteria may cite a related line. The five-device claim points to a line about device count, but that line says no maximum is specified. Topic overlap is not entailment. Require the cited text to support the full predicate, not merely share nouns.

Also test scope changes. "Signed-in users can revoke remembered devices" does not support "administrators can revoke any user's devices." Actor, object, condition, and permission changed even though the action verb stayed the same. Build minimal pairs that vary one field so reviewers can identify the unsupported expansion.

Claim-level citation entailment testing covers citation support for grounded answers. Requirement records add omissions and open decisions, so claim-level verdicts are necessary but not sufficient. A perfect set of supported claims can still ignore an unresolved product question.

Do not use fluent wording as evidence. Normalize whitespace and harmless formatting for comparison, but preserve negation, numeric values, modal verbs, and conditions. Those tokens often carry the requirement's actual meaning.

How do you test omitted ambiguities and missing fields?

Claim audits inspect what the agent said. Coverage audits inspect what it failed to say. Label source facts that must become confirmed criteria and source uncertainties that must become open questions. After matching output items to those labels, report any required source item with no acceptable match.

For R-17, expected open questions include remembered-session duration, error copy, and maximum device count. A generated openQuestions: [] is therefore incomplete even if every returned criterion is supported. This is a recall-style requirement, but avoid publishing an unsupported percentage. Report missing item IDs and categories directly.

Validate the structured schema before semantic matching. Missing confirmedCriteria, openQuestions, or citation fields should produce a schema failure rather than a misleading provenance score. Empty collections may be valid for some sources, so their acceptance depends on labeled expectations for that case.

Test duplicate and conflicting output. Two paraphrases of the same supported fact can inflate apparent coverage. One supported criterion and one contradictory criterion about the same source decision must not average into a pass. Group claims by normalized subject and decision so conflicts remain visible.

The distinction also matters in requirements interviews and reviews. QA analyst scenarios about ambiguity and risk explains why unresolved decisions should be surfaced. The agent should assist that work, not silently choose a common default.

Missing-field tests should cover null, wrong type, omitted key, empty text, unknown source ID, duplicated claim ID, and a citation without a source version. Keep parser failures separate from semantic failures so the owner knows whether to fix output formatting or requirement reasoning.

Test ordering independence unless the output contract promises a sequence. A model may list revocation before device selection while preserving both facts. Match by claim identity or reviewed semantic label, not array position. Preserve order only when it carries product meaning, such as an explicit workflow or priority supplied by the source.

Coverage labels should be authored independently from the candidate flow. If the same prompt creates both expected facts and actual output, shared omissions can pass unnoticed. Use reviewed requirement analysis as the oracle, version it with the source, and record who approved changes to required facts or open questions.

How do you verify citation IDs and source spans?

Citation verification has three levels. First, syntax: does the reference follow the accepted document and location format? Second, resolution: does that document version and span exist? Third, support: does the resolved text entail the claim? Passing one level does not imply the next.

Store source references as structured fields when possible instead of one free-form string. A document ID, version, start line, end line, and optional quote can be validated independently. If the agent returns a quote, compare it with the source after a narrowly defined whitespace normalization. Do not accept a quote that cannot be found merely because its meaning seems similar.

Multi-line support needs an allowed span policy. A criterion can rely on a precondition in one line and an outcome in the next. Preserve both lines in the evidence presented to the evaluator. Restrict spans enough that a citation cannot point to an entire document and claim support from unrelated text somewhere inside it.

Test stale references by changing the source version in a fixture while keeping line IDs. The audit should detect a version mismatch or re-resolve against the intended snapshot. Line numbers alone are unstable when documents are edited, so a stable requirement ID plus version is safer than an unversioned location.

Citation correctness belongs in the dataset as an explicit label. Building LLM evaluation datasets describes versioned cases and review. For requirement agents, store both positive and negative citation examples, including plausible but unrelated lines.

Which Langflow outputs and trace spans should you retain?

Retain enough evidence to reproduce the semantic audit without storing unrelated secrets or personal data. At minimum, preserve the case ID, source version, flow version or exported definition ID, run and session IDs, structured output, claim verdicts, citation resolutions, missing expected items, and sanitized trace references.

Langflow traces can include flow-level status and component spans with inputs, outputs, latency, errors, model metadata, and token usage where available. For provenance diagnosis, find the earliest component where the relevant source line disappears or changes. The defect may be source loading, retrieval, prompt construction, model output, parsing, or post-processing.

The second code block gives each flow call a unique session ID, verifies that Langflow returns that ID, and keeps only trace records whose documented sessionId matches. Production tests should use environment-managed credentials and redact captured evidence.

import os
from uuid import uuid4

import requests


BASE_URL = os.environ["LANGFLOW_SERVER_URL"]
API_KEY = os.environ["LANGFLOW_API_KEY"]
FLOW_ID = os.environ["LANGFLOW_REQUIREMENT_FLOW_ID"]
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}


def run_requirement_agent(source_text: str) -> tuple[str, dict]:
    session_id = f"provenance-{uuid4()}"
    response = requests.post(
        f"{BASE_URL}/api/v1/run/{FLOW_ID}",
        headers=HEADERS,
        json={
            "input_value": source_text,
            "input_type": "chat",
            "output_type": "chat",
            "session_id": session_id,
        },
        timeout=60,
    )
    response.raise_for_status()
    body = response.json()
    if body.get("session_id") != session_id:
        raise RuntimeError("Langflow did not return the requested session_id")
    return session_id, body


def list_correlated_traces(session_id: str) -> list[dict]:
    response = requests.get(
        f"{BASE_URL}/api/v1/monitor/traces",
        headers=HEADERS,
        params={"flow_id": FLOW_ID, "page": 1, "size": 50},
        timeout=30,
    )
    response.raise_for_status()
    traces = response.json().get("traces", [])
    return [
        trace
        for trace in traces
        if trace.get("sessionId") == session_id
    ]

The official Langflow monitor endpoint guide currently documents GET /api/v1/monitor/traces as a flow-level query. It does not document a session_id query parameter for that endpoint. Its response includes sessionId on each trace, so correlation happens after retrieval in this example. A production harness may need bounded polling and pagination because a newly completed trace might not appear immediately or on the first page.

An empty correlated list means trace evidence is unavailable, not that the run had no relevant spans. Do not attach an unmatched flow trace to the case, and do not grade the candidate output using evidence from another session. If the installed Langflow release omits sessionId, returns duplicate matches, or changes the response shape, retain the run response and mark trace correlation unresolved until a deployment-specific identifier provides an unambiguous join.

Tracing Langflow component spans goes deeper into locating a failing component. This audit uses those spans as supporting evidence while keeping the source-to-claim verdict as the release decision.

How do you score provenance without hiding critical errors?

Report counts and case IDs by error class rather than collapsing everything into one average. Useful categories include unsupported claim, contradiction, bad citation, uncited confirmed claim, missing source fact, missed ambiguity, duplicate claim, conflicting claim, and malformed record. The release policy can declare which categories block a candidate flow.

Severity depends on the requirement. An invented security permission or retention period can be release-blocking even if many minor facts are supported. Do not let supported claims compensate for a critical invention. Slice results by requirement area, source type, ambiguity type, and risk label where those dimensions are part of the reviewed dataset.

If an automated evaluator judges entailment, retain its verdict and reason, evaluator version, prompt or rubric version, and human adjudication for disputed cases. Calibrate it against labeled examples before using it as a gate. Deterministic checks for missing IDs and explicit unknown markers should remain outside model grading.

Use three release states when evidence can be incomplete: pass, fail, and needs review. A missing trace should not turn a contradicted claim into a pass, but it may prevent confident root-cause assignment. Likewise, an evaluator disagreement can route a case to review without erasing the deterministic citation failure.

Testing AI agents discusses broader tool, memory, retrieval, and behavior layers. Provenance is one named quality dimension. Keep it visible alongside task success instead of treating a polished requirement document as sufficient evidence.

How do you run the requirement-agent audit workflow?

Use this workflow for each labeled requirement case:

1. Freeze the source text, document ID, version, and stable line or span references.
2. Label required confirmed facts, explicit ambiguities, prohibited inventions, and accepted paraphrases.
3. Run the Langflow requirement agent with a unique case and session identity.
4. Validate the output schema before semantic evaluation.
5. Resolve every citation against the frozen source version.
6. Classify each confirmed claim as supported, contradicted, unsupported, uncited, or badly cited.
7. Match expected facts and ambiguities to output items and report omissions.
8. Retrieve sanitized flow and component trace evidence for failed cases.
9. Record verdicts by category and route uncertain entailment to adjudication.
10. Gate the candidate flow on named critical failures and retain the regression case.

Run deterministic checks on every change to parsing, prompts, source handling, or output schemas. Run broader evaluator suites according to model and flow risk. Do not update expected labels merely because a new candidate disagrees; require source review and an explicit dataset change.

The source fixture should remain the authority. If product owners resolve an ambiguity, create a new source version, update the expected open question into a confirmed fact, and keep the old case where version compatibility matters. This makes requirement evolution visible rather than retroactively rewriting evidence.

Use a thin API contract check alongside the semantic suite. Testing Langflow flow APIs with Java and REST Assured covers the wider HTTP approach; this workflow needs only enough API proof to confirm the intended flow version, authenticated execution, response envelope, and trace correlation. Keep the source-to-claim matrix independent from transport assertions.

During triage, compare a passing and failing flow revision against the same frozen case. Locate the first changed component output, then rerun the offline audit against both final records. This shows whether the change affected source presence, citation assignment, claim wording, open-question coverage, or only trace metadata. It also prevents prompt edits from being approved solely because one polished example looks better.

FAQ

What is provenance in a requirement agent?

It is the traceable, verifiable relationship between generated content and its source. Each confirmed criterion or open question should identify the requirement version and span that supports it, allowing another reviewer to reproduce the classification without trusting the agent's wording.

Is a valid citation ID enough to prove support?

No. Resolve the ID and compare the complete claim with the cited text. A valid line can contradict the claim, discuss only the same topic, omit a changed actor or condition, or expose an unknown instead of a concrete value.

How should undecided requirements be represented?

Place them in a separate open-question collection with source references. Preserve what is unknown and ask the responsible owner to decide it. Do not fill duration, limits, permissions, copy, or other missing product choices from convention.

How do you test omitted source facts?

Label expected facts and ambiguities before execution, then match output items back to those labels. Report every required item without a supported criterion or sourced question. Auditing only returned claims cannot detect silence.

Which trace data helps diagnose a bad criterion?

Use run and session IDs, flow version, component names, sanitized inputs and outputs, status, errors, and final structured output. Compare spans to find where the relevant source line was first dropped, retrieved incorrectly, rewritten, or parsed into the wrong field.

Should unsupported claims and missed ambiguities share one score?

No. Keep unsupported claims, contradictions, citation errors, missing facts, and missed ambiguities as separate reported categories. A release gate can combine explicit rules, but one favorable category should never conceal a critical invention or unresolved decision that the agent omitted.

Conclusion: Keep every requirement claim attached to evidence

A requirement agent is useful only when its output remains reviewable against the product source. Structured fields and fluent criteria are starting points, not proof. Citation resolution, claim classification, omission checks, and trace evidence turn a generated record into an auditable artifact.

Begin with a small labeled source such as R-17, preserve confirmed facts and unknowns separately, and block contradicted or invented criteria. As the dataset grows, keep error categories and source versions visible so automation assists product clarification without manufacturing product intent.