The code assistant found the right symbol in the wrong file

A code assistant returns a clean example for createSession, complete with a plausible import. The snippet came from a migration shim that the repository stopped calling six months ago, while the current implementation lives two directories away. The answer looks grounded because retrieval found matching code, but it found the wrong code.

Code retrieval fails differently from prose retrieval. Identifiers are exact, names are overloaded, repository paths carry meaning, and a one-line signature may depend on types defined elsewhere. Testing only whether a chunk “looks relevant” misses the failures that waste an engineer’s time.

Understand what the retriever ranked, not what the answer implied

A retrieval pipeline usually turns the query into one or more search signals, finds candidate chunks, ranks them, and sends only the first few to the answer stage. The generator cannot repair evidence that never enters its context. That makes retrieval quality an upstream contract with its own fixtures and artifacts.

Embedding similarity captures relatedness, not code authority. Official embedding documentation describes embeddings as vectors whose distance reflects relatedness. A close vector can represent a tutorial, test, deprecated adapter, generated client, or call site rather than the definition the user needs. Exact names, repository metadata, and code structure supply different signals.

Ask four questions about every result:

1. Did it come from the right repository revision or supported version?
2. Does it contain the required symbol or behavior, not merely related vocabulary?
3. Is the chunk useful on its own, or was the signature separated from its contract and types?
4. Did useful evidence arrive before the context cutoff?

The last question is why an unranked “found somewhere” metric is weak. If the answer stage receives five chunks and the required definition ranks ninth, retrieval failed for that configuration. Reporting recall at 20 would hide the production failure.

Authority is case-specific. For “How do I construct a payment client?” the canonical implementation and its public interface may be acceptable. A test helper that happens to construct one is usually not. For “Show me how tests fake the payment client,” the helper becomes relevant. Relevance labels must follow the user task, not a permanent judgment that one file is always better.

Code chunks also have roles. Label a result as a definition, declaration, call site, test, configuration, generated file, documentation example, or migration artifact when that distinction affects the task. A chunk containing ten uses of createSession can win lexical ranking while omitting the function’s signature and error behavior. A basic symbol-presence check would call it relevant; an engineer would still be stuck.

Do not infer retrieval success from a good final answer. A model may know a common API from training and answer correctly despite empty retrieval. It may also produce a wrong answer despite receiving the exact definition. Evaluate the ranked evidence first, then evaluate whether the answer uses that evidence.

Build gold cases around repository decisions

A useful fixture begins with a question an engineer might ask and records the smallest acceptable evidence set. Pin it to a repository revision. Without a revision, a refactor can turn yesterday’s gold file into today’s stale label and make the retriever look broken for finding the new implementation.

Avoid demanding one exact chunk ID. Chunk boundaries change when the parser or chunk size changes. Express acceptance in stable terms where possible: file path patterns, symbol identities, source roles, and repository revision. Keep exact chunk IDs in run artifacts for diagnosis, not as the sole long-lived oracle.

The TypeScript below models a query, acceptable evidence alternatives, forbidden paths, and ranked results. It calculates hit at the production cutoff and reciprocal rank of the first acceptable result. The evaluator is deterministic and has no dependency on a retrieval vendor.

type EvidenceTarget = {
  file: string;
  symbol?: string;
  roles: Array<"definition" | "declaration" | "call_site" | "test" | "docs">;
};

type RetrievalCase = {
  id: string;
  query: string;
  repositoryRevision: string;
  accepted: EvidenceTarget[];
  forbiddenPathPrefixes: string[];
  cutoff: number;
};

type RetrievedChunk = {
  chunkId: string;
  file: string;
  symbol?: string;
  role: EvidenceTarget["roles"][number];
  repositoryRevision: string;
  score: number;
};

type RetrievalResult = {
  caseId: string;
  passed: boolean;
  firstRelevantRank: number | null;
  reciprocalRank: number;
  forbiddenChunkIds: string[];
  revisionMismatches: string[];
};

function targetMatches(chunk: RetrievedChunk, target: EvidenceTarget): boolean {
  return (
    chunk.file === target.file &&
    (target.symbol === undefined || chunk.symbol === target.symbol) &&
    target.roles.includes(chunk.role)
  );
}

export function evaluateRetrieval(
  testCase: RetrievalCase,
  ranked: RetrievedChunk[],
): RetrievalResult {
  const visible = ranked.slice(0, testCase.cutoff);
  const firstIndex = visible.findIndex((chunk) =>
    testCase.accepted.some((target) => targetMatches(chunk, target)),
  );
  const forbiddenChunkIds = visible
    .filter((chunk) =>
      testCase.forbiddenPathPrefixes.some((prefix) => chunk.file.startsWith(prefix)),
    )
    .map((chunk) => chunk.chunkId);
  const revisionMismatches = visible
    .filter((chunk) => chunk.repositoryRevision !== testCase.repositoryRevision)
    .map((chunk) => chunk.chunkId);
  const firstRelevantRank = firstIndex === -1 ? null : firstIndex + 1;

  return {
    caseId: testCase.id,
    passed:
      firstRelevantRank !== null &&
      forbiddenChunkIds.length === 0 &&
      revisionMismatches.length === 0,
    firstRelevantRank,
    reciprocalRank: firstRelevantRank === null ? 0 : 1 / firstRelevantRank,
    forbiddenChunkIds,
    revisionMismatches,
  };
}

An accepted list may contain alternatives. If an interface declaration and a concrete implementation both answer the question, list both. If the answer requires a signature plus a configuration example, model two required evidence groups rather than one broad target. That supports coverage checks without forcing one chunk layout.

Gold cases should include negative constraints when the repository contains tempting traps. Mark generated clients, archived versions, vendored code, build output, fixtures, or deprecated directories as forbidden for queries where they are not authoritative. This catches a retriever that improves apparent identifier recall by flooding the context with duplicates.

Build the dataset from real engineering work: search logs, failed assistant traces, code review comments, onboarding questions, and incidents caused by stale examples. Remove secrets and user data before storage. Keep a short label rationale explaining why each target is acceptable. Future maintainers need that reasoning after paths change.

Include query variety without manufacturing keyword permutations. One case may use an exact identifier, another a stack-trace phrase, and another a behavior description such as “where do we reject expired refresh tokens?” Those queries exercise lexical, structural, and semantic signals differently.

Worked example: a deprecated implementation outranks the current one

The query is “How does createSession handle an expired refresh token?” The top result comes from src/legacy/session.ts, which contains the exact name and a long explanatory comment. The current code in src/auth/session-service.ts uses the same public symbol but delegates expiration checks to validateRefreshToken.

The fixture pins the current repository revision, accepts the service definition plus validator, and forbids src/legacy/. The old chunk is not harmless extra context. If it reaches the generator, the answer may recommend a control flow the product no longer executes. The failure record should show both firstRelevantRank and the forbidden chunk ID.

The fix might be an index filter that excludes legacy paths, a metadata preference for active source roots, or a deletion from the searchable corpus. Each has a cost. Excluding a directory can make historical migration questions impossible. A soft rank penalty preserves access but may still leak stale code into a large context window. Choose based on supported user tasks.

Worked example: an overloaded name resolves to the wrong module

The query asks where parse validates webhook signatures. A repository contains parse functions for dates, command-line arguments, JSON fixtures, and webhooks. Pure lexical search returns many exact matches. Semantic search may favor a well-documented JSON parser because its surrounding prose mentions validation.

Add expected module and role to the target. If the user’s current file or stack trace identifies the webhook package, pass that path context into retrieval and store it in the fixture. A module filter is stronger than asking the model to choose among six unrelated parse functions after retrieval.

Do not hard-code the answer path into every production query. Context filters should come from reliable user state, such as the open file, repository, language, or package. Guessing a module from ambiguous text can suppress the correct result. The evaluator should include cases with and without reliable path context.

Worked example: usages crowd out the definition

A query for RetryPolicy retrieves five test files where the type is instantiated. The actual definition ranks sixth, just beyond a five-chunk cutoff. Identifier hit rate looks perfect because every result contains the name. Definition recall at five is zero.

This is where source role matters. A structural index or parser can mark definitions separately from references. A reranker can prefer definitions for “what is” and “how is configured” questions, while still favoring usages for “show me examples” queries. Increasing the cutoff would also recover the definition, but it adds context, latency, and noise to every answer.

Measure the rank that the answer stage actually receives

Calculate metrics at the configured cutoff, not at a convenient larger value. Useful deterministic metrics include hit rate at k, recall of required evidence groups at k, first relevant rank, reciprocal rank, forbidden-source rate, and revision-mismatch rate. Report them by query class and repository area.

Precision needs careful labels. A call site may be useful secondary context without being sufficient primary evidence. Binary relevant or irrelevant labels can flatten that nuance. If precision matters, define graded roles or separate “required evidence” from “supporting evidence.” Do not let a reviewer’s personal preference for a file decide the metric.

The next runnable function handles cases that require several evidence groups. Each group is satisfied when any target in that group appears. This supports queries that need both a public contract and a concrete configuration example.

type Target = { file: string; symbol?: string };
type EvidenceGroup = { id: string; anyOf: Target[] };
type Chunk = { chunkId: string; file: string; symbols: string[] };

function matches(chunk: Chunk, target: Target): boolean {
  return (
    chunk.file === target.file &&
    (target.symbol === undefined || chunk.symbols.includes(target.symbol))
  );
}

export function groupRecallAtK(
  groups: EvidenceGroup[],
  ranked: Chunk[],
  k: number,
): { recall: number; missingGroupIds: string[] } {
  if (groups.length === 0) throw new Error("At least one evidence group is required");
  if (!Number.isInteger(k) || k < 1) throw new Error("k must be a positive integer");

  const visible = ranked.slice(0, k);
  const missingGroupIds = groups
    .filter(
      (group) =>
        !group.anyOf.some((target) =>
          visible.some((chunk) => matches(chunk, target)),
        ),
    )
    .map((group) => group.id);

  return {
    recall: (groups.length - missingGroupIds.length) / groups.length,
    missingGroupIds,
  };
}

const outcome = groupRecallAtK(
  [
    { id: "public_contract", anyOf: [{ file: "src/retry.ts", symbol: "RetryPolicy" }] },
    { id: "configuration", anyOf: [{ file: "config/retry.ts" }] },
  ],
  [
    { chunkId: "c-test", file: "test/retry.spec.ts", symbols: ["RetryPolicy"] },
    { chunkId: "c-def", file: "src/retry.ts", symbols: ["RetryPolicy"] },
  ],
  2,
);

console.log(outcome);

The illustrative call reports recall: 0.5 and missingGroupIds: ["configuration"]. It does not claim the retrieved test is useless. It says the evidence package lacks one requirement for this query.

Aggregate only after preserving case results. A mean reciprocal rank can improve while critical repository areas regress. Slice by exact-identifier queries, descriptive queries, stack traces, language, repository size, and source role when those dimensions reflect real use. Always show the number of cases behind a slice.

Repeated runs are useful when the retrieval pipeline itself includes variable query rewriting or model-based reranking. Store every attempt rather than averaging before persistence. For a deterministic index and fixed query, repeated identical runs usually add cost without information. If results change anyway, investigate index updates, tie-breaking, or nondeterministic components.

Diagnose ranking, indexing, and label failures separately

Begin with the recorded query after any rewrite. A weak rewrite can delete an exact identifier, change a file path, or generalize a stack-trace phrase. Compare the raw user query, rewritten query, filters, and candidates. Do not blame embeddings for a term removed before search.

Then verify index identity. The result artifact should include repository revision, index build ID, chunker version, and searchable path policy. If the case expects revision abc123 but the index was built from another commit, stop. That run cannot decide whether ranking for abc123 is good.

Inspect ranked chunks before their scores. Scores are not interchangeable across retrievers, rankers, or configurations. A higher score only has meaning within the documented scoring system that produced it. File, symbol, role, revision, and excerpt usually reveal more than an unexplained decimal.

The command below finds failed rows and prints the top five result identities. It expects a JSON array of evaluation records and uses only standard jq operations.

jq -r '
  .[]
  | select(.passed == false)
  | "CASE \(.caseId) query=\(.query)",
    ("  expected_revision=" + .repositoryRevision),
    (.ranked[:5][]
      | "  rank=\(.rank) file=\(.file) symbol=\(.symbol // "-") role=\(.role) revision=\(.repositoryRevision)")
' artifacts/code-retrieval-results.json

Illustrative output for the deprecated-file case would be:

CASE session-expiry query=How does createSession handle an expired refresh token?
  expected_revision=abc123
  rank=1 file=src/legacy/session.ts symbol=createSession role=definition revision=abc123
  rank=2 file=test/session.spec.ts symbol=createSession role=call_site revision=abc123
  rank=3 file=src/auth/session-service.ts symbol=createSession role=definition revision=abc123

The current definition is present at rank three, so ingestion worked. The issue is ranking and forbidden-source handling. If the definition is absent from every candidate while its file exists in the corpus, inspect parsing and chunk creation. If its file is missing from the corpus, inspect path filters and index build logs. If it appears in candidates but disappears after reranking, isolate the reranker.

Stale gold labels look similar to retrieval failures. Open the pinned revision and verify that the expected target exists and still represents supported behavior. If the dataset accidentally points at the current default branch instead of the pinned revision, a recent refactor can invalidate the investigation. Update labels through review and retain the prior version.

Chunk-boundary failures have distinctive evidence. The top chunk may contain a function body without its signature, or a class name without the method that enforces the behavior. Search the neighboring chunks from the same file. If together they form the required evidence, change structural chunking, add controlled overlap, or retrieve adjacent chunks after a hit. More overlap improves continuity but duplicates tokens and can crowd out diverse evidence.

An ingestion delay is not a ranking regression. Immediately after a source update, an asynchronous index may still serve the previous revision. Compare index build completion with the query time. Either wait for the intended index before evaluation or mark the run invalid. In production, separately monitor freshness because users still experience stale answers even when the quality suite correctly refuses to score them.

Separate missing code from missing symbol metadata

Two failed rows can both report firstRelevantRank: null even when only one retriever failed to surface the code. In the first, the required definition never entered the candidate set because the file was excluded, the query lost its identifier, or ranking left it below the inspected depth. In the second, the returned chunk contains the definition text, but its metadata says role=call_site, names the wrong symbol, or has no symbol at all. The evaluator rejects it because the reviewed target requires the definition role. Tuning similarity for the second failure wastes time because the useful text is already present.

Start with the exact ranked chunk identities and excerpts before changing a score. A healthy result at the production cutoff has the expected repository revision, an accepted file, the required symbol and role, and a rank no larger than the cutoff. A true retrieval miss has no acceptable text in the candidate set at the inspected candidate depth. A metadata failure has recognizable definition text in a returned chunk while the file, symbol, or role fields prevent targetMatches from accepting it. A misleading result can also show the right file and symbol while the chunk body contains only a reference, which means the metadata looks healthy but the parser attached it to the wrong span.

To prove a metadata failure, compare the candidate excerpt with the parser’s recorded source span and symbol classification at the pinned revision. The decisive evidence is that the returned bytes cover the reviewed definition while the attached symbol or role points to a neighboring reference. A true retrieval miss has no such excerpt in the returned candidates. Reproduce the classification against a small parser fixture before rebuilding the whole index. If that fixture emits the correct role while the stored chunk does not, the index was built with different parser inputs or retained stale metadata. Similarity tuning cannot repair either mismatch.

Several diagnostic fields are easy to overread. firstRelevantRank is null when no chunk satisfies the complete acceptance rule, not necessarily when the repository text is absent. reciprocalRank is zero for the same reason and adds no new cause. forbiddenChunkIds identifies prohibited context inside the cutoff, even if an acceptable definition also appears. revisionMismatches invalidates source freshness, but an empty list is meaningful only if every returned chunk carries a trustworthy revision. The raw similarity score is the most misleading field across configuration changes because its scale and calibration may differ. File, symbol, role, revision, cutoff, and rank provide the actionable comparison.

The ownership handoff should stop at the first broken boundary. Repository or indexing owners handle a source file omitted by path policy. Parser and chunker owners handle a definition whose span or role is wrong. Retrieval owners handle an indexed correct chunk missing from candidates. Reranker owners handle a correct candidate demoted below the cutoff. Dataset owners handle a gold target that is absent or no longer authoritative at the pinned revision. The answer-generation team receives the issue only when acceptable evidence reached its actual context and the answer failed to use it.

Give that owner a reproducible evidence bundle: raw and rewritten queries, reliable context filters, case and repository revisions, index build identity, chunker and evaluator versions, configured cutoff, searchable inventory entry for the target file, ranked candidates before and after reranking, safe excerpts around the expected symbol, accepted target rationale, and final context IDs. A final answer screenshot lacks nearly every fact needed to distinguish these causes.

Role-aware metadata has a concrete maintenance cost. Every supported language needs a parser or another dependable way to identify definitions and references. Parser upgrades can change chunk boundaries and roles, which triggers a reindex and can make chunk-level artifacts incomparable with older runs. Keeping pre-rerank candidates and post-rerank results also enlarges each evaluation artifact. Teams that support only one small repository and mostly exact identifiers may spend less time using deterministic symbol lookup than maintaining this machinery.

This evaluation does not catch source that is absent from the searchable corpus by design. Generated code created only during a build, behavior selected by runtime configuration, macro expansion, or a dependency resolved outside the indexed repository can determine the real answer while every indexed chunk looks correct. Add build-aware or runtime-specific evidence tests for those paths. Retrieval quality cannot validate evidence it was never intended to observe.

Fix the signal that failed and pay its actual cost

Exact identifier failures often justify hybrid retrieval. Combine semantic candidates with lexical or symbol-search candidates, then rerank the union. The code below demonstrates a local, transparent scoring rule for already-normalized candidate scores. It is not a claim about any vendor’s ranking API.

type Candidate = {
  chunkId: string;
  semanticScore: number;
  text: string;
  symbols: string[];
  path: string;
};

export function rerank(
  query: string,
  candidates: Candidate[],
  preferredPathPrefix?: string,
): Candidate[] {
  const identifiers = query.match(/[A-Za-z_$][A-Za-z0-9_$]*/g) ?? [];
  const uniqueIdentifiers = [...new Set(identifiers)];

  return [...candidates].sort((left, right) => {
    const score = (candidate: Candidate): number => {
      const exactSymbolHits = uniqueIdentifiers.filter((identifier) =>
        candidate.symbols.includes(identifier),
      ).length;
      const pathMatch =
        preferredPathPrefix && candidate.path.startsWith(preferredPathPrefix) ? 1 : 0;
      return candidate.semanticScore + exactSymbolHits * 0.2 + pathMatch * 0.1;
    };
    return score(right) - score(left) || left.chunkId.localeCompare(right.chunkId);
  });
}

The weights are illustrative configuration values, not measured optimal settings. Tune them on a training portion of reviewed cases, then judge the chosen configuration on held-out cases. If you tune and report on the same small set, the score describes your tuning effort more than future retrieval.

Metadata filters offer a stronger fix when a constraint is known. Repository revision should usually be exact. Language and package can be exact when supplied by reliable context. Source role may be a preference rather than a filter because users sometimes need tests or docs. Every strict filter improves precision at the risk of excluding unexpected but useful evidence.

Structural chunking improves definitions and class context but costs parser maintenance across languages. Plain text chunking is easier and may be sufficient for documentation-heavy repositories. Query-time neighbor expansion repairs split context but consumes more of the context budget. Increasing top k improves recall but adds latency, retrieval cost, and distractors. There is no free retrieval fix; measure the downstream context package after each change.

Roll out the evaluation where it can catch index drift

Start by replaying recorded retrieval outputs through the deterministic evaluator. This validates labels and metrics without paying to rebuild an index. Then run live retrieval against a pinned repository snapshot. Keep recorded and live modes separate so a ranking change is not confused with an evaluator change.

For pull requests that change retrieval code, chunking, metadata, or indexed source policy, run a compact cross-section of cases. A scheduled job can rebuild the full index and run broader repository coverage. Always persist ranked results, not only aggregate metrics.

name: code-retrieval-quality

on:
  pull_request:
    paths:
      - "src/retrieval/**"
      - "evals/code-retrieval/**"
      - "config/indexing/**"
  schedule:
    - cron: "17 2 * * 1-5"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Build pinned code index
        run: npm run index:code -- --manifest artifacts/index-manifest.json
      - name: Run retrieval cases
        run: >-
          npm run eval:code-retrieval --
          --cases evals/code-retrieval/cases.json
          --results artifacts/code-retrieval-results.json
      - name: Enforce retrieval policy
        run: npm run eval:code-retrieval:gate -- artifacts/code-retrieval-results.json
      - name: Upload ranked evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: code-retrieval-evidence
          path: artifacts/

Migrate an answer-only evaluation by capturing retrieval results beside existing answers. For several runs, score retrieval without changing the release gate. When an answer fails, identify whether required evidence was absent, present but ignored, or itself stale. Those categories tell you which new gate is useful.

An answer-only runner usually breaks first at the case-to-retrieval relationship. A generation retry may reuse the original context, perform a new search, or combine results, while the old artifact stores only the final answer. Land an immutable retrieval-attempt identity and the exact context IDs sent to generation before introducing rank thresholds. Attach the completed index manifest to that attempt next, then add reviewed target labels and report-only grades. Promote revision integrity and forbidden-source rules before required-rank rules because their evidence is deterministic and does not depend on ranking preference.

The change is working when every answer case resolves to the context it actually received, each retry either points to the prior retrieval attempt or names a new one, and an incomplete manifest invalidates the run instead of borrowing metadata from the latest index. Only after those relationships hold should a retrieval failure block an answer release. That order prevents the new gate from blaming ranking for an artifact association bug the earlier suite could never expose.

Do not block on every rank movement. Swapping two equally acceptable chunks may change reciprocal rank without changing evidence quality. Gate on required evidence at the real cutoff, forbidden sources, revision integrity, and proven high-risk slices. Use rank metrics for trend and diagnosis unless the exact ordering has a demonstrated effect.

Avoid this evaluation when the product does not use retrieval for the tested path. If an exact language server lookup supplies symbol definitions, test that deterministic integration directly. Do not add vector search merely to make the evaluation more sophisticated.

Keep answer validation in place. Retrieval passing means the system had access to acceptable evidence. It does not prove the final code compiles, uses supported APIs, cites the right file, or preserves behavior. That boundary makes failures easier to route and prevents a good retrieval score from granting trust it did not earn.