Chunking and Reranking Interview Questions for AI QA Roles

Chunking and reranking interviews are architecture investigations disguised as parameter questions. A senior AI QA engineer should ask what evidence users need, how source structure is preserved, whether the first stage supplies a viable candidate set, and what the reranker changes under a controlled ablation.

The scenarios below demand artifacts: corpus snapshots, chunk manifests, parent links, candidate lists, ranks before and after reranking, answer claims, and latency traces. "Try a bigger chunk" is an experiment idea, not an answer.

Design the Retrieval Experiment

The official LangChain retrieval documentation describes a modular retrieval pipeline that can include document loaders, text splitters, embeddings, vector stores, and retrievers. Testing should preserve each stage's output so a chunking defect is not misdiagnosed as an embedding or generation defect.

A fair comparison freezes source documents, query set, labels, embedding configuration, generator, and prompt unless that component is the declared variable.

Before the run, materialize an immutable chunk manifest that maps each source revision to chunk text hashes, offsets, parents, metadata, and permissions. After the run, store candidate identifiers before filtering, before reranking, and after final context assembly. These checkpoints let reviewers replay one failed query without rebuilding the index from a moving corpus. They also reveal accidental experimental changes, such as a parser update or stale metadata cache, that would otherwise be credited to the chunk size under test.

Corpus and Boundary Choices

1. Why should chunk boundaries follow document semantics when possible?

Headings, paragraphs, clauses, table rows, and code symbols often define units that can support a claim independently. Blind character windows may split a condition from its exception or a function signature from its behavior. I would retain structural path and offsets in every chunk, then test boundary-spanning questions. Semantic splitting can produce uneven sizes, so hard limits and parent retrieval remain necessary for extreme sections.

2. How would you test a policy corpus with exceptions at the end of sections?

Create queries where the default rule and exception are both required, including paraphrases and users who qualify only for the exception. Inspect whether each chunk contains enough context or links to a parent containing both. Compare fixed windows, heading-aware chunks, and parent-child retrieval. A failure is not merely low similarity; it is an answer that states the default while omitting the controlling exception.

3. Why can one chunk size not serve every document type?

API references, contracts, chat transcripts, tables, and narrative guides have different evidence units. I would classify corpus types and test a small set of structure-aware policies behind one versioned interface. Per-type policies add operational complexity and can create inconsistent ranking scores, so the manifest must expose chunker identity and evaluation slices must include cross-type queries.

4. How would you preserve provenance through chunk creation?

Each chunk needs source identifier, source revision, structural path, start and end offsets, permissions, effective date, chunker configuration, and parent identifier. Hash the normalized text and manifest. The answer layer can then cite a source span, and an investigator can reconstruct why it appeared. Missing provenance should fail ingestion validation rather than becoming an untraceable context string.

Size, Overlap, and Special Content

5. How would you evaluate chunk overlap?

Build boundary-focused queries and compare no overlap with several illustrative overlap settings while holding other components fixed. Measure candidate recall, unique-source diversity, duplicate occupancy, index size, context use, and grounded answers. Overlap is valuable when evidence crosses an artificial boundary, but excessive duplication can consume top ranks. I would select the smallest overlap that resolves demonstrated boundary failures, not a habitual percentage.

6. Why should tables be chunked differently from prose?

A row without headers can lose meaning, while splitting related rows can break comparison questions. Preserve table title, headers, units, row keys, footnotes, and source coordinates. Test cell lookup, row filtering, and cross-row aggregation separately. Flattened text may work for simple search but must be validated against structure-aware representations. Evidence includes the exact cells used in the generated claim.

7. How would you test code or API reference chunking?

Label queries requiring symbol declaration, parameter constraints, examples, and related error behavior. Keep signatures and doc comments together, retain module and symbol paths, and consider parent expansion for long implementations. Verify that overloads and similarly named symbols do not collide. A generated code answer that cites the wrong version or sibling symbol exposes retrieval ambiguity even if lexical match appears strong.

8. Why can larger chunks reduce retrieval quality?

They mix several topics into one representation, so a small decisive span may be diluted by unrelated text. Larger payloads also consume context and can introduce conflicting statements. I would compare retrieval of the same evidence under controlled chunk sizes, then inspect rank, highlighted support span, and answer use. Larger chunks can improve continuity, so parent retrieval may offer continuity without making every indexed unit broad.

Candidate Retrieval and Reranking

9. How would you prove whether a miss occurred before or during reranking?

Persist ordered first-stage candidates and ordered reranked candidates with identifiers and scores. If the relevant chunk is absent before reranking, investigate indexing, query representation, filters, or candidate depth. If present and then demoted, inspect the reranker and competing chunks. This simple stage boundary prevents teams from tuning reranker prompts or models for evidence they never receive.

10. Why evaluate candidate depth independently from final context size?

The reranker may need a broad candidate pool, while the generator should receive a smaller context. Sweep candidate depth and final context depth separately, measuring candidate recall, reranking latency, final relevance, and answer quality. A shallow pool caps recoverable recall; an unnecessarily deep pool raises cost. Any chosen values are illustrative until validated on production-like query and corpus slices.

11. How would you test a cross-encoder reranker without changing chunking?

Freeze the chunk manifest, first-stage candidate IDs, query set, and relevance judgments. Run baseline ordering and reranked ordering over identical candidates. Compare rank-sensitive relevance, critical-claim availability at final k, contradiction placement, and latency. Then run end-to-end answers. The fixed candidate set isolates ordering effects; a later full-pipeline run checks interaction with real candidate retrieval.

12. Why test reranker behavior on near-duplicate chunks?

Near-duplicates reveal whether top positions collapse onto one source family. Construct candidates containing repeated templates, revised documents, and overlapping siblings, with one authoritative or current passage. Inspect diversity and effective-date use. Deduplication before or after reranking has different tradeoffs, so preserve both traces. A result filled with equivalent chunks can have good per-item relevance but poor claim coverage.

{
  "experiment": "illustrative_chunk_rerank_ablation",
  "fixed": ["corpus_snapshot", "queries", "qrels", "embedding", "generator_prompt"],
  "variables": {
    "chunk_policy": ["heading", "fixed_window", "parent_child"],
    "reranker": ["off", "on"]
  },
  "artifacts": ["chunk_manifest", "candidates_before", "candidates_after", "answer_claims"]
}

Metadata, Filters, and Freshness

13. How would you test metadata filters that run before reranking?

Create cases for allowed and denied tenants, document status, locale, product, and effective date. Verify the filter against the source authorization model, then inspect filtered candidate counts before ranking. An overly strict filter creates unrecoverable recall loss; a permissive filter can leak data. Reranker quality cannot compensate for either, so permission and applicability filters need deterministic contract tests and audit logs.

14. Why include document revisions in chunking tests?

Revisions can leave stale chunks indexed beside current ones, alter offsets, or produce duplicate source families. Ingest a controlled update and deletion, then verify old chunk removal, new manifest identity, effective-date ranking, and citation resolution. Query both changed and unchanged sections. A correct answer citing an obsolete source is still a pipeline failure when current policy controls the task.

15. How would you diagnose good offline ranking but poor production answers?

Compare production query distribution, filters, corpus freshness, candidate traces, context assembly, and answer prompt with the offline harness. Offline qrels may reward topic relevance while production needs sufficiency or current authority. Also inspect truncation and formatting between reranker and generator. Replay redacted traces against the frozen offline pipeline to identify where parity breaks, then add the discovered slice to evaluation.

End-to-End Tradeoffs and Release Gates

16. Why can a reranker improve relevance metrics but hurt latency objectives?

Scoring more query-document pairs adds work and may lengthen a critical request path. Measure stage latency distributions, timeouts, fallback behavior, and quality under candidate-depth ablations. The release policy should name whether degraded requests bypass reranking, reduce depth, or abstain. Do not invent a universal latency budget; use the product's objective and label any interview threshold as illustrative.

17. How would you choose between parent-child retrieval and large indexed chunks?

Compare a small searchable child with parent expansion against directly indexing large chunks. Evaluate candidate specificity, parent relevance, context duplication, permission consistency, citation precision, and update complexity. Parent expansion can restore surrounding conditions, but an irrelevant parent may add noise. I would favor the design whose failure artifacts show required evidence with the least uncontrolled context for target query types.

18. Why should a chunking or reranking release include qualitative review?

Metrics can miss broken sentences, lost headers, mixed permissions, stale passages, and contexts that are technically relevant but unusable together. Review sampled wins, losses, and unchanged cases by slice, with retrieved spans and generated claims side by side. Block the release for systematic critical-evidence loss or authorization defects. A favorable average does not outweigh a structurally unsafe retrieval path.

The review sample should include the largest rank movements, newly retrieved sources, dropped previously relevant sources, boundary-spanning queries, and cases where retrieval improved but the answer worsened. Record the reviewer decision against the experiment identifier so a later rollback can restore both the index artifact and the reasoning behind its approval.

Reach a Senior QA Verdict

Strong candidates isolate stages, preserve provenance, and design ablations that change one declared variable. They know reranking cannot recover absent candidates and that overlap, large chunks, and deep pools each purchase recall with distinct duplication, latency, and context costs.

The decisive recommendation should name the chosen chunk policy, candidate and context strategy, measured tradeoff, and rollback evidence. Approve only when required claims survive real document boundaries and the full answer pipeline uses them correctly.