Testing Vector Databases: QA Guide for Search and RAG Systems

A vector-store migration copied every reported record and passed a smoke query, but deleted documents continued to appear for several hours in one replica. A RAG assistant then cited a policy that legal had withdrawn. The embedding model was fine. The failure was consistency, deletion propagation, and routing across the storage layer.

Testing vector databases requires database thinking plus ranking evaluation. You need evidence for data integrity, namespace and filter correctness, approximate-search recall, concurrency behavior, operational recovery, latency, and cost. Final answer quality belongs in a separate end-to-end suite.

Define the storage and search contract

Specify the record shape and lifecycle. A vector record commonly includes a stable ID, vector, metadata, source version, tenant or namespace, and timestamps. Document allowed dimensions, distance metric, update semantics, delete semantics, and consistency expectations.

Write service objectives around user-visible behavior. Examples include: an acknowledged insert becomes searchable within a defined interval; a revoked document is no longer returned within the deletion objective; tenant filters never leak records; top-k query latency meets its percentile target at expected concurrency; and a migration preserves acceptable recall.

Separate database responsibility from embedding quality. Use fixed synthetic vectors for storage correctness and a frozen real embedding dataset for search relevance. Otherwise a provider or model change can make a database regression look like semantic variation.

Create deterministic vector fixtures

Small geometric fixtures make exact behavior testable. Use low-dimensional vectors in a test-only index where nearest neighbors are obvious. Include identical vectors, ties, zero or invalid vectors where the product must reject them, and metadata combinations that exercise filters.

{
  "records": [
    { "id": "a", "vector": [1.0, 0.0, 0.0], "metadata": { "tenant": "red", "active": true } },
    { "id": "b", "vector": [0.9, 0.1, 0.0], "metadata": { "tenant": "red", "active": false } },
    { "id": "c", "vector": [0.0, 1.0, 0.0], "metadata": { "tenant": "blue", "active": true } }
  ],
  "query": { "vector": [1.0, 0.0, 0.0], "filter": { "tenant": "red", "active": true } },
  "expectedIds": ["a"]
}

Generate larger fixtures with a known seed and store the seed, generator version, and expected exact-neighbor results. Keep a brute-force reference implementation for manageable datasets. The reference should use the same distance definition and normalization rule as the configured index.

Use unique markers in metadata to detect cross-namespace and stale-record failures. Never use production secrets or customer content in load tests.

Test CRUD, versioning, and data integrity

Verify insert, upsert, fetch, query, update, delete, batch operations, and duplicate IDs. Check dimension mismatch, nonfinite numbers, oversized metadata, unsupported filter types, partial batch failure, and retry behavior.

After an update, assert both the new vector behavior and metadata state. Ensure an upsert does not accidentally retain old metadata fields when replacement semantics are expected. For deletes, test direct fetch, vector query, filtered query, replicas, caches, and downstream export. Measure propagation time rather than sleeping for an arbitrary interval.

Use source-version metadata or optimistic concurrency when concurrent writers can update the same record. Test out-of-order delivery: an older ingestion event must not overwrite a newer policy version. Confirm idempotency for repeated batch requests and define how partially acknowledged batches are reconciled.

Reconcile counts by namespace, source, version, and ingestion batch. Total count equality is insufficient because one missing record can be offset by one duplicate. Compare stable IDs and checksums for metadata payloads.

Validate filters and tenant isolation

Metadata filtering is a correctness and security boundary. Build truth tables for equality, ranges, lists, boolean combinations, missing fields, null values, and type mismatches. Test filters with vector queries that would otherwise rank a forbidden record first.

Create paired tenants with similar vectors and distinctive canary IDs. Run queries using every application role and namespace configuration. Assert that forbidden IDs are absent from results, diagnostic payloads, pagination, caches, backups used by the test environment, and query logs available to the caller.

Test fail-closed behavior when tenant metadata is absent or malformed. If the application composes filter expressions, verify escaping and precedence with deterministic unit tests before exercising the database. A model-generated filter should be parsed into an allowlisted structure, not passed as unrestricted query text.

Also test pagination or continuation tokens with filters. Changing the filter between pages must not expose records from the previous scope.

Measure approximate recall against exact search

Approximate nearest-neighbor indexes trade recall for speed and resource use. Evaluate the database by comparing returned neighbors with brute-force exact neighbors on a representative frozen vector set.

def recall_at_k(approx_ids, exact_ids, k):
    expected = set(exact_ids[:k])
    observed = set(approx_ids[:k])
    return len(expected & observed) / k

Run queries across dense regions, sparse regions, ties, outliers, and filter selectivity levels. Report recall distribution, not only a mean. Low-selectivity and high-selectivity filters may use different execution paths. Measure at the k values the application actually consumes.

If tuning parameters control search effort, sweep them under the same load and dataset. Plot or tabulate recall against latency and resource use. Select an operating point based on product needs. Do not claim a universal recall value from a tiny synthetic index.

For semantic relevance, layer a separate labeled query dataset on top. Exact-neighbor agreement tells you whether the database approximates its distance function, not whether the embeddings represent user intent.

Load test realistic query and ingestion mixes

Model production traffic as a mix of reads, upserts, deletes, filter patterns, vector dimensions, top-k values, and batch sizes. Warm-cache benchmarks alone are misleading. Include cold starts, index build periods, compaction, replica changes, and background ingestion.

Measure P50, P95, and P99 query latency, write acknowledgment latency, searchable-after-write delay, error and throttling rates, throughput, memory, CPU, and network use where visible. Separate client-side queueing from server latency. Capture result counts and correctness during load, not merely response times.

Use step tests to find the knee where latency or errors rise sharply, then soak at expected peak to expose resource leaks and compaction effects. Apply backpressure in the test client so it does not create an unrealistic retry storm. Verify that throttling and partial failures are observable to ingestion recovery code.

Cost should be normalized to the workload: indexed vectors, dimensions, replicas, storage, queries, writes, and data transfer. A lower P95 achieved by doubling replicas is a capacity decision, not a free performance win.

Test migrations, backup, and recovery

Treat a model, dimension, metric, index-type, or vendor change as a migration. Build the candidate index from a versioned source manifest. Reconcile IDs, metadata checksums, vector model version, failed batches, and deletion tombstones.

Shadow-read the candidate with recorded or privacy-reviewed queries. Compare top-k overlap, exact recall, filter correctness, empty-result rate, latency, and cost. Differences are expected, so send meaningful semantic changes to domain review rather than demanding identical rankings.

Exercise rollback before cutover. Use versioned aliases or routing configuration so the previous index remains available until the acceptance window closes. Test backups and restores for data plus configuration. Measure recovery time and verify restored access controls, not just record counts.

Simulate interrupted migrations and replay. Confirm the process resumes idempotently without mixing vector versions. If dual writes are used, test divergence detection and reconciliation before relying on them.

Investigate with query and index evidence

For a failure, retain query vector provenance, filter, namespace, k, index configuration, returned IDs and scores, exact neighbors when available, replica or region, timestamps, and recent write history. This evidence distinguishes ranking approximation from stale replication or bad filtering.

Classify defects as ingestion loss, version conflict, stale delete, metadata corruption, isolation failure, approximate-recall miss, capacity saturation, client retry error, or migration divergence. Route each class to the correct owner. Prompt tuning cannot fix an index that still serves deleted records.

Human review is useful for semantic ranking changes and migration samples. It is not a substitute for deterministic integrity, isolation, and lifecycle assertions.

Set a database-specific release gate

Require zero tenant-isolation failures, zero acknowledged-but-lost writes, and no deletion propagation beyond the agreed safety objective. Set minimum exact-neighbor recall by operating k and filter slice. Add latency, error-rate, ingestion-lag, recovery, and cost thresholds under the target load profile.

Compare the candidate with the approved baseline using the same infrastructure class and dataset. Review any small but high-impact slice, such as revoked-policy retrieval, separately from aggregate recall. Document known consistency behavior and application mitigations.

The sign-off package should include schema and lifecycle results, filter truth tables, recall distributions, load profile, percentile latency, migration reconciliation, restore proof, cost model, and rollback procedure. A vector database release is ready when it stores the right vectors, returns only authorized and current records, approximates search within the agreed quality budget, and remains operable under failure.