Compatibility Testing Guide: Devices, Browsers, OS, and Data

A warehouse app passed its desktop regression suite, then failed during a real stock count. Rugged Android scanners could open the item page, but the camera permission prompt never returned control to the app. Operators typed 18-character codes by hand, and one shift created duplicate adjustments after retrying on an unstable network. The feature was functionally correct. The supported operating context was not.

That is the release problem compatibility testing must answer: which combinations can complete a valuable workflow, preserve data, and recover from interruption? The useful output is not a wall of device names. It is evidence tied to a declared support policy.

Define compatibility as a product contract

Start with claims the product already makes. Collect the support page, mobile store requirements, enterprise contracts, installation guide, import formats, API version policy, and database deployment matrix. Convert each claim into an observable obligation. “Android supported” is too broad. “Stock count works on Android 13 and 14 devices with a rear camera, 4 GB RAM, and intermittent Wi-Fi” is testable.

Compatibility has several independent dimensions:

Record exclusions too. A combination outside policy is not automatically a defect, but the product should fail clearly if users can reach it. An installer that permits an unsupported OS and then corrupts an upgrade is still a product risk.

Build a matrix from evidence, not possibility

No team can cover the full Cartesian product. Select combinations from production analytics, contract obligations, revenue exposure, recent code changes, support tickets, and technical boundaries. Group equivalent combinations only when you can state why they are equivalent.

For the scanner release, a compact matrix could be:

Use pairwise selection for low-risk interactions, not for critical paths with known coupling. Camera permission, OS lifecycle, background sync, and network recovery interact directly, so those combinations deserve explicit cases. Keep an owner and review date beside the matrix. Otherwise an old list quietly becomes an accidental promise.

Design one end-to-end compatibility probe

A probe is a short workflow repeated unchanged across combinations. It should cross the components most likely to vary while leaving a traceable business result. For inventory counting:

1. Install or upgrade from the supported previous version.
2. Sign in as operator WH-07.
3. Scan item QA-COMP-184729563001 and enter quantity 12.
4. Disable the network before submission.
5. Restore the network and allow one retry.
6. Confirm one adjustment, one audit event, and the correct local status.
7. Background the app, reopen it, and print the reconciliation label.

The expected result is not merely “sync succeeds.” Capture the client build, OS build, device model, permission state, network profile, local event ID, server adjustment ID, timestamps, and retry count. The database or API evidence must show exactly one quantity delta of 12. If the UI reports success twice but the server holds one row, the client still has a misleading-state defect.

Exercise installation, upgrade, and old data

Fresh installation hides many compatibility failures. Preserve representative artifacts from released versions: application data, encrypted caches, configuration files, documents, and database backups. Never fabricate an “old” file by changing a version field. Generate it with the released application and retain its checksum.

An upgrade case should distinguish migration from normal startup:

Artifact: count-session-v4.db
Created by: mobile build 4.8.2 on Android 13
Precondition: two synced counts, one pending count, expired access token
Action: upgrade in place to candidate 5.0.0 and reconnect
Expected: all three counts visible; pending count syncs once; reauthentication requested;
          original created_at values retained; migration recorded as successful

The text block is a test fixture manifest. Reviewers should be able to identify the creating version and the exact invariants after migration. Also test downgrade behavior if the product permits downgrade. If it does not, verify that the attempt is blocked before any irreversible data change.

Test constrained resources and interruption

Compatibility includes conditions around a device, not only its label. Run focused tests with low storage, denied permissions, memory pressure, clock skew, proxy authentication, slow DNS, packet loss, screen rotation, sleep, and peripheral disconnects. Choose conditions supported by real evidence or architecture risk.

For network work, observe requests rather than relying on a spinner. A retryable write should carry a stable idempotency key. After a forced timeout, compare client logs and server records:

POST /inventory-adjustments HTTP/1.1
Content-Type: application/json
Idempotency-Key: qa-comp-c02-001

{"itemId":"QA-COMP-184729563001","warehouseId":"WH-07","delta":12}

The http example shows the key that must remain unchanged across a transport retry. The expected evidence is one committed adjustment. A new key on each retry explains duplicates; a stable key with duplicates points to server-side idempotency handling.

Resource tests need safe stop conditions. Filling a device until the OS becomes unstable creates noise. Define the boundary, such as 150 MB free storage, and record the measurement before the action.

Separate product defects from environment failures

When one combination fails, repeat in a control environment with the same account and data. Then vary one dimension at a time. Useful comparisons include same device with another OS version, same OS on another device, same build on another network, and previous product build on the failing combination.

A diagnosis record can be concise:

Do not close a failure as “device-specific” without a causal boundary. Device-specific defects are exactly what this testing is meant to find.

Automate stable layers and retain physical checks

Automate the probe against high-value browser and mobile configurations, API contract versions, file fixtures, and supported database containers. Store environment metadata with every result. A pass without the actual OS or engine version cannot support a compatibility decision.

Physical devices remain important for camera focus, biometric prompts, thermal throttling, Bluetooth reconnects, printers, keyboards, and operating-system permission flows. Emulators are valuable for rapid breadth, clean snapshots, and boundary versions. Use each for the evidence it can produce.

Quarantine is especially dangerous in a matrix. Five quarantined cases can erase all evidence for one platform while the aggregate dashboard stays green. Report coverage by combination: planned, executed, passed, failed, blocked, and not run.

Make the release decision explicit

Summarize results against the support contract, not against the number of tests. A compatibility report should include the tested build, matrix revision, workflows covered, artifact versions, environment evidence, open failures, untested commitments, and recommended action.

Release when every P0 combination has completed the probe with correct persisted state, upgrade evidence is clean, recovery does not duplicate or lose work, and remaining differences are accepted with an owner and customer-facing position. A conditional release may disable a peripheral path or narrow a support claim. “Passed on most devices” is not a decision rule.

The strongest compatibility result is traceable: this customer environment, this workflow, this data, this observed state. That evidence lets a team ship, restrict, or delay with its eyes open.