Why Android tests pass on Pixel and fail on Samsung

A release candidate passes every Pixel emulator test, then a Samsung user cannot complete checkout after dismissing the keyboard. The APK is identical. What changed is the operating environment: firmware, system components, permissions, input method, and lifecycle timing. That gap is where Android OEM compatibility testing earns its place.

The first job is not to prove that one manufacturer is broken. It is to reproduce one customer-visible failure while keeping the app build, account, data, and action sequence stable. Once that comparison is clean, the device becomes useful evidence instead of a convenient suspect.

Work out whether the device is really the variable

An Android model name is a bundle of variables, not a diagnosis. Two phones sold under the same product family can have different Android releases, security patches, carrier packages, WebView versions, accessibility settings, and power policies. A factory-reset phone in a lab can also behave differently from a customer phone that has upgraded through several releases. Labeling a ticket “Samsung issue” before recording those details throws away most of the information needed to reproduce it.

Start with the control run that already passes. Record the APK checksum, account, backend environment, locale, network path, permission state, and exact test revision. Then run the same journey on the reported device. Do not change the test data to make setup easier. Do not install a newer build on one side. If the failure disappears after those controls are aligned, the original comparison was not about the OEM at all.

The mechanism is often visible at a boundary. A button can be present in the app’s layout but covered by the active keyboard. A permission decision can cause the activity to resume through a path the test never exercises on its emulator snapshot. A hybrid screen can still look correct while Appium remains in the native context, where DOM locators cannot see the web content. A backgrounded process can return to a valid screen while the business action behind it was never persisted. These failures look unrelated in a screenshot, but each is a mismatch between what the test observes and what the user needs.

Appium does not erase those differences. It sends commands through a platform driver to the selected device. Its session capability documentation is explicit that capabilities describe the requested session and cannot be changed after that session starts. That is why the requested values and the returned capabilities belong in the evidence. A job name such as android-14-samsung is not proof that the server created the intended session.

Capture the device identity before Appium starts. The following script fails early when the serial is missing or empty, confirms that ADB can address the device, and saves properties that let another engineer identify the same configuration. It does not claim that any property caused the defect. It gives the investigation a stable comparison point.

Note the exact guard. Writing : "$ANDROID_SERIAL" under set -u aborts only when the variable is completely unset, and an empty value passes straight through into adb -s "" get-state, where it silently addresses whichever device happens to be attached. That is the common CI shape, because an unconfigured vars.CI_ANDROID_SERIAL expands to an empty string rather than disappearing. The :? form rejects unset and empty alike, which is the behavior the job actually needs.

#!/usr/bin/env bash
set -euo pipefail

: "${ANDROID_SERIAL:?a device serial must be allocated to this job}"
: "${APP_PACKAGE:?the application package under test must be named}"

artifact_dir="${ARTIFACT_DIR:-artifacts/device-profile}"
mkdir -p "$artifact_dir"

adb -s "$ANDROID_SERIAL" get-state > "$artifact_dir/adb-state.txt"

properties=(
  ro.product.manufacturer
  ro.product.model
  ro.product.device
  ro.build.version.release
  ro.build.version.sdk
  ro.build.fingerprint
  ro.build.version.security_patch
)

: > "$artifact_dir/properties.txt"
for property in "${properties[@]}"; do
  value="$(adb -s "$ANDROID_SERIAL" shell getprop "$property" | tr -d '\r')"
  printf '%s=%s\n' "$property" "$value" >> "$artifact_dir/properties.txt"
done

adb -s "$ANDROID_SERIAL" shell wm size > "$artifact_dir/display-size.txt"
adb -s "$ANDROID_SERIAL" shell wm density > "$artifact_dir/display-density.txt"
adb -s "$ANDROID_SERIAL" shell dumpsys package "$APP_PACKAGE" \
  > "$artifact_dir/package-state.txt"

Read the ADB state before reading a test exception. A serial reported as unauthorized or offline points to device preparation or connectivity, not an application assertion. An empty package report means the expected package may not be installed under that identifier. A build fingerprint that differs between the passing and failing attempts means the comparison includes a firmware change. None of these facts proves the app is healthy, but each prevents a lab problem from being filed as a product defect.

The next boundary is the Appium session. The official Python quickstart uses UiAutomator2Options, starts a remote session, finds an element, and calls quit() during teardown. Keep that lifecycle. If setup creates a driver and the test fails before teardown owns it, the next case inherits a busy device or a stale application state. A leaked session can then make every later result misleading.

Build a matrix that can answer one question at a time

A useful matrix is a set of hypotheses. It is not a shopping list of popular phones. Each row should exist because it represents customer exposure, a platform boundary, or a defect history that another row does not cover.

Begin with operating-system support. If the product supports Android 12 through 15, select at least one reliable control for each release that matters to a release decision. Then layer manufacturer coverage where it changes risk. A camera-heavy app may care about devices used by its customers and the system surfaces around capture. A field-service app may care more about background recovery, offline storage, battery state, and managed-device settings. An authentication app may prioritize biometrics, browser handoff, and process restoration. The same generic five-phone list would be poor for all three.

Keep a reason beside every selected combination. “Largest production cohort,” “oldest supported Android release,” “only device that reproduced incident MOB-482,” and “hybrid flow uses this WebView family” are reviewable reasons. “Samsung,” “cheap phone,” and “latest Android” are labels, not selection logic. When a device ages out, the reason tells you whether to replace it, retain it for regression, or remove it.

Separate breadth from depth. A smoke matrix can run login, one core transaction, and logout on several devices. A smaller release matrix can exercise denial and regrant of permissions, background and foreground transitions, offline recovery, keyboard-sensitive forms, deep links, notifications, and upgrade state. Running every scenario on every device usually creates more queue time than information. It also makes failures arrive in a pile, which encourages retries instead of diagnosis.

Use pairs when you investigate. If checkout fails on one physical device, compare it with a control that matches the Android version before comparing it with a newer emulator. If a WebView flow fails, record the contexts and the system web component before changing locators. If a permission path fails, preserve whether the app was freshly installed, upgraded, or previously denied. One variable will not always be perfectly isolated on commercial phones, but deliberate pairing narrows the candidate set.

Three different worked examples show why this matters.

In the first, the keyboard covers the final action on a compact display. The automation still finds a matching element in the hierarchy, but the user cannot tap the visible control. A click command alone is a weak oracle. The test should verify the post-action state, capture a screenshot before the click, and inspect whether the keyboard is shown. If the same layout fails on another compact device from a different manufacturer, screen geometry or app layout becomes more plausible than OEM firmware.

In the second, a draft appears after the app returns to the foreground, but the server never received it. The correct assertion is not “draft screen visible.” Use a unique business identifier, query the server through a supported test interface, and prove that exactly one record exists after recovery. If the screen restores but the record does not, the defect sits across lifecycle and persistence. If neither device reaches the server under the same network, investigate the backend or data before the phone.

In the third, a hybrid payment page is visible but a DOM locator times out. Appium’s context guide explains that drivers can expose native and web contexts and that command behavior changes with the selected context. Save the available context names at the failure point. A list containing only the native context is different evidence from a list containing the expected web context while the test selected the wrong one. The first points toward WebView exposure, readiness, or driver compatibility. The second is test logic.

Permission history creates a fourth, genuinely different comparison. A clean emulator may grant or deny a permission during first launch, while a customer phone carries a previous denial, an app upgrade, or a setting changed outside the app. Record the starting permission state and the route used to change it. Then assert the feature outcome after the app resumes. If the failure follows “previously denied” state across two manufacturers, the lifecycle path is a stronger suspect than the OEM. If it occurs only on one recorded firmware build with the same starting state, the device comparison remains useful.

Upgrade testing needs its own lane for the same reason. Installing a release APK over a supported older build exercises retained data and migration code that a fresh Appium install does not. Prepare the old version deliberately, create a named piece of state, install the candidate through the lab’s owned procedure, and verify both UI restoration and the server record. Keep that result separate from clean-install smoke. Combining the two under one device row makes a pass impossible to interpret because the starting application state changed with the test.

Do not merge those examples into one “OEM compatibility” bucket. They need different owners, artifacts, and fixes. The matrix finds the boundary. A well-designed assertion explains what crossed it incorrectly.

Automate the journey and keep the evidence together

Real-device automation should identify the device from outside the test, request it explicitly, and save evidence under one attempt directory. The appium:udid capability matters on a host with more than one connected device. Appium’s capability guide defines it as the unique device identifier, while the UiAutomator2 ecosystem guidance recommends setting a unique device target for parallel work. Leaving it out and accepting the first connected device makes a device matrix untrustworthy.

The following pytest example expects an application whose accessibility identifiers are Email, Password, Sign in, Cart, Place order, and Order received. Those identifiers are part of this example app’s test contract. Replace them with identifiers your product team owns. Do not replace them with vendor-specific visible text merely to make one phone pass.

import json
import os
import sys
from pathlib import Path

import pytest
from appium import webdriver
from appium.options.android import UiAutomator2Options
from appium.webdriver.common.appiumby import AppiumBy
from selenium.webdriver.support.ui import WebDriverWait


def wait_for_accessibility_id(driver, value: str):
    return WebDriverWait(driver, 20).until(
        lambda current: current.find_element(AppiumBy.ACCESSIBILITY_ID, value)
    )


@pytest.fixture
def driver():
    capabilities = {
        "platformName": "Android",
        "appium:automationName": "UiAutomator2",
        "appium:udid": os.environ["ANDROID_SERIAL"],
        "appium:app": os.environ["APP_APK"],
    }
    options = UiAutomator2Options().load_capabilities(capabilities)
    session = webdriver.Remote(
        os.environ.get("APPIUM_SERVER_URL", "http://127.0.0.1:4723"),
        options=options,
    )
    yield session
    session.quit()


def test_signed_in_customer_can_place_one_order(driver):
    attempt = Path(os.environ.get("ARTIFACT_DIR", "artifacts/appium"))
    attempt.mkdir(parents=True, exist_ok=True)

    try:
        wait_for_accessibility_id(driver, "Email").send_keys(
            os.environ["TEST_CUSTOMER_EMAIL"]
        )
        wait_for_accessibility_id(driver, "Password").send_keys(
            os.environ["TEST_CUSTOMER_PASSWORD"]
        )
        wait_for_accessibility_id(driver, "Sign in").click()
        wait_for_accessibility_id(driver, "Cart").click()
        wait_for_accessibility_id(driver, "Place order").click()

        confirmation = wait_for_accessibility_id(driver, "Order received")
        assert confirmation.is_displayed()
    except Exception:
        safe_capability_names = {
            "platformName",
            "automationName",
            "deviceName",
            "platformVersion",
            "udid",
        }
        safe_capabilities = {
            name: value
            for name, value in driver.capabilities.items()
            if name in safe_capability_names
        }

        def save_failure_screenshot():
            saved = driver.save_screenshot(str(attempt / "failure.png"))
            if not saved:
                raise RuntimeError("Appium did not save the failure screenshot")

        evidence_writers = [
            save_failure_screenshot,
            lambda: (attempt / "page-source.xml").write_text(
                driver.page_source, encoding="utf-8"
            ),
            lambda: (attempt / "capabilities.json").write_text(
                json.dumps(safe_capabilities, indent=2, default=str),
                encoding="utf-8",
            ),
        ]
        for write_evidence in evidence_writers:
            try:
                write_evidence()
            except Exception as evidence_error:
                print(
                    "evidence capture failed: " + repr(evidence_error),
                    file=sys.stderr,
                )
        raise

This test has an oracle that can fail when the application changes. If the order action no longer produces the confirmation state, the assertion fails. In a production suite, strengthen it with a server-side lookup keyed by a unique order reference. Do not assert that a hard-coded fixture contains the value you placed in that same fixture. The evidence must come from the system under test.

The screenshot, source, and capabilities answer different questions. The screenshot shows what a user could see. The page source shows what the automation driver exposed at that moment. The filtered capabilities show which reviewed session fields the server returned. A screenshot with a visible button and a source without the expected accessibility node suggests a different problem from a source containing the node beneath a system overlay. Keep all three, and extend the capability allowlist only after reviewing provider-specific fields for tokens or tunnel data. Treat a physical-device serial as internal lab data and apply the artifact access policy to it.

The allowlist is deliberately written without appium: prefixes, and that asymmetry with the fixture above is the point. The prefix is a request-side W3C vendor namespace. The client sends appium:udid, the server strips the namespace while matching the request, and the session response carries the matched values under their plain names. If the allowlist repeats the prefixed spelling, four of its five entries can never match anything, the filter quietly reduces to platformName, and capabilities.json becomes a one-line file that looks like successful evidence collection. Write the request with the prefix and read the response without it. When you widen the allowlist, confirm each new name against a captured response from your own server rather than against the options object the test built.

A second test should cover lifecycle rather than repeat checkout with different data. Appium's application-management endpoints define app activation and termination. The Python client exposes those commands through the driver. Terminating and activating an app is a controlled way to check cold restoration, but it is not the same as every customer background scenario. State that limitation in the test name and report.

Wait for an app-owned save confirmation before terminating the process. Otherwise, the test races persistence and cannot tell a restoration defect from a save that never finished.

The test below follows the rule stated earlier in this article rather than contradicting it. A screen assertion alone cannot separate a restored local cache from a persisted record, and those two outcomes have different owners. So the draft carries a reference generated for this attempt, the UI assertion proves that the user sees their work again, and a query against a product-owned test interface proves that exactly one record exists on the server. Both assertions have to hold.

import os
import uuid

import requests
from appium.webdriver.common.appiumby import AppiumBy
from selenium.webdriver.support.ui import WebDriverWait


def fetch_persisted_drafts(reference: str) -> list[dict]:
    response = requests.get(
        os.environ["TEST_SUPPORT_URL"] + "/drafts",
        params={"reference": reference},
        headers={
            "Authorization": "Bearer " + os.environ["TEST_SUPPORT_TOKEN"],
        },
        timeout=20,
    )
    response.raise_for_status()
    return response.json()["drafts"]


def test_saved_draft_survives_app_termination(driver):
    package = os.environ["APP_PACKAGE"]
    reference = "draft-" + uuid.uuid4().hex[:12]

    driver.find_element(AppiumBy.ACCESSIBILITY_ID, "New draft").click()
    editor = driver.find_element(AppiumBy.ACCESSIBILITY_ID, "Draft text")
    editor.send_keys(reference)
    driver.find_element(AppiumBy.ACCESSIBILITY_ID, "Save draft").click()
    WebDriverWait(driver, 20).until(
        lambda current: current.find_element(
            AppiumBy.ACCESSIBILITY_ID, "Draft saved"
        )
    )

    driver.terminate_app(package)
    driver.activate_app(package)

    restored = WebDriverWait(driver, 20).until(
        lambda current: current.find_element(
            AppiumBy.ACCESSIBILITY_ID, "Draft text"
        )
    )
    assert restored.text == reference

    persisted = fetch_persisted_drafts(reference)
    assert len(persisted) == 1, (
        f"expected one persisted draft for {reference}, "
        f"found {len(persisted)}"
    )
    assert persisted[0]["state"] == "saved"

Each of those three failure modes lands on a different desk. A restored screen with zero server records is the lifecycle and persistence defect this section is about, and it is exactly the outcome a UI-only assertion would have called a pass. Two server records mean the save path ran twice across termination, which is a duplication defect rather than a loss. A server record in a state other than saved means the write reached the backend but the transition did not complete. The reference is generated per attempt so that a rerun cannot inherit a record left by the previous one and read it as today's evidence.

TEST_SUPPORT_URL and TEST_SUPPORT_TOKEN describe a contract your product team owns and authenticates. Appium cannot supply that endpoint, and no mobile framework can invent one. If the product exposes no query for saved drafts, say so in the test name and the release report, keep the UI assertion, and record explicitly that persistence is unproven for this journey. That is a narrower claim than the one above, and a narrower claim is better than a broad claim the evidence does not support.

The cost of this automation is state management. A clean install produces repeatable setup but does not cover upgrade paths or retained permissions. noReset can preserve useful state, yet a dirty device makes tests influence one another. Keep separate lanes: one for clean-install behavior, one with a deliberately prepared upgrade or retained-state fixture. Never let “whatever was left on the phone” become an unnamed third lane.

Separate an OEM defect from three convincing lookalikes

The strongest near-miss is a device-lab failure. A disconnected cable, locked screen, stale Appium server, unavailable serial, or exhausted cloud allocation can stop a test before the application receives its first action. The evidence boundary is simple: did the Appium session start, and did the first application command complete? If not, classify the attempt as infrastructure until app evidence says otherwise. Retrying may be appropriate after preserving the server log, but the original result must remain visible.

Dirty state is the second lookalike. One phone has a cached session and granted permissions, another has a fresh install, and the test calls both results a device comparison. Inspect package state, app data preparation, account records, and permission setup. A failure that follows the account after the devices swap is a data or backend problem. A failure that follows retained application state after reinstall strategy changes is a fixture problem. Only a failure that remains tied to the controlled device configuration supports an OEM hypothesis.

The third lookalike is the automation surface. Hybrid applications are especially good at producing it. The human sees one screen, but the driver operates within one active context. Capture the context list and current context near the failing action. If the expected web context exists and the test never selects it, fix the framework. If the context never appears on one device, compare WebView exposure, app configuration, and driver logs. Do not add a sleep and call the issue solved. A longer delay changes timing without explaining readiness.

Selector instability can create the same illusion in native screens. Vendor system dialogs may use resources owned by the system package, localized labels, or layouts that differ from the application’s own view hierarchy. Tests that reach into those dialogs with copied XPath expressions are coupled to that device image. Prefer app-owned accessibility identifiers for app controls. For system UI, isolate a small adapter per supported flow and assert the result inside the app, such as the permission-dependent feature becoming available or remaining safely disabled.

Server evidence is the tie-breaker for many “it looked successful” reports. Assign a unique identifier to a transaction before the test starts. After the UI action, query a supported test or API surface for that identifier. Zero records means the UI did not create the outcome. Two records means retry or lifecycle behavior duplicated it. One record with the wrong state means the transition was incomplete. These are product assertions. The phone brand alone cannot answer them.

Logcat is valuable when it is bounded. Clear it immediately before the case or record precise start and stop markers. Save the raw capture, then search for the application process, fatal exceptions, activity lifecycle events, and the unique transaction identifier. A device-wide log gathered for an entire overnight suite is noisy and can contain unrelated user or system data. Treat it as a protected artifact, not a text blob to paste into a public ticket.

When a crash occurs, preserve the first exception and its causal chain. Do not quote a generic Appium timeout as the crash reason if logcat shows the application process died earlier. Conversely, do not call every system warning a crash. The decisive evidence is the process termination or crash output tied to the app and the failing interval, plus the missing product outcome.

A useful triage note reads like this: the control and target ran APK checksum X with account Y; the target session capabilities named serial Z and Android build fingerprint Q; the action reached “Place order”; the app process stopped before confirmation; no order with reference R exists; the bounded log contains the application’s fatal exception. That note gives a developer a reproducible boundary. “Fails on Samsung” does not.

Roll real devices into CI without hiding failures

Run the emulator control frequently because it is fast and replaceable. Run physical-device coverage where its evidence changes a decision: platform-sensitive pull requests, scheduled regression, release candidates, and verification of an escaped defect. A real-device job that takes a scarce phone for every documentation change creates queues without buying relevant coverage.

Give each device one worker at a time unless the lab has explicitly isolated parallel Appium ports and devices. The udid must identify the allocated phone. The device must begin in a declared state. The job must always release the Appium session and archive evidence even when the assertion fails. Queue ownership, session ownership, and application cleanup are separate responsibilities, so make all three visible.

This workflow is intentionally a single self-hosted lab lane. It assumes the runner already has Python, ADB, Appium, the UiAutomator2 driver, a connected device, and the APK at the declared path. Those are runner-image responsibilities, not hidden install steps inside every test. Additional models should be separate jobs or a documented matrix, each with its own runner label and serial allocation.

One assumption deserves to be spelled out because it is the easiest to leave implicit. Having Appium installed on the image is not the same as having an Appium server listening at APPIUM_SERVER_URL. If nothing is listening, webdriver.Remote() fails with a connection error before any device work begins, and the report will show a mobile test failure for what is really a missing process. So the job either starts the server itself and waits for /status to answer, as below, or the lab runs Appium as a long-lived service and that service becomes a named part of the runner contract alongside ADB and the connected phone. Pick one and write it down. The version that starts its own server keeps ownership inside the job, cleans up on every exit path, and preserves the server log next to the test evidence.

name: android-real-device-regression

on:
  workflow_dispatch:
  schedule:
    - cron: "30 1 * * 2-6"

jobs:
  samsung-smoke:
    runs-on: [self-hosted, android-lab, samsung-smoke]
    timeout-minutes: 35
    env:
      ANDROID_SERIAL: R5CT-LAB-01
      APP_PACKAGE: com.example.shop
      APP_APK: /opt/test-apps/shop-release.apk
      APPIUM_SERVER_URL: http://127.0.0.1:4723
      ARTIFACT_DIR: artifacts/samsung-smoke
    steps:
      - uses: actions/checkout@v4
      - name: Record device profile
        run: ./scripts/capture-android-profile.sh
      - name: Run the owned smoke journey
        shell: bash
        run: |
          set -euo pipefail
          appium --address 127.0.0.1 --port 4723 \
            >appium-server.log 2>&1 &
          appium_pid=$!

          cleanup() {
            kill "${appium_pid}" 2>/dev/null || true
            wait "${appium_pid}" 2>/dev/null || true
          }
          trap cleanup EXIT

          for attempt in 1 2 3 4 5 6 7 8 9 10; do
            if ! kill -0 "${appium_pid}" 2>/dev/null; then
              sed -n '1,200p' appium-server.log
              exit 1
            fi
            if curl --fail --silent \
              "${APPIUM_SERVER_URL}/status" >/dev/null; then
              python -m pytest tests/mobile/test_checkout.py -q
              exit
            fi
            sleep 1
          done

          printf 'Appium did not answer /status before the probe budget\n' >&2
          sed -n '1,200p' appium-server.log
          exit 1
      - name: Preserve first-attempt evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: samsung-smoke-evidence
          path: |
            artifacts/samsung-smoke
            appium-server.log

The readiness loop is a routing check, not a device check. A 200 from /status proves that a server answered at that origin and base path. It does not reserve the phone, select UiAutomator2, or install the APK, and the loop is deliberately named after what it polls rather than after what the job hopes is true. If the server process dies during startup, the loop notices the dead PID and prints the log rather than spending its full budget waiting for a socket that will never open.

The serial in this illustrative workflow is a lab allocation name, not a measurement and not a recommendation for a real device identifier. In a shared lab, inject the allocated serial through the runner or scheduler rather than committing it. The important contract is that the job records what it used and fails if that allocation is absent.

Do not let a blanket retry rewrite the result. Store attempt one under its own directory. If the lab retries a session-start failure, store attempt two separately and classify the pair. “Infrastructure failure, retry passed” is operationally different from “product assertion failed, retry passed.” The second is a flaky product or test signal until investigated.

Roll out the matrix in stages. First, run it without gating and measure only operational facts you actually observe: queue availability, session-start reliability, test duration, and failure categories. Next, gate one stable journey on devices that represent a release promise. Add deeper cases after their setup and teardown have proven independent. Finally, review each row quarterly or when support policy and customer usage change. A permanent matrix accumulates obsolete phones and duplicated coverage unless someone owns subtraction.

Every added device costs reservation time, maintenance, charging, OS-update control, account data, artifact storage, and triage attention. Device clouds trade hardware maintenance for provider capabilities, queue rules, and less physical access. Local labs give control but require operational ownership. There is no free “run on all phones” option. Spend the budget where a distinct device result would change a release decision.

Know when broad OEM coverage is the wrong investment

Do not send every business rule through a phone. Pricing, scoring, authorization, serialization, and retry idempotency usually belong in faster component or API tests. Keep one or two mobile journeys that prove the layers connect, then test the combinatorial logic below the UI. A device matrix is expensive evidence for a pure calculation.

Avoid broad manufacturer coverage when the support promise is narrow. If a managed enterprise app is deployed to two approved models, test those models deeply and document the boundary. Testing ten consumer phones may look impressive while leaving the managed configuration under-tested. Coverage should match the promise, not a generic market-share slide.

Do not call a cosmetic difference an incompatibility until it violates a requirement. Font rendering can vary without blocking reading or action. A clipped legal disclosure, unreachable button, lost draft, duplicated payment, or inaccessible control crosses a product boundary. Record that boundary in the assertion so the test survives harmless visual change.

Manual testing is often the better tool for a rare vendor panel, stylus interaction, fold transition, camera behavior, or hardware-backed flow. Automate it only when repeat frequency, release risk, and stable control justify the framework cost. A scripted manual case can still require the APK checksum, device profile, starting state, action sequence, server evidence, screenshot, and log window. “Manual” does not mean unrepeatable.

Finally, stop expanding the matrix when failures cannot be triaged. Five unexplained red jobs do not provide more protection than one. Stabilize allocation, state preparation, artifact capture, and ownership first. Then each new device contributes a distinct comparison instead of another place for the same lab defect to appear. That discipline is what makes Android OEM compatibility testing credible to an engineer deciding whether the build can ship.