Find the broken handoff in an Appium session

Your Android smoke test never reaches its first locator, yet the report calls it an application launch failure. The same APK and emulator work on a laptop, while CI returns a 404, reports that no suitable driver is available, or waits until the client gives up. Those outcomes belong to different handoffs, and changing capabilities at random makes all three harder to diagnose.

Follow the command through the actual stack

An Appium test starts outside the phone. Your test code calls an Appium client library, and that library encodes a WebDriver request over HTTP. The Appium server receives the request and selects a platform driver for a new session. That driver then uses platform-specific tools to operate the target. For Android with UiAutomator2, those tools include the Android SDK, ADB, code running on the device, and Google's automation technology.

That sequence matters because the word "driver" hides several very different responsibilities. The Python client object is not the UiAutomator2 driver. The Appium server is not ADB. UiAutomator2 is not bundled merely because an appium executable is present. A connected emulator is not proof that the server loaded the requested extension. Treating the stack as one black box turns every setup fault into "Appium is flaky."

The Appium architecture guide describes a client-server design. The client can run on a different machine from the server because the boundary is a network protocol. That is useful for device farms, but it introduces ordinary network and routing failures before any mobile automation begins. A DNS failure, refused TCP connection, proxy rewrite, TLS problem, or wrong HTTP path can prevent the platform driver from seeing a command.

Once a session request reaches Appium, the server interprets the requested capabilities. platformName is a standard WebDriver capability. Values specific to Appium use the appium: vendor namespace on the wire, including appium:automationName and appium:udid. The official Python client can add that prefix when its options classes are used, which is why readable Python code can contain automationName while the transmitted payload contains its namespaced form. Do not "fix" working options code by adding a second, conflicting copy of every value.

The automation name is a selection instruction. UiAutomator2 tells Appium which installed driver should handle an Android session. It does not download that driver during a session, authorize a USB device, start an emulator, or identify an application package. Each of those prerequisites has its own evidence. If the server cannot select the requested driver, the request has already passed the client and HTTP routing layers but has not reached application startup.

Drivers add another level of architecture behind the server. Appium's driver introduction explains that a driver maps WebDriver commands to whatever technology its platform requires. Some commands may be implemented in the Node.js portion of a driver, while others may be proxied to another WebDriver-speaking component. XCUITest, for example, involves WebDriverAgent on the iOS side. A server log can therefore show that Appium accepted a command even when the later failure belongs to the driver, a platform helper, Xcode, ADB, or the operating system.

The practical debugging rule is to find the last boundary that definitely worked. A client stack trace proves only what happened in the test process. An HTTP response proves a server answered at that address and path. An installed-driver listing proves an extension is registered in that Appium environment. adb devices proves what ADB can currently see. A returned session identifier proves session creation. A successful element query in a known fixture proves the driver can reach that screen. None of these facts automatically proves the next one.

This is also why a green /status request is deliberately modest evidence. It says the server route answered. It does not create a session, choose UiAutomator2, reserve a device, install an application, or execute a locator. Keep that distinction visible in logs and test names. Calling the status check mobile_ready invites the next engineer to trust a claim the check never made.

Read the failure at the boundary where it occurs

Begin with the earliest artifact for the same attempt. Use one run identifier in the client output and server-log filename. Record the complete server URL after environment variables have been resolved. Save the Appium version, installed-driver list, doctor result, and device list before creating a product session. These are small artifacts, and they answer different questions.

The first diagnostic split is whether the client produced an HTTP request. A Python import error, an unsupported constructor argument, or an exception while building options occurs before Appium can respond. The useful lines are the first frame in your own test code and the exception type. Changing the server base path cannot repair a local TypeError. Conversely, a WebDriver error response with an HTTP status and server-side log entry means the request crossed the client boundary.

Absence from a server log is evidence only after you verify you are reading the right log at a level that records requests. Teams often tail a laptop server while CI points to a remote grid, or collect yesterday's container log after a pod replacement. Print the resolved origin and base path in the client job. On the server side, print the address and port at startup. Align timestamps or a request identifier where your environment supports one. If those facts disagree, stop diagnosing the emulator.

Routing failures should be tested without allocating a device. Appium's current default server base path is the root. A client copied from an Appium 1 setup may still send traffic below /wd/hub. The server can also be intentionally started with --base-path=/wd/hub for compatibility. Neither choice is inherently broken. The failure appears when the client and server choose different paths.

This script asks both likely status URLs and prints the HTTP status code. It does not label either result "device ready." A 200 on one path and 404 on the other gives you a routing answer without starting a mobile session. A 000 from curl means it did not receive an HTTP response, so investigate reachability before interpreting Appium routes.

#!/usr/bin/env bash
set -euo pipefail

server_origin="${APPIUM_SERVER_ORIGIN:-http://127.0.0.1:4723}"

printf 'Appium version: '
appium --version

appium driver list --installed
if ! appium driver doctor uiautomator2; then
  printf 'UiAutomator2 doctor check failed\n' >&2
fi
if ! adb devices -l; then
  printf 'ADB device discovery failed\n' >&2
fi

for base_path in "" "/wd/hub"; do
  status_url="${server_origin}${base_path}/status"
  http_code="$(curl \
    --silent \
    --output /dev/null \
    --write-out '%{http_code}' \
    "${status_url}" || true)"
  printf '%s -> HTTP %s\n' "${status_url}" "${http_code}"
done

Run those commands under the same operating-system user, container, and environment variables as the Appium process. Appium stores separately installed extensions under its configured Appium home. A driver listing from your interactive shell can be irrelevant if the service starts with another home directory or another user account. The reliable comparison is the listing captured beside the exact server launch.

Server startup output is valuable here. The official installation guide notes that Appium lists the valid connection URLs and available drivers when it starts. Preserve those lines. If UiAutomator2 appears in an earlier installation job but not in the server startup log, compare the executable path, Appium home, user, and container layer. Reinstalling into the same wrong shell does not change the service.

The Extension CLI exposes appium driver list --installed and appium driver doctor uiautomator2. The first answers whether Appium knows about the extension. The second runs health checks supplied by that driver. A clean doctor result is not a session oracle. It does not prove that a particular device is allocated, unlocked, online, or free from another worker at the moment the session starts.

Device evidence comes next. Appium's Android quickstart uses adb devices to verify that an emulator or physical device is connected. Read the state column, not only the serial. A line that contains the requested identifier but does not have the device state is not a connected target. Even the device state is limited evidence: Android's ADB documentation cautions that the operating system may still be booting. An empty list points toward emulator startup, USB access, container device mapping, or ADB-server ownership rather than Appium routing.

Multiple usable devices create a subtler problem. A generic deviceName is not a reservation system. In a shared runner, select the allocated target by its unique identifier and pass that contract through the job. Then compare the requested identifier with the device-pool lease and the server log for that session. If two workers receive the same identifier, capability retries merely increase contention.

After session creation, save the returned session ID and negotiated capabilities, with secrets removed. Negotiated capabilities show what the server and driver accepted, not merely what a helper intended to send. If your source names one application package while the response names another, the response is the evidence to investigate. If no session ID was returned, do not describe later element or application assertions as having run.

Read those returned names carefully, because they are not spelled the way the request was. The appium: prefix is a request-side vendor namespace, and the server strips it while matching, so the response carries the matched values under plain names such as udid and automationName. Code that filters or looks up returned capabilities by their prefixed spelling silently finds nothing and produces an evidence file that looks empty rather than wrong. Matching capabilities are a second trap in the other direction: platformName is negotiated before the session exists, so the response can only ever echo a value the server already agreed to. It is useful for the record and useless as an assertion.

Logs need restraint. Capability values can contain application locations, cloud-provider credentials, device identifiers, and account-related metadata. Store the server log and client stack in restricted CI artifacts, redact secrets before sharing them, and avoid dumping an entire environment. A useful diagnostic bundle identifies the boundary without turning a mobile test failure into a credential incident.

Work through three failures that look like one

Consider a migration where local tests use http://127.0.0.1:4723 but an old CI variable still ends with /wd/hub. The APK is identical and the emulator is online. CI fails before a session is allocated because its request path does not match the server's root base path. An engineer who edits appium:appPackage sees no improvement because the server has not reached application targeting.

The distinguishing evidence is a successful root status check, a failing legacy-path status check, and the resolved CI URL. The repair is to change the client URL to the root. If other consumers still require the legacy convention, start the server with the documented --base-path=/wd/hub option and keep those consumers consistent. The second choice carries a compatibility cost: every health check, reverse proxy rule, client, and runbook must include that prefix. The root path is simpler for a new environment, but changing a shared endpoint can break old clients, so roll it out as an interface change.

Connection refusal is the near-miss. It may appear beside a route error in a generic "session not created" report, but curl prints no received HTTP status and the server log has no matching request. A base-path edit will not start a dead process or open a firewall. Check whether Appium is listening on the address visible to the client. In containers, 127.0.0.1 refers to the current network namespace, not automatically to a neighboring container or host service.

The following launcher keeps the server and client on the root path. It waits for the status endpoint before running the Python probe shown later, records the server output, and always stops the local server. The loop is readiness polling, not a claim that the device is ready.

#!/usr/bin/env bash
set -euo pipefail

server_log="${APPIUM_LOG_FILE:-appium-server.log}"
server_url="http://127.0.0.1:4723"

appium \
  --address 127.0.0.1 \
  --port 4723 \
  >"${server_log}" 2>&1 &
appium_pid=$!

cleanup() {
  kill "${appium_pid}" 2>/dev/null || true
  wait "${appium_pid}" 2>/dev/null || true
}
trap cleanup EXIT

server_ready=false
for attempt in 1 2 3 4 5 6 7 8 9 10; do
  if ! kill -0 "${appium_pid}" 2>/dev/null; then
    sed -n '1,200p' "${server_log}"
    exit 1
  fi
  if curl --fail --silent "${server_url}/status" >/dev/null; then
    server_ready=true
    break
  fi
  sleep 1
done

if [[ "${server_ready}" != "true" ]]; then
  sed -n '1,200p' "${server_log}"
  exit 1
fi

APPIUM_SERVER_URL="${server_url}" \
  python tests/appium_architecture_probe.py

Now consider a clean runner image where appium --version succeeds and /status returns successfully, but UiAutomator2 is absent from appium driver list --installed. That state is valid for the core server: Appium does not bundle every platform driver. The new-session request reaches Appium, but the requested automation name cannot select an installed extension.

Install the reviewed driver, with the server stopped, as part of image creation or the project's controlled dependency setup. Then verify the resolved installation in the runtime image. The official Extension CLI accepts appium driver install uiautomator2. Do not make an unreviewed "latest" download the first step of every test run if reproducibility matters. A driver and server can release independently, so an unnoticed driver update can change a previously stable image even when the Appium server version stays fixed.

The common near-miss is a driver that is installed, but not in the environment used by the server. One shell writes extensions below one Appium home, while a service account or container starts from another. Both logs can honestly say installation succeeded. Only the runtime server's list tells you what it loaded. Fix the image or environment ownership; repeated session attempts cannot bridge two filesystems.

There is a real maintenance trade-off. Baking the driver into an image makes runs repeatable and avoids downloading an extension while a device waits. It also means the team must rebuild the image to take a reviewed driver update. Installing on every job makes changes arrive sooner, but adds network dependency and increases variation unless the installation is locked by project inputs. Choose one ownership model and record both the server and driver versions.

The third case begins with a healthy root status response and a visible UiAutomator2 installation. The session still fails during device preparation. The allocated identifier may be absent or not connected, or ADB may show the device state while Android is still booting. A physical device may not have authorized the runner, a container may lack access to the USB device, or another ADB server may own the connection. Those causes live after driver selection but before a reliable application interaction.

Do not hide this state behind deviceName: Android and hope the driver selects correctly. Pass the unique identifier assigned by the pool. Fail the job before session creation if that identifier is absent or lacks the device state. If it is connected but Android is still booting, the session probe supplies the next piece of evidence. This is a genuine oracle because changing the allocated device, its authorization, or its ADB state changes the result. A hard-coded fixture that merely asserts its own string would prove nothing.

A wrong app target is the next near-miss. The device can be online and the session machinery can begin, but the requested package or activity may not exist in that build. Compare the exact package and activity in the final capabilities with the application installed on that device. Do not change the driver installation because a renamed application ID fails to launch. Conversely, do not blame the APK when the requested device never appeared in ADB.

These examples often collapse into the same top-level CI label because frameworks report setup failure before any test method runs. Preserve the nested exception and server log rather than reporting only "before all failed." The useful question is not whether the test case started. It is whether an HTTP route answered, a driver was selected, a device was usable, and a known application screen became queryable.

Build a probe that can prove each handoff

A boundary probe should be smaller than a business smoke test. Use a stable application that the runner already owns, create one session, locate one known element, and quit in a finally path. It should not sign in, seed data, call a changing backend, accept marketing prompts, or depend on a production account. Every additional dependency creates another explanation for failure.

Android's built-in Settings application is useful for a runner check when the device image and language are controlled. Appium's official Python quickstart uses com.android.settings, .Settings, and the visible "Apps" item. The example below follows that documented shape and adds an explicit device identifier from the CI allocation. It fails before making a request if the pool did not supply that identifier.

import json
import os
import unittest
from pathlib import Path

from appium import webdriver
from appium.options.android import UiAutomator2Options
from appium.webdriver.common.appiumby import AppiumBy


class AppiumArchitectureProbe(unittest.TestCase):
    def setUp(self) -> None:
        server_url = os.environ.get(
            "APPIUM_SERVER_URL",
            "http://127.0.0.1:4723",
        )
        device_udid = os.environ.get("ANDROID_UDID")
        if not device_udid:
            raise RuntimeError("ANDROID_UDID must identify the allocated device")

        capabilities = {
            "platformName": "Android",
            "automationName": "UiAutomator2",
            "deviceName": "Android",
            "udid": device_udid,
            "appPackage": "com.android.settings",
            "appActivity": ".Settings",
            "language": "en",
            "locale": "US",
        }
        options = UiAutomator2Options().load_capabilities(capabilities)
        self.driver = webdriver.Remote(server_url, options=options)
        self.addCleanup(self.driver.quit)

    def test_known_screen_is_reachable(self) -> None:
        evidence_dir = Path(
            os.environ.get("PROBE_EVIDENCE_DIR", "artifacts/appium-probe")
        )
        evidence_dir.mkdir(parents=True, exist_ok=True)
        (evidence_dir / "negotiated-capabilities.json").write_text(
            json.dumps(
                {
                    "sessionId": self.driver.session_id,
                    "capabilities": self.driver.capabilities,
                },
                indent=2,
                default=str,
            ),
            encoding="utf-8",
        )

        apps_item = self.driver.find_element(
            by=AppiumBy.XPATH,
            value='//*[@text="Apps"]',
        )
        self.assertTrue(apps_item.is_displayed())


if __name__ == "__main__":
    unittest.main()

The probe carries exactly one assertion, and that is deliberate. If session creation does not complete, webdriver.Remote raises inside setUp and the test method never runs, so that boundary is already decided before any assertion could speak to it. A missing, renamed, hidden, or translated Settings item fails the UI check, which is the only oracle in the method. The probe does not assert that the requested UDID equals the same hard-coded UDID, and it does not claim to validate your product.

An earlier version of this probe also asserted that the negotiated platformName was Android, and that check was worth removing rather than keeping. platformName is a W3C matching capability. If the server cannot match the requested value, it never creates the session, so setUp raises and the assertion is unreachable. If a session does exist, the server returns the value it matched, so the assertion compares the request with its own echo. Both branches were decided before the line executed. It re-tested the precondition that this article has already assigned to session creation, while reading like independent coverage of a negotiated platform.

The capabilities are still worth having, just as evidence rather than as an oracle. Writing them beside the run gives a reviewer the session ID, the driver's reported values, and anything the driver added during session creation, which is what you want when the UI assertion fails and the question becomes whether the probe reached the intended device at all. Treat that file the same way as the server log, because capability values can carry device identifiers and provider credentials. The distinction to hold onto is that recording a fact and asserting a fact are different jobs, and a comparison that cannot come out two ways belongs in the first category.

There are limits to the fixture. Device manufacturers can customize Settings, and translations change visible text. If your pool contains mixed images, own a tiny fixture application instead. Give it a stable package, launch activity, accessibility identifier, and static screen. Version that fixture alongside the runner image. This costs engineering time, but it removes operating-system UI variation and gives the team a target it can change deliberately.

Use the final capabilities and server log to interpret where the probe stopped. If webdriver.Remote raises before assigning self.driver, no UI assertion ran. If the session exists but the element lookup fails, routing and driver selection have already succeeded. At that point inspect the foreground app, locale, page source, and fixture version. Do not reinstall UiAutomator2 merely because a known text label changed.

Quit belongs in registered test cleanup because a failed assertion still owns a session. Python's documented addCleanup hook runs after the test outcome is recorded, so the returned driver is closed even when the UI assertion fails. A leaked session can keep a real device busy and make the next worker appear broken. Cleanup itself should be logged, but a cleanup error must not erase the original failure. Most test runners retain chained exceptions; if yours does not, attach the quit failure as secondary evidence and preserve the first exception as the result.

Keep product flows separate from this probe. Once the known screen is reachable, the architecture contract has enough evidence to release the device to the real suite or proceed on the same allocated device. A login failure after that point belongs to product state, test data, synchronization, authentication, or a locator until evidence shows otherwise. The probe prevents teams from debugging those layers while the server route itself is wrong.

Negative testing also needs discipline. You do not need to uninstall a driver on a shared runner to prove the "missing driver" branch. Validate that the expected name is present and test the failure path in a disposable image used for infrastructure tests. Do not send deliberately malformed capabilities to a production device cloud unless its contract permits that traffic. A diagnostic exercise should not evict other teams' sessions.

Roll the checks into an existing CI suite

Start the rollout in observation mode. Capture the resolved server URL, Appium version, installed drivers, doctor output, allocated device identifier, and ADB listing on a small set of jobs. Do not change retries or capability helpers during the first comparison. This creates a baseline from real executions without inventing timing figures or assuming which boundary fails most often.

Next, centralize the server URL and options factory. Search for hard-coded /wd/hub suffixes, duplicate automationName values, and jobs that start Appium differently. Keep one source of truth for the base path. If a proxy adds a prefix, name it explicitly in configuration and probe its status URL. Hidden string concatenation is how one shard reaches the root while another reaches a legacy route.

Move extension ownership into the runner image or a locked project setup. Capture appium driver list --installed after the image is built and again at runtime. The build check proves the artifact contained the driver. The runtime check catches a different Appium home, a broken mount, or a service account mismatch. Neither check requires a device.

Add the device gate only on jobs that have received a lease. The pool should hand the job a unique identifier, and the job should verify that exact value in ADB immediately before the session probe. Release the lease in an unconditional cleanup step. If your provider exposes devices through a remote API rather than local ADB, use the provider's supported allocation evidence instead of pretending a local command can see a remote phone.

The workflow below assumes a self-hosted runner image labelled android already contains Node.js, Appium, the reviewed UiAutomator2 driver, Python, the Appium Python client, ADB, and the probe file. It intentionally verifies those image promises instead of downloading new versions during the test. The background server lives only for the probe step, and its log is uploaded even when the probe fails.

name: appium-boundary-probe

on:
  workflow_dispatch:
  pull_request:

jobs:
  android-probe:
    runs-on: [self-hosted, android]
    timeout-minutes: 15
    env:
      APPIUM_SERVER_URL: http://127.0.0.1:4723
      ANDROID_UDID: ${{ vars.CI_ANDROID_UDID }}

    steps:
      - uses: actions/checkout@v4

      - name: Verify runner contract
        shell: bash
        run: |
          set -euo pipefail
          appium --version
          appium driver list --installed
          appium driver doctor uiautomator2
          adb devices -l
          if [[ -z "${ANDROID_UDID}" ]]; then
            printf 'CI_ANDROID_UDID is not configured\n' >&2
            exit 1
          fi
          device_state="$(
            adb devices | awk -v serial="${ANDROID_UDID}" \
              '$1 == serial { print $2; exit }'
          )"
          if [[ "${device_state}" != "device" ]]; then
            printf 'Allocated device %s has state %s\n' \
              "${ANDROID_UDID}" "${device_state:-missing}" >&2
            exit 1
          fi

      - name: Run architecture probe
        shell: bash
        run: |
          set -euo pipefail
          appium --address 127.0.0.1 --port 4723 \
            >appium-server.log 2>&1 &
          appium_pid=$!

          cleanup() {
            kill "${appium_pid}" 2>/dev/null || true
            wait "${appium_pid}" 2>/dev/null || true
          }
          trap cleanup EXIT

          for attempt in 1 2 3 4 5 6 7 8 9 10; do
            if ! kill -0 "${appium_pid}" 2>/dev/null; then
              sed -n '1,200p' appium-server.log
              exit 1
            fi
            if curl --fail --silent \
              "${APPIUM_SERVER_URL}/status" >/dev/null; then
              python tests/appium_architecture_probe.py
              exit
            fi
            sleep 1
          done

          sed -n '1,200p' appium-server.log
          exit 1

      - name: Preserve Appium server log
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: appium-server-log
          path: appium-server.log
          if-no-files-found: warn

That job has an explicit environmental contract. A hosted runner without Android tooling cannot satisfy it merely by adding the android label to YAML. Treat the image definition and pool lease as dependencies owned by the infrastructure team. If the driver is installed during image creation, record the source lock or reviewed version there, not in prose copied between test repositories.

Canary the probe on one runner class before making it a required gate. Then enable it for every distinct image, server route, or device-provider integration, rather than blindly before every individual test. Running one extra session before every test method consumes device capacity and creates more setup churn. The probe earns its place when it protects a boundary that could otherwise invalidate a whole shard.

Remove broad session retries only after the new evidence is available. Classify retryable conditions narrowly, preserve each attempt, and keep the original exception. A retry may be reasonable when a device provider explicitly reports a temporary allocation condition. It is not reasonable for a deterministic 404, an absent installed driver, an invalid capability set, or an unauthorized USB device. Those states need a configuration change.

For an Appium 1 migration, change one axis at a time. First align the URL and confirm status. Then verify separately installed drivers. Next migrate capability construction through the supported client options class. Finally run the known-screen probe. Upgrading the client, server, driver, Java toolchain, emulator image, and application in one commit leaves too many plausible causes when the first session fails.

Know the costs and when to stop using the probe

Every diagnostic layer adds work. The CLI checks add startup steps and artifact volume. Doctor can inspect prerequisites but cannot reserve a device. A session probe occupies a device and starts an application. A dedicated fixture app needs maintenance. Server logs can expose sensitive context and require retention rules. These costs are justified when they turn an opaque suite-wide failure into an owned handoff, but they are not free.

Do not run the architecture probe before every test if all tests share one already-created session. In that design, the probe would add another session without checking the ownership model that matters. Put assertions around the shared session lifecycle instead: creation, exclusive allocation, cleanup, and the first safe command. Be honest that a shared session also couples tests and can preserve state.

Do not use the probe to certify product behavior. Reaching Android Settings says nothing about your APK, backend, account, feature flags, or release candidate. Reaching a team-owned fixture proves more about the device automation path, but still says nothing about a checkout or login flow. Keep product smoke tests because they answer a different question.

Stop investigating architecture once the failing command is clearly inside an established session. If the server log shows a session ID, the known fixture was reachable, and the next product locator cannot find an element, inspect the current screen, context, locator, wait condition, and app build. The architecture can still fail later, especially through platform helpers, but you need command-level evidence before restarting the server or reinstalling a driver.

Avoid using --relaxed-security, broad insecure-feature flags, or disabled TLS checks as generic troubleshooting fixes. A permission error may be accurately protecting the server. Read the specific feature's documentation and grant only what the test requires. Making a remote device server more permissive can turn a test problem into an infrastructure security problem.

Do not clear Appium's extension home or uninstall drivers on a shared host during diagnosis. That is destructive state shared by other sessions. Reproduce the missing-driver case in a disposable image, or compare the installed list with the image manifest. If the production runner is wrong, replace or repair it through the runner's controlled provisioning path.

Avoid a base-path compatibility switch when the real failure is network reachability. A 000 curl result, name-resolution error, connection refusal, or TLS handshake failure happens before Appium can choose a route. Likewise, do not debug the network after a well-formed server response rejects a capability. The response demonstrates that the network carried the request far enough to receive an application-layer answer.

Cloud providers change the available evidence. You may not control the Appium executable, inspect its extension home, run ADB, or see the raw server startup log. In that case, use the provider's documented session log, device allocation record, and supported capabilities. Do not paste local Extension CLI commands into a cloud job and treat their output as proof about the remote server.

Finally, do not turn timestamps into performance claims. Readiness polling can tell you that a server did or did not answer within the job's chosen limit. It does not establish an Appium performance benchmark, and one run does not justify a new timeout for every environment. If startup duration matters, design a separate measurement with a defined population, clock boundary, and retained results.