Trustworthy retry numbers in parallel Selenium tests

Two Selenium tests fail together in CI, then both report attempt 2. One actually failed before Chrome started; the other reused a browser that belonged to another worker. The counter looks convincing, but it has hidden the sequence you need to debug.

Why attempt numbers become unreliable in parallel runs

Selenium does not decide whether a failed test should run again. That policy belongs to your test runner, an extension, or a wrapper in your framework. The counter sits inside that lifecycle, so its scope matters more than the choice of integer class.

A single static number produces a suite-wide sequence. Checkout might receive attempt 17 because sixteen unrelated tests ran first. A counter created inside the retry callback has the opposite problem: every callback starts at 1. ThreadLocal is also a poor default for this job. It identifies a worker thread, not a logical test, and a runner can put a later attempt on a different worker.

Start by naming the unit being counted. A useful identity represents one invocation, not merely a Java method. It normally includes the build, shard, test case, parameter set, and repetition when the same case is scheduled more than once. For example, checkout-card-visa/build-4812/shard-2 is specific enough to distinguish two parameterized checkout tests. Generate that ID once, before the first attempt, and pass it unchanged to every retry.

Use the terms consistently too. The first execution is attempt 1. If the policy allows two retries, the maximum is three attempts. Logs that call the first failure retry 0 and the next execution attempt 1 force every dashboard query to interpret an undocumented offset.

The counter should be advanced at the start of an attempt, before browser creation. A session creation failure is still an attempt: the worker accepted the job, consumed time, and returned a result. If the increment happens after new ChromeDriver(), those failures disappear from the history.

What AtomicInteger guarantees, and what it cannot protect

incrementAndGet() performs the read, increment, and write as one atomic operation and returns the updated value. Two threads using the same AtomicInteger instance will not both receive the same result from that operation. That is the narrow guarantee we need for allocating an ordinal.

The surrounding lookup must also be safe. A ConcurrentHashMap keyed by logical run ID can create one counter per invocation with computeIfAbsent(). A plain HashMap plus AtomicInteger is still unsafe because concurrent map access is outside the counter.

Several important things remain unprotected:

• AtomicInteger does not make a WebDriver instance safe to use from multiple threads.
• It does not make a compound action such as “increment, write JSON, take screenshot” atomic.
• It does not persist a number across JVM restarts.
• It does not coordinate separate CI processes or containers.
• It does not prevent two framework layers from each scheduling their own retry.

That last case creates confusing multiplication. If a runner retries twice and a custom wrapper also allows two retries, one logical failure can execute up to nine times. Pick one owner for scheduling. Let the tracker observe that owner rather than adding another retry loop behind its back.

Browser ownership is a separate concern. Selenium’s Java-only ThreadGuard wrapper detects calls made from a thread other than the one that created the driver. It is useful as a tripwire, but Selenium’s documentation is explicit that ThreadGuard does not replace a proper driver-lifecycle design. A local driver variable, created and quit on the worker executing the attempt, is simpler than trying to make a shared driver safe.

A runnable Java example with an isolated browser per attempt

The following class can run inside a Java project that already includes the Selenium Java binding. Selenium Manager handles the local driver discovery used by new ChromeDriver(). The example launches two logical tests in parallel, gives each one a unique run ID, and creates a fresh browser for every attempt.

import java.util.List;
import java.util.UUID;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ThreadGuard;

public final class SeleniumRetryExample {
    private static final int MAX_ATTEMPTS = 3;
    private static final AttemptTracker ATTEMPTS = new AttemptTracker();
    private static final String PAGE =
            "data:text/html;charset=utf-8,"
            + "%3Ch1%20id%3D%22status%22%3EOrder%20confirmed%3C%2Fh1%3E";

    public static void main(String[] args) throws Exception {
        ExecutorService workers = Executors.newFixedThreadPool(2);

        try {
            List<Callable<Void>> cases = List.of(
                    caseTask("checkout-" + UUID.randomUUID()),
                    caseTask("profile-" + UUID.randomUUID()));

            for (Future<Void> result : workers.invokeAll(cases)) {
                result.get();
            }
        } finally {
            workers.shutdown();
        }
    }

    private static Callable<Void> caseTask(String testRunId) {
        return () -> {
            runWithRetries(testRunId, MAX_ATTEMPTS, driver -> {
                driver.get(PAGE);
                String actual = driver.findElement(By.id("status")).getText();

                if (!"Order confirmed".equals(actual)) {
                    throw new AssertionError("Unexpected status: " + actual);
                }
            });
            return null;
        };
    }

    private static void runWithRetries(
            String testRunId,
            int maxAttempts,
            Consumer<WebDriver> test) {
        if (maxAttempts < 1) {
            throw new IllegalArgumentException("maxAttempts must be at least 1");
        }

        try {
            for (int index = 0; index < maxAttempts; index++) {
                int attempt = ATTEMPTS.next(testRunId);
                RemoteWebDriver sessionOwner = null;
                WebDriver driver = null;
                String sessionId = "not-created";
                long started = System.nanoTime();

                try {
                    sessionOwner = new ChromeDriver();
                    driver = ThreadGuard.protect(sessionOwner);
                    sessionId = sessionOwner.getSessionId().toString();

                    test.accept(driver);
                    record(testRunId, attempt, sessionId, "passed",
                            "none", started);
                    return;
                } catch (RuntimeException | AssertionError failure) {
                    record(testRunId, attempt, sessionId, "failed",
                            describe(failure), started);

                    if (index + 1 == maxAttempts) {
                        if (failure instanceof RuntimeException) {
                            throw (RuntimeException) failure;
                        }
                        throw (AssertionError) failure;
                    }
                } finally {
                    if (driver != null) {
                        try {
                            driver.quit();
                        } catch (RuntimeException cleanupFailure) {
                            System.err.printf(
                                    "testRunId=%s attempt=%d cleanupFailure=%s%n",
                                    testRunId, attempt, describe(cleanupFailure));
                        }
                    }
                }
            }

            throw new IllegalStateException("Retry loop ended unexpectedly");
        } finally {
            ATTEMPTS.clear(testRunId);
        }
    }

    private static void record(
            String testRunId,
            int attempt,
            String sessionId,
            String result,
            String cause,
            long started) {
        long durationMs = (System.nanoTime() - started) / 1_000_000;

        System.out.printf(
                "testRunId=%s attempt=%d thread=%s session=%s "
                + "result=%s durationMs=%d cause=%s%n",
                testRunId,
                attempt,
                Thread.currentThread().getName(),
                sessionId,
                result,
                durationMs,
                cause);
    }

    private static String describe(Throwable failure) {
        String message = failure.getMessage() == null
                ? ""
                : failure.getMessage().replace('\n', ' ');
        return failure.getClass().getSimpleName() + ":" + message;
    }

    private static final class AttemptTracker {
        private final ConcurrentHashMap<String, AtomicInteger> counters =
                new ConcurrentHashMap<>();

        int next(String testRunId) {
            return counters
                    .computeIfAbsent(testRunId, ignored -> new AtomicInteger())
                    .incrementAndGet();
        }

        void clear(String testRunId) {
            counters.remove(testRunId);
        }
    }
}

The test uses a data URL so it does not depend on a public site. With the assertion shown, each case passes on attempt 1. To exercise the failure path, temporarily change the expected text, run the class, and confirm that each logical ID produces attempts 1, 2, and 3 with three different session IDs.

In a real suite, place the tracker at the same lifetime as the test worker or retry extension, not inside the test method. Keep testRunId stable across its attempts. Clear it only after the final result. The example catches both Selenium runtime failures and AssertionError because either can make an attempt fail, but your framework may classify assertion, infrastructure, and cleanup failures differently.

The fresh browser is intentional. Selenium recommends a new WebDriver instance per test to keep state isolated. Here, every retry is treated as another independent execution, so cookies, local storage, open tabs, and a broken session do not cross the boundary.

How to prove the counter is the problem

Capture one structured line for every attempt. At minimum it needs the logical run ID, allocated attempt, process or shard, thread, browser session ID, result, duration, and original exception type. Screenshots and page source should use the same ID and attempt in their filenames.

Run the existing parallel suite without changing its concurrency, and retain the console:

mvn test 2>&1 | tee retry-run.log
grep 'testRunId=checkout-' retry-run.log

Use your repository’s normal command if it is not Maven. The important part is preserving a complete run and filtering by the stable identity, not by the test method name alone.

Read the records in allocation order and ask specific questions:

1. Does one logical ID have exactly one record for each attempt number?
2. Does each attempt have a different nonempty session ID?
3. Does a failed browser startup say session=not-created rather than vanish?
4. Does the first failure retain its exception, even if a later attempt passes?
5. Does the counter disappear only after the terminal result?

Duplicate numbers inside one JVM point to separate tracker instances, premature removal, or a value logged from some other retry index. Duplicate attempt 1 across different JVMs is expected because AtomicInteger has process scope. Add process and shard fields before treating that as a defect.

A gap is not proof that incrementAndGet() lost an update. The process may have incremented and then crashed, the task may have been cancelled, or the log sink may have dropped the record. Compare the application log with the runner’s lifecycle events.

Repeated session IDs are stronger evidence. They mean the same browser was reused or the session field was copied from stale context. If ThreadGuard reports access from a different thread, fix driver ownership first. Do not add synchronization around WebDriver and call it isolated.

For Grid runs, filter the Grid event log or trace on session.id. Check that every reported browser session has a session-creation path and a matching DELETE at teardown. Grid traces identify WebDriver traffic, while your attempt log supplies the logical test ID, so keep both artifacts for the same build.

The reliable fix and what it costs

Make one component responsible for the retry budget. Give that component a stable run ID and a per-run AtomicInteger stored in a concurrent map. Allocate the number before any setup that can fail, emit the attempt record once, and remove the entry after the terminal outcome. Create and quit the browser on the worker that executes that attempt.

This design costs browser startup time and Grid capacity. A fresh session per retry loses warm caches and authenticated state, and a burst of retries can occupy slots needed by healthy tests. Limit retries to failures you have deliberately classified as retryable. Back pressure at the runner or CI job is safer than allowing a failure storm to double the suite’s concurrency.

Structured evidence also costs storage. Three page sources, screenshots, command logs, and Grid traces can be large. Keep the small attempt record for every run, but set a retention policy for heavy artifacts. Preserve all artifacts from the first failure and final result at minimum.

There is an operational trade-off around cleanup failures. The sample records quit failures separately so they do not overwrite the test’s original exception. Your CI policy may still mark that invocation as an infrastructure failure, especially when leaked sessions exhaust Grid slots. What matters is retaining both outcomes instead of letting a finally block erase the first cause.

If global numbering across machines is truly required, use a durable coordinator that supports atomic increments. That adds network latency, availability concerns, and cleanup rules. Most teams do not need it. A compound identity such as build, shard, process, logical run, and local attempt is cheaper and tells a clearer story.

When AtomicInteger is the wrong tool

Do not add a second counter when the test runner already exposes a reliable attempt or invocation number. Duplicate sources drift, especially when retry settings change. Enrich the runner’s record with session and shard identity instead.

Avoid whole-test retries when the real need is waiting for a page condition. A WebDriverWait polls one expected condition within the same attempt. Restarting the browser because an element needs 500 milliseconds confuses synchronization with flakiness and makes the suite slower.

A process-local atomic value is also unsuitable when the number must survive a JVM crash, coordinate containers, or support billing and compliance. Those requirements need durable storage and idempotent event handling, not an in-memory counter.

Never use the counter as permission to share a driver. Locks can serialize calls, but they cannot undo cookies, navigation, alerts, windows, or storage left by the previous attempt. ThreadGuard can expose accidental cross-thread use, yet the real correction is exclusive ownership.

Finally, do not retry destructive scenarios until the product operation is known to be safe to repeat. Payment submission, account deletion, and one-time token redemption can succeed on the server even when the browser times out. An accurate attempt number does not make the business action idempotent. Check server-side state first, then decide whether another attempt is legitimate.