Grid says the relay node is healthy while your vendor runs a different browser

A platform team consolidated four test pipelines behind one Grid URL by putting relay nodes in front of a cloud vendor. It worked immediately. Every suite pointed at https://grid.internal:4444, the vendor account stayed hidden behind the relay, and the Grid UI showed a tidy row of nodes advertising Safari, Chrome and Edge. Three weeks later a rendering defect shipped that the Safari suite should have caught, and the post-mortem found that not one session in those three weeks had run on the Safari version the stereotype declared.

Nothing had broken. No node went down, no session failed, no alert fired. The relay node had advertised a browser and version, the Distributor had matched requests against that advertisement, the vendor had created sessions with something else, and no code anywhere in the path ever compared the two. The Grid was working exactly as documented. The team's mental model was not.

What a relay node actually promises

The relevant configuration is small enough to hold in your head. From the Grid CLI reference, the Relay section defines:

• --service-url, example http://localhost:4723, described as the "URL for connecting to the service that supports WebDriver commands like an Appium server or a cloud service."
• --service-host and --service-port (example 4723), as an alternative way of naming the same endpoint.
• --service-status-endpoint, example /status, described as "Optional, endpoint to query the WebDriver service status, an HTTP 200 response is expected."
• --service-protocol-version, default HTTP/1.1, described as "Optional, enforce a specific protocol version in HttpClient when communicating with the endpoint service status."
• --service-configuration, a string array, with the documented example max-sessions=2 stereotype='{"browserName": "safari", "platformName": "iOS", "appium:platformVersion": "14.5"}', described as "Configuration for the service where calls will be relayed to."

The TOML form, which the reference itself recommends for readability, looks like this:

# grid/relay-vendor-safari.toml
# A relay Node that advertises two remote combinations and forwards to one endpoint.

[node]
# Critical. Without this the Node also advertises whatever drivers exist on
# the host, and you get a node that is half real and half relayed.
detect-drivers = false
grid-url = "https://grid.internal:4444"

[relay]
url = "https://vendor.example.com/wd/hub"
status-endpoint = "/status"
protocol-version = "HTTP/1.1"
# configs is a flat list of alternating "slot count" and "stereotype JSON".
# The slot count is what THIS Grid believes it may run concurrently.
# It is not negotiated with the vendor.
configs = [
  "4", "{\"browserName\": \"safari\", \"browserVersion\": \"17\", \"platformName\": \"mac\"}",
  "8", "{\"browserName\": \"chrome\", \"browserVersion\": \"126\", \"platformName\": \"windows\"}"
]

[server]
port = 5556

Start it with the Event Bus connection supplied on the command line, because a relay node registers over the bus exactly like any other Node:

java -jar selenium-server-<version>.jar node \
  --config grid/relay-vendor-safari.toml \
  --publish-events tcp://selenium-hub.internal:4442 \
  --subscribe-events tcp://selenium-hub.internal:4443

Now read the three sentences that follow slowly, because the entire failure lives in them.

A stereotype is a matching template. The Distributor is documented as "taking any incoming new session requests and assigning them to a slot", and the CLI reference names --slot-matcher as the component "used to determine whether a Node can support a particular session." The stereotype is the input to that decision. It answers "should this request come to me". It does not travel onward as an enforceable instruction.

The health check is an HTTP 200. That is the documented contract for --service-status-endpoint, in those words. Any status page that returns 200 satisfies it. A vendor with a degraded region, a saturated queue, or an account that has hit its concurrency ceiling almost always still returns 200.

The vendor decides what it creates. The W3C WebDriver specification defines the New Session command as returning the capabilities of the session that was actually created. That response is the only authoritative statement in the entire chain about what browser you are driving, and in the failing team's setup nothing read it.

Put together: you declare an intention, Grid routes on that intention, the vendor honours it or does not, and by default nobody checks. Capability drift is not a bug in Grid or in the vendor. It is a missing assertion.

Worked example one: the version that quietly floated

The Safari case above unpicked like this.

The stereotype declared browserVersion: "17". The suite requested browserName: safari and, sensibly enough, did not repeat the version, on the reasonable theory that the Grid stereotype already pinned it. The Distributor matched on browser name and platform, the relay forwarded to the vendor, and the vendor created a session on its current default Safari because nothing in the forwarded request asked for a specific version.

Two ordinary decisions produced the gap. The stereotype was written to describe what the team wanted the pool to be, and the test was written to trust the pool. Neither is unreasonable in isolation. Together they mean the version exists only as a label on a node in a UI.

The diagnostic that would have caught it in five minutes:

#!/usr/bin/env bash
# relay-truth-check.sh
# Compares three views: what the relay advertises, what the vendor status says,
# and what a real session actually returns. Disagreement between 1 and 3 is the bug.
set -euo pipefail
ROUTER="${ROUTER:-https://grid.internal:4444}"
RELAY_NODE="${RELAY_NODE:-http://relay-safari.internal:5556}"
VENDOR_STATUS="${VENDOR_STATUS:-https://vendor.example.com/wd/hub/status}"

echo "== 1. What the relay node advertises to the Distributor =="
curl -sS --fail -X POST -H 'Content-Type: application/json' \
  --data '{"query":"{ nodesInfo { nodes { id uri status slotCount maxSession sessionCount stereotypes } } }"}' \
  "$ROUTER/graphql" | jq '.data.nodesInfo.nodes[] | select(.uri | contains("relay"))'

echo "== 2. The relay Node's own W3C status (does not touch the vendor's capacity) =="
curl -sS --fail "$RELAY_NODE/status" | jq '{ready: .value.ready, message: .value.message}'

echo "== 3. The endpoint Grid health-checks. Remember: any HTTP 200 satisfies it. =="
curl -sS -o /dev/null -w 'vendor status HTTP %{http_code} in %{time_total}s\n' "$VENDOR_STATUS"

echo "== 4. Ground truth: create one session and read the capabilities back =="
SESSION_JSON="$(curl -sS --fail -X POST -H 'Content-Type: application/json' \
  --data '{"capabilities":{"alwaysMatch":{"browserName":"safari","platformName":"mac"}}}' \
  "$ROUTER/session")"

echo "$SESSION_JSON" | jq '{
  sessionId: .value.sessionId,
  browserName: .value.capabilities.browserName,
  browserVersion: .value.capabilities.browserVersion,
  platformName: .value.capabilities.platformName
}'

SESSION_ID="$(echo "$SESSION_JSON" | jq -r '.value.sessionId')"
curl -sS --fail -X DELETE "$ROUTER/session/$SESSION_ID" > /dev/null
echo "session $SESSION_ID released"

Step 4 is the one that matters and it is the one nobody runs. sessionId and capabilities in the New Session response are specification-defined, as is DELETE /session/<session-id> for tearing it down, which the Grid endpoints documentation also lists. Everything else in the script is context; step 4 is evidence.

An illustrative comparison of what steps 1 and 4 produced, with values invented to show the shape of the disagreement rather than reported from any real vendor:

{
  "advertised_stereotype": {
    "browserName": "safari",
    "browserVersion": "17",
    "platformName": "mac"
  },
  "returned_capabilities": {
    "browserName": "safari",
    "browserVersion": "18.2",
    "platformName": "mac"
  },
  "verdict": "MATCH on browserName and platformName, DRIFT on browserVersion",
  "_note": "Illustrative values chosen to show the failure shape. Run the check against your own relay."
}

Reading that table is the whole skill. browserName matching is what let the request route. browserVersion drifting is what let the defect through. A pass or fail on the pair is not enough; you need to know which field disagreed, because the fix differs.

Making the assertion part of every session

The durable fix is not a script you run occasionally. It is a rule that every session in the suite is checked against, at creation, with the check owned by the fixture rather than by individual tests.

package dev.example.grid.relay;

import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Verifies that the session a relay node produced matches the contract the
 * suite asked for, and records the identifiers needed to find the vendor job.
 */
public final class RelayContract {

  private static final Logger LOG = LoggerFactory.getLogger("grid.relay.contract");

  /** Capability keys that must match exactly. Extend per pool, not per test. */
  private static final String[] ENFORCED = {"browserName", "browserVersion", "platformName"};

  private RelayContract() {}

  public static void verify(RemoteWebDriver driver, Capabilities expected, String ciRunId) {
    Capabilities actual = driver.getCapabilities();
    Map<String, String> drift = new LinkedHashMap<>();

    for (String key : ENFORCED) {
      Object want = expected.getCapability(key);
      if (want == null) {
        continue; // The suite did not pin it, so we do not police it.
      }
      Object got = actual.getCapability(key);
      if (!matches(String.valueOf(want), got)) {
        drift.put(key, "requested=" + want + " received=" + got);
      }
    }

    // One line that lets a human jump from a CI failure to the vendor console.
    LOG.info("relay session ci_run={} selenium_session={} browser={} version={} platform={} drift={}",
        ciRunId,
        driver.getSessionId(),
        actual.getBrowserName(),
        actual.getCapability("browserVersion"),
        actual.getPlatformName(),
        drift.isEmpty() ? "none" : drift);

    if (!drift.isEmpty()) {
      throw new IllegalStateException(
          "Relay capability drift. The vendor created a session that does not honour the "
              + "requested contract, so this run does not prove what it claims to prove. "
              + drift + " (selenium session " + driver.getSessionId() + ")");
    }
  }

  /**
   * Version comparison is prefix-based on purpose: requesting "126" should accept
   * "126.0.6478.127" but reject "127.0.1". Exact-match on full build strings is
   * unmaintainable, and ignoring version entirely is how this article started.
   */
  private static boolean matches(String want, Object got) {
    if (got == null) {
      return false;
    }
    String actual = String.valueOf(got);
    return Objects.equals(want, actual) || actual.startsWith(want + ".");
  }
}

Three choices in there are load-bearing and worth defending in review.

It only enforces keys the suite actually pinned. If a test asks for browserName: chrome and says nothing about version, policing the version would fail runs for a contract nobody agreed to. Silence means "any", and that is the correct default.

Version matching is prefix-based. Vendors return full build strings, and a suite that pins "126" genuinely wants any 126 build. Demanding exact equality on a full build string produces a fixture that breaks every time the vendor patches, which is how these checks get deleted three weeks after they are added.

The failure message says what the drift means, not just what it was. "Requested 17, received 18.2" is a fact. "This run does not prove what it claims to prove" is the reason someone should care, and it is what stops the fix being a quiet loosening of the assertion.

The correlation line is the other half of the value. It puts the CI run id and the Selenium session id in one place, which is the minimum needed to walk from a red pipeline to a vendor job record. If your vendor returns its own job identifier in its capability namespace, add it to that log line; the exact key is vendor-specific and their documentation is the place to get it, not this article.

Wiring it into CI so drift is visible before it is a defect

An assertion that fails a single test tells one team. A scheduled contract check tells the platform team before anyone else is affected.

# .github/workflows/relay-contract.yml
name: relay contract

on:
  schedule:
    - cron: "0 6 * * 1-5"     # weekday mornings, before the first suites run
  workflow_dispatch:

jobs:
  verify-relay-pools:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false        # one drifting pool must not hide the others
      matrix:
        pool:
          - { browser: safari,  version: "17",  platform: mac }
          - { browser: chrome,  version: "126", platform: windows }
          - { browser: firefox, version: "128", platform: linux }
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "17"

      - name: Confirm the relay node is registered before asserting anything
        run: |
          set -euo pipefail
          curl -sS --fail -X POST -H 'Content-Type: application/json' \
            --data '{"query":"{ grid { nodeCount totalSlots } }"}' \
            "${{ secrets.GRID_ROUTER_URL }}/graphql" \
            | tee /tmp/grid.json
          test "$(jq '.data.grid.nodeCount' /tmp/grid.json)" -gt 0

      - name: Assert the pool honours its declared contract
        env:
          GRID_ROUTER_URL: ${{ secrets.GRID_ROUTER_URL }}
          POOL_BROWSER: ${{ matrix.pool.browser }}
          POOL_VERSION: ${{ matrix.pool.version }}
          POOL_PLATFORM: ${{ matrix.pool.platform }}
        run: ./gradlew relayContractTest --tests '*RelayContractIT'

      - name: Publish the capability report even on failure
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: relay-capabilities-${{ matrix.pool.browser }}
          path: build/reports/relay/
          retention-days: 30

fail-fast: false is not a style preference here. When a vendor changes a default, it often changes several at once, and a matrix that stops at the first failure will report one drifting pool and hide two more. You want the full picture in a single run.

Uploading the report on failure matters for the same reason: the capability comparison is the evidence, and the evidence is what you send to the vendor when you open a ticket. A red check with no artifact starts a conversation you cannot win.

Second failure mode: Grid oversubscribing the vendor

This one costs money and looks like flakiness, which is a bad combination.

--service-configuration carries a max-sessions value, and the TOML configs array carries the same number as its first element in each pair. That number is what your Grid believes it may run concurrently against the relay. It is not negotiated with the vendor and the vendor does not see it. Your vendor plan has its own concurrency ceiling, defined by your contract with them.

When the Grid number exceeds the vendor number, the Distributor happily matches more requests than the vendor will serve. The Grid's own view stays healthy: slots exist, they are being filled, sessionCount climbs. The sessions that exceed the vendor ceiling fail at the vendor, and depending on how the vendor reports that, the error surfaces as a session creation failure that reads like an infrastructure blip.

The signature is a sessionCount that plateaus at a value lower than maxSession while failures accumulate. Grid thinks it has more room; the vendor disagrees. Compare the two numbers directly:

# Does the Grid believe it has more relay capacity than the vendor plan allows?
curl -sS --fail -X POST -H 'Content-Type: application/json' \
  --data '{"query":"{ nodesInfo { nodes { id uri status slotCount maxSession sessionCount stereotypes } } }"}' \
  "$ROUTER/graphql" \
  | jq '.data.nodesInfo.nodes[]
        | select(.uri | contains("relay"))
        | {id, uri, slotCount, maxSession, sessionCount}'

# Then compare maxSession against your vendor contract's concurrency limit.
# If Grid's number is larger, Grid will oversubscribe and the vendor will refuse.

The fix is to set the relay max-sessions at or below the vendor ceiling, and to leave headroom if any other pipeline uses the same vendor account directly. That second clause catches more teams than the first: a Grid sized exactly to the plan will still oversubscribe the moment somebody's local run or a legacy job talks to the vendor without going through the relay.

There is a documented Node flag worth knowing here, --node-down-failure-threshold, described as the "Maximum number of consecutive session creation failures before the Node is marked as DOWN. A value of 0 (default) disables this feature and allows unlimited retries." On a relay node this is a useful circuit breaker: rather than letting a saturated or broken vendor endpoint absorb every request in the queue, a non-zero threshold takes the node out of matching after a run of failures. Set it deliberately, because the default of 0 means the relay will keep accepting work into a failing endpoint indefinitely.

Third failure mode: the vendor incident that never turned the node red

A vendor had a partial regional outage. Their status endpoint kept returning HTTP 200, because a status page returning 200 during a degradation is normal and not dishonest. The relay node stayed green in the Grid UI for the entire incident. Sessions were being created and then failing partway through, so the suite reported a wall of assertion failures rather than an infrastructure error, and two engineers spent a morning bisecting application code.

This is the direct consequence of the documented health check contract: an HTTP 200 response is expected, and an HTTP 200 response was received. Grid did what it says it does.

The countermeasure is to stop treating the relay node's green status as evidence of vendor health, and to add a synthetic check that creates and destroys one real session per pool on a short interval. That is the only check that exercises the same path your tests do. It costs one session per pool per interval against your vendor concurrency, which is a real cost and should be sized deliberately: a five-minute interval across four pools is a meaningful slice of a small plan.

The second countermeasure is cheaper and often enough on its own. Make the driver factory distinguish an infrastructure failure from an assertion failure and label the run accordingly. A suite that reports "42 assertion failures" during a vendor incident sends people to the application. A suite that reports "42 sessions created, 39 lost mid-session" sends them to the vendor status page, which is where they should have started.

How to tell it is relay routing and not a look-alike

Four things produce "the cloud tests are broken" and they separate cleanly on evidence.

Capability drift. Sessions are created successfully and the returned capabilities disagree with the stereotype. Evidence: the New Session response. This is the only failure in the list where everything looks green.

Grid-side queueing. Requests never reach the vendor. Evidence: sessionQueueSize is elevated in the Grid GraphQL view while the relay node's sessionCount sits below its maxSession. The Grid is holding the request, so the relevant deadline is --session-request-timeout, and the vendor console will show no corresponding job at all.

Vendor-side queueing or refusal. Requests reached the vendor and the vendor did not serve them. Evidence: the mirror image, with relay sessionCount at its ceiling, and jobs visible in the vendor console. Now the vendor's own limits and documentation govern, not the Grid's.

Relay node not registered. No sessions route at all, for any capability that only the relay can serve. Evidence: the relay is absent from nodesInfo. A relay node is an ordinary Node in every respect except where it forwards commands, so it registers over the Event Bus like any other, using --publish-events and --subscribe-events or the --hub shorthand. The most common cause is a relay TOML that configures [relay] beautifully and omits [events] entirely.

The fastest discriminator between the middle two is the vendor console. If the job exists there, the request left your Grid, and everything upstream of the vendor is exonerated. That is a thirty-second check and it should be step one of the runbook.

Rolling this out on an existing Grid

Step one, measure before you enforce. Deploy the contract verification in log-only mode: record drift, throw nothing. Run it for a week. You will almost certainly find drift you did not know about, and finding it while the build is green is a much better experience than finding it in a blocked release.

Step two, fix the stereotypes, not the assertion. The temptation after step one is to loosen the enforced key list until everything passes. Resist it. If the stereotype claims a version the vendor does not deliver, the stereotype is wrong and should be corrected to say what you actually get, or the suite should be changed to request the version explicitly so the vendor receives the instruction.

Step three, enforce on one pool. Pick the pool where drift matters most, usually the one covering a browser you cannot test locally, and turn the exception on there first.

Step four, add the scheduled contract job. Once enforcement is stable on one pool, the matrix job above extends coverage without adding risk to the main pipeline.

Step five, reconcile concurrency. Compare relay max-sessions against the vendor plan and against every other consumer of that vendor account. Write the reconciliation down, because it silently rots whenever someone adds a pipeline.

Step six, add the synthetic session check. Last, because it consumes vendor concurrency, and because by this point you understand your own consumption well enough to size it.

The trade-offs

A relay adds a hop, a queue and a timeout. You get one Grid URL, one place to govern capabilities, and vendor credentials that never touch a pipeline. You also get a second queue in the path, a second deadline to reason about, and one more component to upgrade. For a single suite talking to a single vendor, that is a bad trade. For six suites across three vendors, it is usually a good one.

Enforced capabilities trade convenience for truth. A strict contract fails runs on the morning the vendor rolls a new default. That is the feature, and it will still feel like an obstacle at 09:00 on a release day. Decide in advance whether a drift failure blocks the release or files a ticket, and write it down before it happens rather than during.

Pinning versions trades coverage for determinism. Pinning browserVersion makes runs reproducible and quietly stops you testing the browser your users are actually upgrading to. A reasonable middle path is a pinned pool for the release gate and a floating pool that runs the same suite on the vendor default, with only the pinned one blocking. That costs a second pool.

Synthetic checks trade vendor concurrency for detection speed. Every synthetic session is a session your suites cannot use. On a large plan this is noise. On a small plan it is a meaningful percentage, and a fifteen-minute interval may be the honest compromise.

When not to do this

Do not put a relay in front of a vendor if exactly one suite uses that vendor. Point the suite at the vendor endpoint directly. The relay's value is consolidation, and consolidating one thing is overhead with a nice diagram.

Do not relay if you depend on vendor features that need their endpoint directly. Some vendor capabilities, local tunnels and debugging surfaces assume a direct connection, and discovering the incompatibility after you have migrated four pipelines is an expensive way to learn it. Prototype one suite through the relay and exercise the features you actually rely on before committing.

Do not relay across an unreliable network path. Every session now depends on your Grid reaching the vendor for the whole session, not just at creation. If your Grid runs somewhere with a flaky egress path, the relay converts a vendor-side problem into a distributed one with two places to look. Run relay nodes close to the network edge that reaches the vendor most reliably.

Do not add the enforcement layer before you have somewhere for its failures to go. A contract assertion that fires into a channel nobody reads gets disabled within a fortnight. Ownership first, assertion second.

And do not use a relay purely to hide credentials if that is the only requirement. A secrets manager and a shared driver factory solve credential exposure at a fraction of the operational cost. Relay routing earns its keep when you need one matching surface across many providers, not when you need one secret in one place.

Related reading

For the base configuration walkthrough, see Selenium Grid relay node configuration. Once sessions cross a vendor boundary, the correlation problem gets harder, and Selenium Grid OpenTelemetry trace correlation covers joining the two halves properly. For the component model that decides where a request goes in the first place, Selenium Grid components interview scenarios is the right refresher, and Selenium Grid Kubernetes interview questions covers relay nodes running as pods. If your suites use BiDi across a relay, the routing considerations in Selenium BiDi cross-context event routing apply on top of everything here.

FAQ

If the stereotype says browserVersion 17, why did my session run on something else?

Because a stereotype is a matching template, not an instruction to the vendor. The CLI reference describes --service-configuration as configuration "for the service where calls will be relayed to", and the Distributor uses it to decide whether this node can serve a request. Nothing in that mechanism forces the remote service to honour what you declared. The only authoritative statement about what you got is the capabilities object the W3C New Session response returns.

What exactly does the relay health check verify?

Very little, deliberately. --service-status-endpoint is documented as an "endpoint to query the WebDriver service status, an HTTP 200 response is expected", so the check is satisfied by any 200. A vendor whose region is degraded, whose queue is saturated, or whose account has hit its concurrency ceiling will usually still return 200 from a status page, which is why relay nodes stay green through vendor incidents.

How do I connect a Grid session back to the vendor's job record?

Write both identifiers into the same log line at session creation, from the client side, and never rely on either system to correlate for you. Capture the Selenium session id from the driver, capture whatever job identifier your vendor returns in its capability namespace, and emit one structured event containing both plus your CI run id. Without that record, a failed run means opening two consoles and matching by timestamp.

Can one relay node serve several browser and platform combinations?

Yes. --service-configuration accepts multiple entries, and the TOML configs array holds alternating slot counts and stereotype JSON strings, so a single relay node can advertise Safari on macOS and Chrome on Windows at the same time. The practical limit is that all of them share one --service-url, so combinations that need different endpoints need different relay nodes.

Which timeout fires first when a session cannot start, the Grid one or the vendor one?

It depends on values you control, and you should make the answer deterministic rather than discovering it during an incident. The Grid side is --session-request-timeout, which governs how long a request may sit in the New Session Queue. The vendor side is its own queueing behaviour and is documented by the vendor. Set the Grid deadline shorter than the vendor's if you want Grid errors, longer if you want vendor errors, but pick one on purpose.

Do I still need the event bus configured on a relay node?

Yes, because a relay node is an ordinary Grid Node in every respect except where it sends WebDriver commands. It registers with the Distributor and sends heartbeats over the same Event Bus as any other Node, using --publish-events and --subscribe-events or the --hub shorthand. Forgetting this produces a relay node that starts cleanly and never appears in the Grid.

Practice this

Go and read one stereotype from your own Grid, then create one session against it and diff the two capability maps by hand. If they match on every key you care about, you have five minutes of reassurance you did not have before. If they do not, you have just found out what your last three release gates were actually testing. Then take the reasoning into the QABattle battle arena: pick a Selenium infrastructure scenario and argue which single artefact distinguishes a Grid-side queue timeout from a vendor-side refusal, before you look at the answer.