Half your Grid is local, half is a vendor, and the test landed in the wrong half

The Chrome regression is supposed to run on the in-house Nodes and the Safari smoke tests on the vendor. Then the monthly bill arrives with vendor minutes charged against Chrome tests, and in the same week a Safari test sits in the queue until it times out while the vendor sits idle with capacity. Nobody changed the routing rules. Somebody added a Node whose advertised stereotype was broader than they realised, and requests started matching two pools instead of one.

That is the hybrid Grid failure in its purest form. It does not throw. There is no red build to investigate on the day it starts. It shows up weeks later as a cost line, a queue timeout, or a browser-version bug that only reproduces on half the runs, and by then nobody remembers which commit widened the stereotype.

The mechanism: what the Grid promises and what it does not

The Selenium documentation is precise about the components, and the precision is where the answer lives.

The Router is the entry point of the Grid and forwards requests to the correct component. For a new session it forwards to the New Session Queue. For a request against an existing session it queries the Session Map, which keeps the relationship between the session id and the Node where the session is running, and forwards the request directly to that Node.

The Distributor does two things. Nodes register with it by sending a registration event through the Event Bus, at which point it confirms the Node's existence over HTTP and tracks all Node capabilities. Separately, it polls the New Session Queue, which holds new session requests in FIFO order, and finds a suitable Node where the session can be created.

Each Node manages the slots for the available browsers on the machine where it is running, and executes commands without making evaluative decisions. Through specific configuration, a Node can also run sessions in Docker containers or relay commands.

Now read the Distributor sentence one more time. It finds a suitable Node. It does not say which suitable Node, and the documentation makes no promise about the tie-break when several qualify. That is not an oversight in the docs, it is a design property: the Grid's job is to satisfy your capability request, not to honour a preference you never expressed.

Everything in a hybrid setup follows from that. If your local Chrome Nodes and your vendor relay both advertise a slot that satisfies {"browserName": "chrome"}, then a request for {"browserName": "chrome"} is ambiguous, and an ambiguous request will be resolved by something you do not control. The bug is not in the Distributor. The bug is that you asked a question with two correct answers and expected one of them.

The three shapes a "hybrid" Grid actually takes

People use the word for at least three different architectures, and the failure modes differ.

Shape one: one Grid, mixed Node types. A single control plane (Router, Distributor, Session Map, New Session Queue) with local Nodes running browsers directly, Nodes configured to launch Docker containers per session, and Nodes configured as relays pointing at an Appium server or a vendor. One endpoint for the test framework. Maximum convenience, maximum ambiguity.

Shape two: several Grids, routed in the framework. Separate endpoints for separate pools, with the test framework choosing. Less elegant, and the routing decision lives in code where a reviewer can see it.

Shape three: one Grid, several regions or failure domains. A different problem again, closer to multi-region Grid design than to cost routing.

This article is about shape one and shape two, because that is where the "landed in the wrong half" bug lives.

Where relay fits, and what it hides

Relay is the documented way to make a Node forward WebDriver commands to another service. The CLI options describe it plainly: --service-url is the URL for connecting to a service that supports WebDriver commands like an Appium server or a cloud service, --service-status-endpoint is an optional endpoint to query that service's status where an HTTP 200 response is expected, and --service-configuration is the configuration for the service where calls will be relayed to.

The TOML form is the one you will actually maintain:

# node-relay.toml
# Launch with:
#   java -jar selenium-server-<version>.jar node --config node-relay.toml

[server]
port = 5556

[node]
detect-drivers = false
max-sessions = 5

[relay]
url = "http://appium-host:4723/wd/hub"
status-endpoint = "/status"
protocol-version = "HTTP/1.1"
# Pairs of (max instances, stereotype JSON). The stereotype is what this
# relay ADVERTISES. Everything downstream matches against these strings.
configs = [
  "5", "{\"browserName\": \"chrome\", \"platformName\": \"android\"}"
]

Two properties of this config decide whether your hybrid Grid works.

detect-drivers = false matters more than it looks. A Node that auto-detects local drivers will advertise them in addition to anything else you configured. A relay host that happens to have Chrome installed can end up advertising both a relayed Android Chrome slot and a local desktop Chrome slot. That is a second, invisible pool, and it is a very common way for the "wrong half" bug to appear on a box nobody thought was a browser Node.

The configs stereotype is the entire contract. It is what the Distributor matches against. If the stereotype says {"browserName": "chrome"} with nothing else, this relay is a candidate for every plain Chrome request in your suite, including the thousands you intended to run locally.

And the health contract deserves suspicion. The documented expectation for status-endpoint is an HTTP 200 response. A device farm with zero free devices, or an Appium host whose driver is wedged, can still return 200. The Node stays UP, the Distributor keeps sending it work, and every session either queues or fails at creation. A status code is not a capacity check.

Worked example one: overlapping stereotypes

The scenario from the opening. Here is the audit that finds it in about thirty seconds.

GRID=${GRID:-http://localhost:4444}

# Every slot the Grid believes it has, grouped by what it advertises.
curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"query":"{ nodesInfo { nodes { id, uri, status, stereotypes } } }"}' \
  "${GRID}/graphql" | jq '.data.nodesInfo.nodes' > /tmp/nodes.json

# 1. Which node URIs claim a browser called "chrome"?
#    More than one distinct URI here means a plain chrome request is ambiguous.
jq -r '.[] | select(.status == "UP")
       | . as $n
       | (.stereotypes | tostring) as $s
       | select($s | test("chrome"; "i"))
       | "\(.uri)\t\($s)"' /tmp/nodes.json

# 2. The blunt version: how many UP nodes are there per advertised browser?
jq -r '[.[] | select(.status=="UP") | {uri, st: (.stereotypes|tostring)}]
       | group_by(.st) | map({stereotype: .[0].st, nodes: map(.uri)})' /tmp/nodes.json

# 3. Anything not UP is a routing surprise waiting to happen when it returns.
jq -r '.[] | select(.status != "UP") | "\(.status)\t\(.uri)"' /tmp/nodes.json

Query one is the finding. If two different URIs both advertise something matching chrome, and one of them is your vendor relay, then a plain {"browserName": "chrome"} request has two correct answers and the Grid will pick one.

The repair has two halves and both are needed.

Narrow the stereotypes so they are disjoint. Use capabilities that are part of the standard request vocabulary and therefore reliably participate in matching: browserName, browserVersion, platformName. A relay advertising {"browserName": "chrome", "platformName": "android"} is not a candidate for a desktop request that specifies platformName. A local pool advertising {"browserName": "chrome", "platformName": "linux"} is not a candidate for the mobile one. The moment both sides name the platform, the ambiguity is gone.

Make the request specific too. Disjoint stereotypes do nothing if the client keeps sending {"browserName": "chrome"} with no platform, because that request still matches both. Narrowing has to happen on both sides or it has not happened.

You may be tempted to add a bespoke key such as a pool name to the stereotype and rely on it for matching. Resist building a routing strategy on that until you have proved it on your exact Grid version. Whether an arbitrary extension capability participates in slot matching is not something to take on faith from an article, including this one. The experiment below settles it in two minutes, and running it is cheaper than a quarter of mis-billed sessions.

The experiment that proves your routing

This is the diagnostic that separates "I think it routes correctly" from "I know it does". Request each pool explicitly, then read back which Node actually served you.

package com.thetestingacademy.grid;

import java.net.URI;
import java.net.URL;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.remote.SessionId;

public final class PoolAssertingFactory {

  public enum Pool { LOCAL, RELAY }

  /**
   * Creates a session and refuses to hand it back if it landed on a Node
   * outside the expected pool. Failing here costs one session. Not failing
   * here costs a month of invoices.
   */
  public static RemoteWebDriver create(String gridUrl, Pool pool, String testId)
      throws Exception {

    ChromeOptions options = new ChromeOptions();
    options.setCapability("se:name", testId);
    switch (pool) {
      case LOCAL -> options.setCapability("platformName", "linux");
      case RELAY -> options.setCapability("platformName", "android");
    }

    RemoteWebDriver driver = new RemoteWebDriver(new URL(gridUrl), options);
    SessionId sessionId = driver.getSessionId();
    if (sessionId == null) {
      driver.quit();
      throw new IllegalStateException("Null session id for " + testId);
    }

    String nodeUri = resolveNodeUri(gridUrl, sessionId.toString());
    if (!matchesPool(nodeUri, pool)) {
      driver.quit();
      throw new AssertionError(
          testId + " requested pool " + pool + " but landed on " + nodeUri);
    }

    System.out.printf("%s -> session=%s node=%s returnedBrowser=%s/%s%n",
        testId, sessionId, nodeUri,
        driver.getCapabilities().getBrowserName(),
        driver.getCapabilities().getBrowserVersion());

    return driver;
  }

  /** Single-session GraphQL query. Returns nodeUri for a session id. */
  private static String resolveNodeUri(String gridUrl, String sessionId) throws Exception {
    String body = """
        {"query":"{ session (id: \\"%s\\") { id, nodeId, nodeUri } }"}"""
        .formatted(sessionId);

    HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create(gridUrl + "/graphql"))
        .timeout(Duration.ofSeconds(10))
        .header("Content-Type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body))
        .build();

    HttpResponse<String> response = HttpClient.newHttpClient()
        .send(request, HttpResponse.BodyHandlers.ofString());

    if (response.statusCode() != 200) {
      throw new IllegalStateException(
          "GraphQL returned " + response.statusCode() + ": " + response.body());
    }
    return response.body();  // parse with your JSON library of choice
  }

  private static boolean matchesPool(String nodeUriPayload, Pool pool) {
    // Match on whatever actually identifies your pools: host prefix,
    // port range, DNS suffix. Keep this ONE line and keep it obvious.
    return switch (pool) {
      case LOCAL -> nodeUriPayload.contains("node-local");
      case RELAY -> nodeUriPayload.contains("node-relay");
    };
  }

  private PoolAssertingFactory() {}
}

The important part is not the HTTP plumbing, it is the AssertionError. A hybrid Grid without a pool assertion is a hybrid Grid where a routing mistake is undetectable until it shows up on an invoice. One failed session is a cheap alarm.

Run it as a standalone check before you trust any routing rule:

# Ask for each pool ten times, record where each session actually landed.
# Any row that does not match the requested pool is your answer.
for pool in LOCAL RELAY; do
  for i in $(seq 1 10); do
    java -cp target/test-classes:target/classes \
      com.thetestingacademy.grid.PoolProbe "${GRID}" "${pool}" "probe-${pool}-${i}" \
      || echo "MISROUTED ${pool} attempt ${i}"
  done
done

# Cross-check against the Grid's own view while the probes run.
watch -n 2 "curl -s -X POST -H 'Content-Type: application/json' \
  --data '{\"query\":\"{ grid { sessionCount, sessionQueueSize, maxSession } }\"}' \
  ${GRID}/graphql | jq -c '.data.grid'"

Ten attempts per pool is not a statistical proof. It is a smoke test, and a single misroute in twenty is enough to reject a design. If you want a number to trust, run it against your real concurrency for a full suite and count.

Worked example two: the Node that registers and then cannot be reached

This one is worth understanding because the symptom points in exactly the wrong direction.

Sessions create successfully. The Grid UI shows the Node as UP with a live session. Then the very first command after creation fails with a connection error, every time, on that Node only.

Go back to the architecture. Node registration travels over the Event Bus, and the Distributor confirms the Node's existence over HTTP. But commands against an existing session do not travel that path at all: the Router queries the Session Map for the Node and forwards the request directly to that Node. Those are two different network paths with two different reachability requirements.

So a Node can be perfectly registered and completely unreachable for commands. It happens whenever the Node advertises a URI that is correct from its own point of view and wrong from the Router's: a container advertising its internal hostname to a Router outside the network, a Kubernetes pod advertising a pod IP across a namespace boundary, a host behind NAT advertising a private address.

The check is direct:

# 1. What URI did each Node actually register?
curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"query":"{ nodesInfo { nodes { id, uri, status } } }"}' \
  "${GRID}/graphql" | jq -r '.data.nodesInfo.nodes[] | "\(.status)\t\(.uri)"'

# 2. Can the ROUTER reach that URI? Run this from inside the router
#    container/host, not from your laptop. That distinction is the whole test.
docker compose exec router sh -lc \
  'for u in $(cat /tmp/node-uris.txt); do
     printf "%s -> " "$u"
     curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 "$u/status" || echo unreachable
   done'

If step one lists a URI and step two cannot reach it from the Router, you have found it, and no amount of capability tuning was ever going to help. In a hybrid Grid this bites hardest on the pool that is architecturally different from the others, which is usually the relay or the containerised one, which is why it gets misdiagnosed as "the vendor is flaky".

Worked example three: the relay that is UP and useless

The third failure needs no misconfiguration at all. The relay's status endpoint returns 200 as documented, so the Node reports UP and the Distributor treats it as available capacity. Behind it, the device farm has no free devices, or the Appium server's driver is wedged, or the vendor is degraded.

The Grid is behaving exactly as specified. The specification is just weaker than your mental model of it.

Signature: queue depth climbing while nodesInfo shows Nodes UP and slots that look free, and session creation failing or timing out specifically for the relayed browser. The distinguishing evidence is that the failure is creation-side and pool-specific: local pools are unaffected in the same window.

The mitigation is a health check that exercises a real session rather than a status code. Run it on a schedule, outside the test suite, and have it drain traffic away from the relay rather than trying to make the Grid smarter:

# .github/workflows/pool-health.yml
name: pool-health
on:
  schedule: [{ cron: "*/15 * * * *" }]
  workflow_dispatch:

jobs:
  probe:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        pool: [LOCAL, RELAY]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: "21"
          distribution: temurin
          cache: maven

      - name: Snapshot Grid state before probing
        run: |
          mkdir -p artifacts
          curl -s -X POST -H 'Content-Type: application/json' \
            --data '{"query":"{ grid { sessionCount, sessionQueueSize, maxSession } }"}' \
            "${GRID}/graphql" | jq . > artifacts/grid-before.json
          curl -s -X POST -H 'Content-Type: application/json' \
            --data '{"query":"{ nodesInfo { nodes { id, uri, status, stereotypes } } }"}' \
            "${GRID}/graphql" | jq . > artifacts/nodes-before.json
        env:
          GRID: ${{ vars.GRID_URL }}

      - name: Create one real session in ${{ matrix.pool }} and assert the node
        run: mvn -B -q test -Dtest=PoolProbeIT -Dpool=${{ matrix.pool }}
        env:
          GRID_URL: ${{ vars.GRID_URL }}

      - name: Snapshot Grid state after a failure
        if: failure()
        run: |
          curl -s -X POST -H 'Content-Type: application/json' \
            --data '{"query":"{ sessionsInfo { sessionQueueRequests } }"}' \
            "${GRID}/graphql" | jq . > artifacts/queue-after.json
        env:
          GRID: ${{ vars.GRID_URL }}

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pool-health-${{ matrix.pool }}
          path: artifacts/
          retention-days: 7

Note fail-fast: false on the matrix. A hybrid setup exists precisely so that one pool being sick does not stop the other, and a health check that cancels the healthy probe when the sick one fails throws that away.

How to tell it is this and not a look-alike

Four diagnoses produce similar noise. The discriminating evidence for each:

Genuine capacity exhaustion. Signature: queue depth high, all Nodes UP, no free slots anywhere, and the pressure is spread across pools rather than concentrated in one. Fix is capacity, not routing. Check grid { sessionCount, maxSession, sessionQueueSize } and compare against the sum of advertised slots.

Capability mismatch, no candidate at all. Signature: queue depth high, slots free, and requests time out at --session-request-timeout without ever being matched. Nothing is misrouted because nothing was routed. Diff your requested capabilities against the stereotypes field character by character; it is almost always a browserVersion string or a platformName casing difference.

Ambiguous match, the actual hybrid bug. Signature: sessions start promptly and succeed, and the nodeUri is not the one you expected. This is invisible unless you read the Node URI back, which is the argument for the pool assertion.

Behavioural difference between pools, correctly routed. Signature: routing is right, and the test still fails only on one pool. Returned capabilities differ (different browser build, different platform, different screen size), or the pool has genuinely different behaviour such as download handling or clipboard access. This is not a routing bug and narrowing stereotypes will not fix it. It is an argument for treating pools as distinct environments with distinct expected results.

The fourth one is the reason "just make it one endpoint" is a weaker idea than it sounds. Uniform routing does not create uniform behaviour, and a design that hides the difference makes the resulting failures harder to explain, not easier.

What the fix costs

Narrowing stereotypes costs flexibility. Once local Chrome advertises platformName: linux and every request names its platform, you have lost the ability to say "any Chrome, wherever". Some suites genuinely want that (a large stateless smoke run that should use whatever is free), and for those, ambiguity is a feature. Split your suites before you split your stereotypes.

Pool assertions cost one round trip per session. A GraphQL call at creation, roughly the cost of one page navigation, against a session creation already measured in hundreds of milliseconds. Not free, not significant. If it were significant, sample it: assert on the first session per pool per run rather than on every session.

Separate endpoints cost a config surface. Two Grid URLs, two health checks, two sets of credentials, and a factory branch. In exchange, a misroute becomes structurally impossible rather than merely improbable, and the routing decision appears in a code review. For most teams that trade is worth it, and it is the recommendation if you are deciding today.

Relay costs you the vendor's diagnostics. Commands go through your Node to their service. When something fails you now have two log surfaces, and the relay hides which one broke. Weigh that against the convenience of one endpoint honestly; on a bad day the convenience is worth very little.

Running the components separately costs operational surface. Distributed mode gives you independent scaling and clean failure domains, and it also gives you six processes to monitor, six log streams to collect, and an Event Bus whose reachability is now a production concern. Do not go distributed for a Grid that a Hub and two Nodes would serve.

Rollout path

1. Audit before you change anything. Run the stereotype query and write down, in a file people can review, which pools can satisfy which requests today. Most teams find at least one overlap they did not know about.
2. Add the pool assertion in warn-only mode. Resolve the Node URI, log a mismatch, do not fail. One week of data tells you the real misroute rate, which is the number that gets the rest of the work funded.
3. Narrow the stereotypes, one pool at a time. Start with the pool that costs money. Change the advertised stereotype and the client request in the same commit, because changing only one produces a Grid where nothing matches.
4. Prove it with the probe loop. Ten sessions per pool, checking the actual Node URI. Do not proceed on the basis that it looks right in the UI.
5. Flip the assertion to failing. Now a misroute stops a build instead of appearing on an invoice.
6. Only then consider consolidating endpoints, or splitting them. By this point you have measurements, so the decision is an engineering one rather than an aesthetic one.
7. Add the scheduled pool health check. Last, because it is the thing that keeps the property true after everyone has moved on to other work.

If your framework does not yet have a single place where drivers are created, do that first. Every step above assumes one session factory you can change once.

Trade-offs worth arguing about

One endpoint versus several. One endpoint is a better developer experience and a worse safety property. Several endpoints put the routing decision in reviewable code and force the framework to be explicit about which pool a suite belongs to. Teams that pick one endpoint and then spend months building routing logic inside capabilities have effectively rebuilt the several-endpoint design with worse ergonomics and no type safety.

Relay versus talking to the vendor directly. Relay gives you unified reporting, unified session IDs, and Grid-side visibility. Direct vendor access gives you their full capability set, their dashboards, and their support team's ability to help you. If you route more than a small fraction of your volume to a vendor, direct access usually wins, and you use the Grid for what you own.

Narrow stereotypes versus flexible capacity. Disjoint pools mean predictable cost and predictable behaviour. Overlapping pools mean better utilisation and unpredictable placement. There is no universally right answer, only a right answer per suite. The mistake is having one policy for a suite mix that needs two.

Assert on the pool versus trust the config. Assertions cost a round trip and catch drift. Trusting the config costs nothing and catches nothing. The honest middle ground is asserting on a sample, and being explicit that a sample is what you are doing rather than pretending to full coverage.

When not to do this

You have one pool. All local Nodes, all the same stereotype, no vendor. Then there is no routing decision to get wrong, and adding pool assertions, disjoint stereotypes and probe loops is machinery guarding a door with nothing behind it. Come back when you add the second pool.

Your vendor usage is a rounding error. A handful of Safari sessions a week does not justify a relay Node, a health probe and a routing policy. Point those tests at the vendor directly from the framework, keep the branch in code, and spend the effort elsewhere.

Utilisation matters more than placement. Some workloads genuinely want any free browser: large stateless smoke suites, link checkers, screenshot sweeps. For these, overlapping stereotypes are correct and narrowing them will leave capacity idle while requests queue. Do not apply a policy designed for a cost-sensitive regression suite to a workload that has no such constraint.

You cannot control the stereotypes. If a platform team owns the Nodes and will not change their advertised capabilities, then narrowing on the client side alone will only make requests match nothing. Separate endpoints are the workable answer in that org, and it is a political constraint dressed as a technical one. Name it as such.

The pools genuinely behave differently and your tests depend on it. Different browser builds, different download behaviour, different network egress. Routing correctly does not make those tests pass. Treat the pools as different environments with their own expected results, rather than trying to make one suite pretend they are interchangeable.

You are debugging a Kubernetes reachability problem. If sessions create and then immediately fail, stop working on capabilities entirely. That is the Router-to-Node path, it is a networking question, and it is covered better by Kubernetes-specific Grid reasoning than by any amount of stereotype tuning.

Practise the reasoning

The habit worth building is small: for any hybrid Grid, be able to answer "which pools can satisfy this request" from the stereotypes field rather than from memory, and be able to answer "which pool actually served this session" from nodeUri rather than from inference.

Try it on your own Grid this week. Pull the stereotypes, list every pool that could satisfy your most common capability request, and see whether the count is one. Then take a session ID from last night's run and resolve its Node URI. If either answer surprises you, you have found something real, and you found it before the invoice did.

You can drill the same decisions against timed scenarios in the QABattle arena. Pick a Grid routing scenario, say out loud which component you would query first, and name the single field whose value would send you somewhere else entirely.