Scale Grid nodes before the queue becomes a timeout

At 09:00, twenty Chrome jobs reach the Grid within a few seconds. The autoscaler notices the queue and asks Kubernetes for more pods, but a Chrome Node takes ninety seconds to schedule, start, and register. Half the clients give up after sixty seconds, just before the new slots appear. Scaling happened, yet it could never meet the wait promised to users.

Scale the capability pool, not the global queue

A new-session request enters Selenium Grid's Session Queue before the Distributor finds a compatible slot. The queue is therefore closer to unmet browser demand than Node CPU or pod CPU. It still is not a ready-made autoscaling command.

The Distributor matches request capabilities against slot stereotypes. A queue with nine Chrome requests and one Firefox request does not ask for ten interchangeable replicas. It asks for capacity in at least two service classes. Browser name is usually the first partition. Browser version, platform, or a custom capability also belongs in the partition when it changes which Node can accept the request.

Start with the smallest set of capability fields that represent real pools. If all Linux Chrome Nodes run the same current browser, browserName=chrome may be enough. If releases require both current and previous Chrome, split by the version constraint. If a vendor-specific option changes browser behavior but does not affect matching, do not turn it into another scaling dimension.

Reactive scale-out has a simple timing limit:

caller wait budget > detection delay + scheduler delay + image/startup delay + Grid registration delay + browser creation delay

A policy that checks every thirty seconds already spends up to thirty seconds before acting. Kubernetes may then need a node, pull a browser image, start the container, and wait for the Selenium Node to register. The slot helps only after the Distributor sees it. If the sum is longer than the remaining client budget, a warm floor or scheduled pre-scaling is mandatory.

Queue length measures a snapshot. Queue persistence measures pressure. Three requests that vanish at the next five-second poll probably used slots that were already turning over. Three requests present for a minute show that supply is not catching up. Track age from the client-side start of the new-session call or from a durable first-seen record. Do not pretend the aggregate GraphQL queue-size field contains age when it does not.

Capture both the total and the payloads. The total catches parsing errors. The payloads let the controller assign demand to a pool. This read-only probe stores the exact GraphQL response before producing any decision.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="$SELENIUM_GRID_URL"
SNAPSHOT_DIR="artifacts/grid-queue"
mkdir -p "$SNAPSHOT_DIR"

stamp="$(date -u +%Y%m%dT%H%M%SZ)"
curl --fail --silent --show-error \
  -H 'Content-Type: application/json' \
  --data '{"query":"{ grid { sessionQueueSize sessionCount maxSession } sessionsInfo { sessionQueueRequests } nodesInfo { nodes { id status uri stereotypes slotCount sessionCount } } }"}' \
  "$GRID_URL/graphql" |
  tee "$SNAPSHOT_DIR/queue-$stamp.json"

Keep the raw response for every scale action. When an operator asks why six replicas appeared at 09:01, the answer should name the queued capability class, the observed samples, the old and new replica count, the cap, and the policy version.

Turn queue evidence into a bounded scale-out decision

A safe first controller only recommends additions. It does not remove Nodes, edit Grid state, or guess at unsupported requests. Observe its recommendations beside real incidents before granting it write access to the platform.

The following Python program polls the documented GraphQL queue three times, decodes each request, finds its requested browser, and calculates additions only for demand that persisted through every sample. It reads current Deployment replica counts with kubectl. Setting APPLY_SCALE=true enables scale-out; the default prints decisions.

#!/usr/bin/env python3
import collections
import json
import math
import os
import subprocess
import time
import requests

GRID_URL = os.environ["SELENIUM_GRID_URL"].rstrip("/")
APPLY = os.environ.get("APPLY_SCALE", "false").lower() == "true"
SAMPLE_SECONDS = int(os.environ.get("SAMPLE_SECONDS", "10"))
SLOTS_PER_REPLICA = int(os.environ.get("SLOTS_PER_REPLICA", "1"))
MAX_REPLICAS = int(os.environ.get("MAX_REPLICAS", "20"))

RESOURCES = {
    "chrome": "deployment/selenium-node-chrome",
    "firefox": "deployment/selenium-node-firefox",
    "MicrosoftEdge": "deployment/selenium-node-edge",
}
QUERY = "{ sessionsInfo { sessionQueueRequests } }"

def find_browser(value):
    if isinstance(value, dict):
        if isinstance(value.get("browserName"), str):
            return value["browserName"]
        for nested in value.values():
            found = find_browser(nested)
            if found:
                return found
    if isinstance(value, list):
        for nested in value:
            found = find_browser(nested)
            if found:
                return found
    return None

def queued_counts():
    response = requests.post(
        f"{GRID_URL}/graphql",
        json={"query": QUERY},
        timeout=10,
    )
    response.raise_for_status()
    items = response.json()["data"]["sessionsInfo"]["sessionQueueRequests"]
    counts = collections.Counter()
    for item in items:
        request = json.loads(item) if isinstance(item, str) else item
        browser = find_browser(request)
        counts[browser or "unsupported"] += 1
    return counts

def current_replicas(resource):
    result = subprocess.run(
        ["kubectl", "get", resource, "-o", "json"],
        check=True,
        capture_output=True,
        text=True,
    )
    return int(json.loads(result.stdout)["spec"]["replicas"])

samples = []
for index in range(3):
    samples.append(queued_counts())
    if index < 2:
        time.sleep(SAMPLE_SECONDS)

for browser, resource in RESOURCES.items():
    persistent = min(sample[browser] for sample in samples)
    if persistent == 0:
        continue
    current = current_replicas(resource)
    additions = math.ceil(persistent / SLOTS_PER_REPLICA)
    desired = min(MAX_REPLICAS, current + additions)
    decision = {
        "browser": browser,
        "persistent_queue": persistent,
        "current_replicas": current,
        "desired_replicas": desired,
        "applied": APPLY,
    }
    print(json.dumps(decision, sort_keys=True))
    if APPLY and desired > current:
        subprocess.run(
            ["kubectl", "scale", resource, f"--replicas={desired}"],
            check=True,
        )

unsupported = max(sample["unsupported"] for sample in samples)
if unsupported:
    print(json.dumps({
        "browser": "unsupported",
        "persistent_queue": unsupported,
        "action": "alert_without_scaling",
    }, sort_keys=True))

This is intentionally conservative. Taking the minimum across samples ignores one-poll spikes. Adding enough replicas for every persistent request favors latency over density. A production policy can include occupied slots, recent completion rate, and already-pending pods, but each extra input needs a clear failure behavior.

The browser mapping is policy, not discovery. An unfamiliar browser name goes to unsupported and raises an alert. Silently mapping it to Chrome risks spending money without serving the request. A real controller should also compare the complete request with the stereotypes expected from the target Deployment, especially where version or platform pools exist.

Track pending additions. Without that state, each poll sees the same queue and adds another batch while earlier pods are still starting. The platform's current replica count includes desired pods, but controllers should also record the action generation and wait for matching slots to register before concluding the scale-out failed. Put a hard replica cap behind every pool.

The trade-off in this policy is overreaction. A request counted in all three samples may receive an existing slot just as the controller adds a Node. That extra Node can sit idle. Requiring more samples reduces waste but adds detection delay. Choose the sampling window from measured session turnover and startup time, not from a fashionable controller interval.

Warm capacity changes the equation. Keep at least enough ready slots to absorb arrivals during one detection and startup interval. If Chrome takes ninety seconds to become ready and the 95th-percentile arrival burst is eight requests over ninety seconds, a zero-replica floor cannot meet a short SLO. The floor costs money during quiet periods, but it buys time for reactive nodes to join.

Diagnose three autoscaling failures before tuning thresholds

The first failure scales the wrong browser. A global metric reports ten queued requests, and a general-purpose autoscaler adds five mixed Nodes. The Node template happens to advertise Firefox because Chrome was disabled in that image. Nine Chrome requests remain queued while the new Firefox slots are free.

The evidence is a capability mismatch: queued payloads ask for Chrome, Chrome-compatible free slots are zero, Firefox free slots increase, and total queue falls by at most one. CPU looks healthy because the wrong Nodes are idle. The fix is a per-stereotype controller and separate scaling target. The cost is more Deployments, dashboards, limits, and rollout paths.

The second failure acts too late. Five Chrome requests enter the queue at 09:00:00. Detection occurs at 09:00:25. Kubernetes schedules pods at 09:00:45, the image finishes pulling at 09:01:30, and the Nodes register at 09:01:42. Clients with a sixty-second wait have already failed.

Raising the client and Grid queue timeouts to two minutes may let those requests succeed, but users wait longer and CI workers remain occupied. Keeping warm nodes or scaling at 08:58 addresses the time budget instead. Scheduled capacity is not a failure of autoscaling; it is an honest response to predictable demand and slow supply.

The third failure oscillates. A controller removes idle pods as soon as queue length reaches zero. An active test still runs on each selected Node, so the platform terminates browsers mid-test or keeps pods in termination while replacements are requested moments later. Session failures and image pulls increase together.

Scale-in must use Grid lifecycle state. Drain a Node so it stops accepting new sessions, wait for its current sessions to finish, and only then reduce the platform workload. Selenium provides documented Distributor and direct Node drain endpoints. Kubernetes replica count alone does not know which pod holds a live browser.

A fourth failure is easy to miss: one impossible request keeps the queue nonzero. A typo asks for browserName=chromee, or a platform constraint names a pool that does not exist. A controller that treats every persistent request as demand climbs to its maximum. New slots stay free because none match.

Compare requests with advertised stereotypes before scaling. When the Grid will never create capacity dynamically for unsupported capabilities, the documented reject-unsupported-caps behavior can fail such requests early. The trade-off is that a temporarily absent pool also fails rather than waiting for recovery. Make that choice explicit per environment.

A fifth failure raises the platform replica count without raising Grid capacity. Kubernetes reports four new pods as Running, but none registers with the Distributor because their Event Bus address is wrong or the Router cannot reach their advertised URI. The queue persists, so a controller that equates pods with slots keeps requesting more.

The evidence crosses two systems. Platform scheduling and container startup complete, yet GraphQL shows no additional stereotypes or slot count. New Node logs repeat registration or connectivity failures, and existing Nodes continue serving sessions. Repair registration or network configuration before changing the queue threshold. Counting only registered, compatible slots makes the controller slower to declare success, but it prevents a green Kubernetes deployment from masquerading as browser supply.

Another deceptive result appears when all new Nodes register and session creation still fails. A broken browser image, missing shared-memory capacity, or driver-browser mismatch can advertise slots that never produce stable sessions. Track the first successful session on each scale generation. Registration is necessary; usable browser creation is the stronger acceptance signal.

Diagnostic output for a useful action record can look like this:

observed_at: "2026-08-04T09:00:30Z"
policy_version: "queue-scaler-3"
pool: "chrome-linux-current"
queue_samples: [5, 5, 4]
current_replicas: 2
pending_replicas: 0
registered_slots:
  occupied: 2
  free: 0
node_startup_seconds:
  p95: 92
decision:
  add_replicas: 4
  desired_replicas: 6
  limited_by_cap: false
client_wait_budget_seconds: 120

That record makes a later review possible. A line saying “scaled because queue > 0” does not.

The values in that YAML are illustrative, not measurements from a production Grid. Read them as relationships rather than independent thresholds. queue_samples shows whether demand survived the observation window. A falling sequence is healthy only when session-creation records explain the fall. The same decline is misleading when callers timed out and abandoned their requests. current_replicas is also easy to overvalue: it describes the platform workload, not usable browser supply. If that value rises while the sum of occupied and free compatible slots stays fixed, the scale action has not reached the Grid.

registered_slots.free needs similar context. Zero free slots is broken when the registered total did not grow after the requested replicas became ready. Zero can be healthy when new slots registered and were assigned between polls, so the occupied count rose while the queue fell. pending_replicas is not a success value. A nonzero value says supply is in flight, while zero can mean either no action or completed supply. The action generation and compatible slot change distinguish those outcomes. The illustrative p95: 92 beside a 120-second client budget appears to leave 28 seconds, but that margin is misleading unless detection and browser creation already fit inside it. A healthy record proves every elapsed component fits. A broken record shows the remaining budget reaches zero before allocation. limited_by_cap: true means the decision is intentionally constrained and needs a capacity alert, not another uncapped retry.

Separate real demand from trapped occupied slots

Another failure looks almost identical to legitimate undercapacity at the autoscaler boundary. The queue persists, matching free slots remain at zero, Node CPU can look ordinary, and adding replicas briefly reduces waiting. In one case, more live jobs arrived than the pool could serve. In the other, sessions from cancelled or crashed jobs still occupy slots, so nominal capacity exists but cannot return to the queue.

Queue snapshots alone cannot separate those roots. Join the session ID recorded when each driver was created to the owning CI attempt and its terminal state. In a real demand surge, occupied sessions map to live attempts, their test output continues, and normal session completions release slots that immediately take queued work. In a retained-session failure, at least one Grid session remains mapped after its owning attempt has finished or been cancelled. Its age continues to grow without corresponding test progress, while newer callers wait. A high session count is not proof of either condition because both produce it. The missing or present live owner is the decisive evidence.

That distinction changes ownership. The platform team owns additional supply when every occupied session has a live owner and arrival exceeds completion. The automation framework owner owns a missing close path when a finished attempt leaves a session behind. The CI team owns termination behavior when cancellation prevents cleanup from running. A useful handoff includes UTC timestamps, session IDs, CI run and attempt identities, requested capabilities, the terminal job state, the last test-progress record, matching Node identity, queue snapshots, and the scale actions taken. Without that bundle, each team can truthfully show a healthy subsystem while the slot remains unavailable.

Correlation has a concrete cost. Session and job lifecycle records must be retained long enough to cover the longest valid test, and every runner integration must emit the same ownership link. That increases log volume and adds maintenance whenever a runner changes. An aggressive age cutoff is cheaper to operate but can kill a legitimate long test. An overly generous cutoff preserves coverage but leaves leaked slots unavailable longer.

Drain scale-in without terminating valid tests

Scale-out can be eager because adding an idle Node mainly costs money. Scale-in can destroy work, so it needs a separate policy and a slower clock. Never mirror the scale-out formula in reverse.

Select Nodes from the target stereotype pool that are up, have zero sessions, and have been idle beyond a cooldown. If all Nodes have active sessions, drain the number you intend to remove. Draining changes availability so the Distributor will not send new sessions there. Existing sessions continue until they close.

The direct Distributor drain endpoint takes a Node ID. This script drains one explicitly selected Node, then polls GraphQL until it has observed that same Node ID in DRAINING state holding zero sessions. That pair of observations, not the Node's disappearance, is what authorizes removal. The script stops before changing Kubernetes replicas, leaving that last step to the platform controller after the condition is proven.

#!/usr/bin/env bash
set -euo pipefail

: "$SELENIUM_GRID_URL"
: "$NODE_ID"
: "$REGISTRATION_SECRET"
export NODE_ID

curl --fail --silent --show-error \
  --request POST \
  -H "X-REGISTRATION-SECRET: $REGISTRATION_SECRET" \
  "$SELENIUM_GRID_URL/se/grid/distributor/node/$NODE_ID/drain"

saw_draining=false
deadline=$((SECONDS + 900))
while (( SECONDS < deadline )); do
  observation="$(
    curl --fail --silent --show-error \
      -H 'Content-Type: application/json' \
      --data '{"query":"{ nodesInfo { nodes { id status sessionCount } } }"}' \
      "$SELENIUM_GRID_URL/graphql" |
    python3 -c '
import json, os, sys
doc = json.load(sys.stdin)
target = os.environ["NODE_ID"]
nodes = doc["data"]["nodesInfo"]["nodes"]
node = next((item for item in nodes if item["id"] == target), None)
if node is None:
    print("ABSENT absent")
else:
    print(str(node["status"]) + " " + str(node["sessionCount"]))
'
  )"
  status="${observation%% *}"
  count="${observation##* }"
  printf 'gridNode=%s status=%s sessions=%s\n' "$NODE_ID" "$status" "$count"

  if [ "$status" = "ABSENT" ]; then
    printf 'node_absent_without_drain_confirmation=%s\n' "$NODE_ID" >&2
    exit 3
  fi

  if [ "$status" = "DRAINING" ]; then
    saw_draining=true
  fi

  if [ "$saw_draining" = true ] && [ "$status" = "DRAINING" ] && [ "$count" = "0" ]; then
    printf 'node_ready_for_platform_removal=%s\n' "$NODE_ID"
    exit 0
  fi

  sleep 10
done

printf 'node_still_has_sessions=%s\n' "$NODE_ID" >&2
exit 1

The script has three distinct endings, and only one of them authorizes removal. Exit 0 means the drained Node ID was seen in DRAINING state holding zero sessions, which is positive evidence that admission stopped and nothing was lost. Exit 1 means sessions were still running at the deadline. Exit 3 means the Node left the Grid model before that confirmation arrived, and the controller must not convert that silence into success.

Disappearance is genuinely ambiguous, which is why it gets its own outcome rather than an early return. Selenium stops a drained Node once its last session ends, so a clean drain really does remove the Node from nodesInfo. A crashed Java process, a lost host, a network partition, and an operator deleting the Pod remove it in exactly the same way, and the GraphQL response looks identical in every case. A lookup that substitutes zero for a missing Node, or a next() call that assumes the Node is still there, reports all of those as a graceful completion. The first silently approves removal of capacity that may have died holding a live browser; the second raises StopIteration, fails the command substitution, and lets set -euo pipefail abort the script on its own success path.

Treat unexplained absence as a condition that needs operator attention. Hold the replica count, keep the last observation for the incident record, and let a human classify it. If the manual reviews become expensive, add a trusted completion signal instead of relaxing the predicate: a drain-complete log line, a container exit code, or an event published by a local Node wrapper all provide the terminal confirmation that mere absence does not. The cost of this position is slower scale-in and occasional review work. The cost of the alternative is a terminated test that the scale-in already reported as clean.

A maximum drain time is still necessary. One hung test can otherwise block infrastructure maintenance forever. When the deadline expires, choose between postponing removal and force-ending sessions. Do not make force termination the silent default. Record the session IDs, owning CI jobs, elapsed durations, and reason before applying it.

Cooldown should reflect session behavior. If the median test takes four minutes and arrivals come in five-minute waves, scaling down after sixty seconds guarantees churn. A cooldown slightly longer than the quiet gap may keep a Node idle, but avoids another cold start and image pull. Compare that cost with the price of the warm pod.

Node shutdown and Grid drain are separate facts. A Node can report draining while Kubernetes still runs it. Kubernetes can mark a pod terminating before Grid has drained it. Your controller must observe both systems and use stable identifiers to connect a Grid Node to a pod. Labels or an explicit registry are safer than matching transient IP addresses.

Prove the policy with bursts, skew, and failure injection

Replay the arrival shape you expect, not a flat stream. A constant rate can make any controller look calm. CI usually sends waves when a build stage completes or a matrix fans out. Capture historical new-session start timestamps, then reproduce representative bursts in a disposable Grid.

Measure from the client's call to session creation. For every attempt, record browser capabilities, queue-entry time if available, session ID, allocation time, failure, and cleanup result. On the supply side, record scale request time, pod scheduling time, Node readiness, Grid registration, and first successful session. Those timestamps reveal whether threshold tuning can help or the startup budget is impossible.

Run at least three cases:

1. a Chrome-only burst that exceeds the warm floor;
2. a mixed Chrome and Firefox burst with a heavily skewed ratio;
3. an unsupported capability that must alert without adding replicas.

Add a fourth case for platform delay. Prevent one new pod from scheduling or force a cold image pull. The controller should count pending supply, respect the pool cap, and surface that capacity is not becoming ready. It should not issue endless identical scale commands.

A controlled acceptance record might require 95 percent of supported requests to receive sessions within sixty seconds, no product test to be terminated during scale-in, and zero replica changes for unsupported requests. Tie the numbers to your delivery needs. A ten-minute end-to-end suite can tolerate a different wait than a two-minute pull-request check.

Roll out in report-only mode. Compare recommended replicas with what operators would have chosen. Then enable scale-out for one low-risk browser pool with a tight maximum. Keep scale-in manual until drain evidence is reliable. Expand to more pools only after the first one survives real browser upgrades and a CI cancellation wave.

For a suite already in daily use, sequence the prerequisites before granting write access. First inventory the capability combinations the suite actually sends and map each supported combination to exactly one capacity pool. Next land a versioned action record that joins raw queue snapshots, replica transitions, compatible registered-slot counts, session outcomes, and client timeouts on the same clock. Build a replay fixture from a known burst so a policy change can be checked against the same arrivals. Then restrict the controller identity to the first workload it is allowed to change. Stable Grid Node to workload identity and drain proof belong before any later permission to remove supply.

Capability classification is the first rollout boundary that can fail silently. A newly introduced browser version or platform constraint can be sent to the old pool, producing a valid replica increase that cannot serve the request. Make the report-only gate fail review when a supported request has no pool or maps to more than one pool. During the first automatic stage, success means matching sessions were created before the unchanged client deadline. A higher replica count, a lower queue count, or a green workload rollout is insufficient by itself.

Avoid changing suite concurrency or wait budgets in the same rollout. Holding them constant leaves a clean before-and-after comparison, but it also means known timeouts continue during the observation period. That is a specific delivery cost of a cautious rollout. If those failures are intolerable, use a scheduled warm floor for the affected window while the reactive policy remains report-only, and record that scheduled capacity separately so it is not credited to the controller.

The operational handoff should name decision ownership before automatic writes begin. QA automation owns the capability inventory and the client-side deadline evidence. The Grid operator owns stereotype, registration, and drain evidence. The platform team owns workload readiness, replica limits, and scheduling failures. Each scale incident should carry the policy version, raw request payload, target pool, old and desired replicas, pending workload state, compatible slot change, first successful allocation or timeout, and the team that accepted the next action. A screenshot of a queue chart cannot substitute for this record because it omits both matching intent and supply state.

Exercise policy boundaries, not only the happy path. Send one request below the threshold and confirm no action. Hold demand above the threshold long enough to trigger one addition, then verify the controller waits for that generation to register before adding again. Drive demand past the cap and confirm it alerts without exceeding the maximum. Finally, let the queue clear while sessions remain active and prove that no pod is removed until drain completes.

Run the same cases after restarting the controller. Pending scale state, cooldown timestamps, and pool caps must survive or be reconstructed safely. A stateless restart that forgets four starting pods can double the requested capacity on its first poll. Persisting more state increases controller complexity; reconstructing from Deployments and Grid registration increases recovery time. Choose one behavior and test it.

Separate controller failures from Grid failures. If GraphQL is unavailable, hold the last safe replica count and alert. Do not scale to zero because the queue query returned no data. If kubectl fails, preserve the intended decision and retry with bounded backoff. If request parsing fails, store the raw payload and quarantine it from automatic scaling.

Review cost with latency. Report warm-node hours, added replica-minutes, image pulls, p95 session-start time, timeout count, and terminated sessions. A policy that saves twenty pod-hours but adds fifteen minutes to every release may be a poor trade. A policy that halves wait time by holding fifty idle Nodes may be equally hard to justify.

Know when another control will work better

Use scheduled pre-scaling when the burst time is known and startup is slow. Nightly suites, regional business-hour checks, and release trains often have predictable arrival windows. Scale ahead of the first request, then use queue data to adjust for variation.

Use CI admission control when the Grid should not absorb unlimited fan-out. A matrix limit turns an uncontrolled spike into a bounded stream. Pipeline duration increases, but the tested application and browser hosts receive stable load. This is often safer than teaching the infrastructure to chase every retry storm.

Do not auto-scale for requests that no supported stereotype can match. Fix or reject the capability. A maximum replica setting limits cost, but it does not turn an impossible request into valid demand.

Do not scale browsers to solve long in-session test duration without checking the application. If login becomes slow and sessions occupy slots three times longer, more browsers can push the application harder. Diagnose session-hold time separately from queue and creation time.

Queue-based autoscaling does not catch a functional failure after a session has been allocated. A test can receive a browser immediately and still fail because the application returned the wrong result, test data was corrupt, or the browser crashed later in the scenario. Those failures require test, application, and in-session browser telemetry. A clean queue proves admission capacity, not test correctness.

Do not build a custom controller when your Grid deployment already creates a disposable Node per request and exposes a supported capacity control. Duplicate controllers can race, over-provision, and disagree about ownership. Pick one authority for desired supply.

Do not scale down a stateful or scarce browser environment merely because the queue is empty. Mobile devices, licensed desktops, and rare platform versions may have long provisioning times or manual recovery. A fixed warm pool with explicit scheduling can be more reliable.

Queue-based autoscaling pays for responsiveness with operational complexity. Per-browser pools multiply policy objects. Warm floors spend money while idle. Conservative cooldown holds excess nodes. Fast scale-in risks sessions. Deep observation and dry-run periods delay automation. Those costs are acceptable only when variable demand and infrastructure savings are large enough to justify another control loop.