Your load test hit the target and still proved nothing

The graph reaches the planned virtual-user count, the summary is green, and production still collapses during a flash sale. The script modeled a fixed set of users who wait for each response. Production receives new work while earlier requests are still waiting.

Choose the workload model before drawing the ramp

A ramp is an input schedule, not a performance requirement. It says how virtual users or iteration starts change over time. Thresholds state which observed outcomes are acceptable. Mixing those concepts creates a test that celebrates generating traffic without deciding whether the service handled useful work.

Start with the client behavior. A closed model uses a fixed or changing number of virtual users. Each virtual user begins its next iteration after the current one and any pacing delay finish. If responses slow, iteration throughput can fall naturally because the users are occupied. That is realistic for some interactive sessions.

An open model schedules iteration starts at a rate independent of response time, provided the load generator has enough virtual users available to execute them. Grafana documents ramping-arrival-rate as an open-model executor. It changes the target iteration rate across stages and dynamically uses allocated VUs. This is useful when arrivals continue even while the system slows, such as queued jobs, webhook deliveries, or independent customer checkouts.

Neither model is universally more realistic. A call-center agent may wait for one record before opening another, which fits a closed loop. Incoming events from many independent devices may keep arriving, which fits an open rate. A site visit can contain both: new sessions arrive independently, while each session follows a paced sequence.

The unit must match the question. Ten iterations per second is not ten requests per second when one iteration performs several requests. Twenty VUs is not a fixed arrival rate because iteration duration changes. Document the operation count and think time. Report offered iterations, generated requests, completed business outcomes, and failures separately.

Then choose the shape:

• A gradual ramp locates the region where behavior begins to change. It is useful for finding a capacity knee without jumping straight to severe overload.
• A stepped profile holds stable levels. It can reveal queue growth or resource drift that a continuously changing ramp hides.
• A spike adds demand quickly. It tests admission control, burst capacity, backlog, and recovery, not steady capacity by itself.
• A ramp down reduces offered load. Continued observation shows whether the system releases resources and drains work.
• A warm-up prepares caches, connections, runtime compilation, or replicas only when the experiment intentionally excludes cold behavior.

Warm-up data is not “bad data.” It answers a different question. If cold-start performance affects real users after deployments or scale-out, keep a separate cold case. If the requirement concerns an already active service, mark the warm-up interval and exclude it from the steady-state verdict by design, not because its values look inconvenient.

Stage duration comes from system dynamics. A queue needs enough time to reveal whether arrival and service rates are balanced. Autoscaling has evaluation, provisioning, and readiness intervals. Connection pools and caches stabilize on their own schedules. A plateau that ends before those mechanisms respond cannot support a steady-state claim.

Build a gradual profile that exposes the capacity knee

The first worked example uses a ramping arrival rate for a checkout API. The targets, durations, and thresholds below are illustrative configuration, not measurements or recommendations. Replace them with a reviewed workload model and product commitments.

import http from 'k6/http';
import { check } from 'k6';
import exec from 'k6/execution';

export const options = {
  discardResponseBodies: true,
  scenarios: {
    checkout_rate: {
      executor: 'ramping-arrival-rate',
      startRate: 2,
      timeUnit: '1s',
      preAllocatedVUs: 20,
      maxVUs: 80,
      stages: [
        { duration: '2m', target: 20 },
        { duration: '5m', target: 20 },
        { duration: '2m', target: 0 },
      ],
      tags: { profile: 'gradual-checkout' },
    },
  },
  thresholds: {
    'checks{scenario:checkout_rate}': ['rate>0.99'],
    'http_req_failed{scenario:checkout_rate}': ['rate<0.01'],
    'http_req_duration{scenario:checkout_rate}': ['p(95)<750'],
    'dropped_iterations{scenario:checkout_rate}': ['count==0'],
  },
};

export default function () {
  const response = http.post(
    __ENV.BASE_URL + '/api/test-checkout',
    JSON.stringify({
      cartId:
        'load-cart-' + exec.vu.idInTest + '-' + exec.scenario.iterationInTest,
    }),
    {
      headers: {
        'content-type': 'application/json',
        'x-load-test': 'qa-ramp-v1',
      },
      tags: { operation: 'checkout' },
    },
  );

  check(response, {
    'checkout accepted': (result) => result.status === 202,
  });
}

This endpoint name and status are application-specific. Point the script only at an approved test environment and use data rules the service supports. A unique cart per iteration may be correct for creation load, while a read scenario may need a controlled reusable pool. The data strategy changes cache behavior and database growth, so it belongs in the test contract.

The ramp has three different jobs. The first stage increases the scheduled arrival rate. The second holds it, giving queues and resources time to show a trend. The final stage stops new starts gradually. External telemetry should continue long enough to observe cleanup. k6's stage ending does not automatically prove a downstream queue is empty.

The thresholds are also illustrative. The check proves useful behavior instead of counting any HTTP response as success. http_req_failed provides protocol-level failure evidence. The duration percentile protects tail behavior better than an average alone. dropped_iterations guards the offered-load contract. Product and reliability owners must supply the actual acceptable values.

Do not abort this diagnostic at the first latency threshold failure if recovery is part of the question. k6 supports abortOnFail, but an abort can remove the ramp-down and observation period you needed. Fast aborts save environment cost during routine regression. Complete profiles provide better evidence during capacity discovery. Use separate modes rather than pretending one setting serves both.

A closed-model script answers a different question. Here virtual users browse, pause, and repeat. When the service slows, the iteration rate may fall because users remain busy. That feedback is part of the model:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    browsing_users: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 25 },
        { duration: '5m', target: 25 },
        { duration: '2m', target: 0 },
      ],
      gracefulRampDown: '30s',
      tags: { profile: 'paced-browsing' },
    },
  },
};

export default function () {
  const response = http.get(__ENV.BASE_URL + '/catalog', {
    tags: { operation: 'catalog' },
  });

  check(response, {
    'catalog returned successfully': (result) => result.status === 200,
  });

  sleep(1);
}

The one-second sleep is illustrative pacing. It is not a statement about real users. Derive pacing and path mix from research or production telemetry with privacy safeguards. Grafana's documentation advises against adding sleep to arrival-rate executors because those executors already schedule starts; sleep is meaningful here because the closed user journey includes pacing.

Run both models only when they answer distinct questions. Comparing them as if they offered identical traffic is misleading. Label the executor, intended arrivals, observed iteration rate, and response behavior in every result.

Use steps and spikes for failures a smooth ramp can hide

A smooth ramp can pass through a dangerous level too quickly. A step test holds several plateaus so the team can see whether latency, errors, queue depth, connections, memory, or completed throughput continue changing while offered load stays fixed. Stable input does not guarantee stable system state.

Design each plateau around a hypothesis. For example, “At the reviewed arrival rate, completed checkouts should keep pace with starts and the queue should not grow across the hold.” The queue metric may live in service telemetry rather than k6. Correlate stage timestamps with that telemetry and preserve the deployment configuration.

If queue depth rises throughout a plateau while request acceptance remains fast, the front door may be acknowledging work faster than workers complete it. An HTTP latency threshold alone can pass while customer outcomes fall behind. Measure the terminal business event or queue drain, not just the enqueue response.

The near-miss is a batch job intentionally building a bounded queue. In that design, queue growth during intake may be expected and recovery time may be the requirement. The same graph has a different verdict because the contract differs. Name the service model before calling any backlog a defect.

A spike needs a clean baseline before and after the burst. One way to express this in k6 is a low constant arrival-rate scenario plus a second scenario that overlaps it for a short period. After the burst scenario ends, the baseline continues, providing requests that reveal user-facing recovery. The following values are illustrative.

import http from 'k6/http';
import { check } from 'k6';
import exec from 'k6/execution';

export const options = {
  discardResponseBodies: true,
  scenarios: {
    recovery_probe: {
      executor: 'constant-arrival-rate',
      rate: 5,
      timeUnit: '1s',
      duration: '12m',
      preAllocatedVUs: 10,
      maxVUs: 30,
      exec: 'checkout',
      tags: { profile: 'recovery-probe' },
    },
    short_burst: {
      executor: 'constant-arrival-rate',
      rate: 95,
      timeUnit: '1s',
      duration: '30s',
      startTime: '5m',
      preAllocatedVUs: 50,
      maxVUs: 150,
      exec: 'checkout',
      tags: { profile: 'short-burst' },
    },
  },
  thresholds: {
    'dropped_iterations{scenario:recovery_probe}': ['count==0'],
    'dropped_iterations{scenario:short_burst}': ['count==0'],
  },
};

export function checkout() {
  const response = http.post(
    __ENV.BASE_URL + '/api/test-checkout',
    JSON.stringify({
      cartId:
        'burst-' + exec.vu.idInTest + '-' + exec.scenario.iterationInTest,
    }),
    {
      headers: {
        'content-type': 'application/json',
        'x-load-test': 'qa-spike-v1',
      },
    },
  );

  check(response, {
    'request reached an expected response class': (result) =>
      result.status === 202 || result.status === 429,
  });
}

Accepting 429 in this illustrative check does not declare throttling successful. It distinguishes an expected admission-control response class from a network failure. Add product-owned limits for how many requests may be rejected and what callers should do. Also measure whether accepted work completes.

The system may behave badly after the burst ends. Queues can remain deep, connection pools can stay exhausted, retries can amplify load, and autoscaled replicas can take time to become useful. The recovery probe shows whether low-rate user requests return to the agreed behavior. Service telemetry shows whether background state returns.

Do not set the post-burst rate to zero if your only user-facing recovery signal comes from requests. Zero load can help measure queue drain, but it cannot show what a user receives. Use separate low-rate probes and passive telemetry where appropriate.

Ramp-down behavior has similar value. Grafana documents gracefulRampDown for ramping VUs as time already-started iterations may finish while VUs are reduced. Setting it to zero can interrupt iterations. Interrupted work can contaminate the recovery result, so choose the setting deliberately and record it.

Prove the generator delivered the planned load

The target graph can look correct while the driver falls behind. In arrival-rate executors, k6 emits dropped_iterations when an iteration could not start because no VU was available. Grafana's built-in metrics reference also exposes iterations, vus, and vus_max. Read them together.

A dropped iteration does not automatically mean the service is at capacity. maxVUs may be too low. The generator may lack CPU, memory, file descriptors, sockets, or network bandwidth. Test data generation may block. A script may parse large bodies or perform expensive JavaScript work. Monitor every load-generator host.

Pre-allocation is part of an arrival-rate test design. k6 needs enough VUs to cover the target start rate multiplied by iteration duration, with margin for variability. As the system slows, more VUs are occupied. Raising maxVUs can preserve arrivals, but it consumes more generator resources and may reveal a deeper target slowdown. Do not hide dropped iterations by blindly allocating an enormous number.

Compare intended and observed work:

• The scenario configuration states the scheduled iteration rate.
• iterations records work that started and completed the script loop.
• dropped_iterations records starts the executor could not make.
• checks show whether responses met the functional oracle.
• http_req_failed separates failed HTTP requests under k6's response classification.
• service telemetry shows accepted, queued, completed, rejected, and retried business operations.

Request count can exceed iteration count because one iteration may make several calls. A retry inside the application or gateway may create additional downstream load that the script's top-level count cannot see. Name metrics by business operation and correlate at the service.

Generator saturation often looks like a service plateau. Offered throughput stops rising, but target CPU and queue depth remain flat. The load-generator CPU is saturated or dropped_iterations increase. A true service knee has evidence at the target: latency or errors change, queues grow, resource limits appear, or completed throughput stops scaling while the driver remains healthy.

Network placement can create another ceiling. A single generator link, NAT gateway, or ephemeral-port pool may limit connections. Distribute load only after proving one generator is the limit and after designing result aggregation. Multiple generators add clock alignment, data partitioning, and operational cost.

A connection-slot ceiling deserves its own comparison because its terminal summary can resemble service saturation. Both cases can show longer iterations, more active VUs, a plateau in completed work, and eventually dropped iterations. The root causes sit on opposite sides of the request boundary.

Read http_req_blocked, http_req_connecting, http_req_tls_handshaking, and http_req_waiting by stage. In a healthy warm stage, blocked and connection components stay near that run's established baseline while the target receives the intended shape. Under a generator-side connection ceiling, blocked or connection time grows as the stage rises, but target ingress never observes the missing attempts. Under service saturation, the driver still establishes requests and target ingress sees them, while waiting time and a server queue or resource signal grow together. A second generator on an independent network path is a useful controlled comparison only when it sends the same operation and data mix.

http_req_duration is the misleading field in the generator-side case. k6 defines it from sending, waiting, and receiving time, without the initial lookup and connection phases. It can remain close to the healthy range while connection setup or a wait for a free connection slot stretches the full iteration. Conversely, vus reaching vus_max does not prove the driver is faulty. Slow target responses can keep VUs occupied until the same limit is reached. The decisive evidence is whether attempts crossed the target ingress boundary and which timing component expanded before drops appeared.

An illustrative output shape makes the distinction concrete. Suppose a stage schedules a steady set of starts. In the healthy record, dropped iterations remain zero, active VUs remain below the available maximum, connection components remain flat, and the service's received-operation series follows the scheduled shape. In the broken driver record, active VUs climb toward the maximum, blocked or connection time rises, and service ingress falls short before server resources change. In the broken service record, ingress follows the offered shape, waiting and queue depth rise, and completed business outcomes stop keeping pace. These are illustrative relationships, not measured rates or universal thresholds.

Run a smoke profile before the full test. Validate authentication, data uniqueness, checks, tags, and cleanup at low load. Then run a generator calibration appropriate to the environment. Calibration is not a benchmark of the application; it proves the driver can produce the planned shape without becoming the first bottleneck.

Capture raw output rather than only the terminal summary. This shell wrapper creates a named result file and prevents an accidental run against an unapproved target. It assumes k6 is already installed.

#!/usr/bin/env bash
set -euo pipefail

: "${BASE_URL:?set BASE_URL to the approved test environment}"
: "${RUN_ID:?set RUN_ID to a unique nonsecret identifier}"
: "${1:?usage: run-k6.sh path/to/approved-script.js}"

test_script="$1"

case "$BASE_URL" in
  https://perf.example.test|https://staging-perf.example.test)
    ;;
  *)
    echo "Refusing to load test an unapproved target: $BASE_URL" >&2
    exit 2
    ;;
esac

case "$test_script" in
  performance/gradual-checkout.js|performance/spike-recovery.js)
    ;;
  *)
    echo "Refusing to run an unapproved script: $test_script" >&2
    exit 2
    ;;
esac

mkdir -p load-results

BASE_URL="$BASE_URL" k6 run --out "json=load-results/${RUN_ID}.json" "$test_script"

Replace the allowlist with domains your organization controls. A safety check is worth the small maintenance cost. Keep credentials outside the command line and result file. Load-test output can contain URLs, tags, and error details, so review it before broad artifact retention.

Distinguish service failure from a bad experiment

Data collisions are common. Hundreds of virtual users reuse one account or cart, triggering locks, rate limits, cache hits, or optimistic-concurrency failures that real users would not share. Sometimes contention is exactly the risk being tested. Usually it is accidental. State whether data is unique per user, per iteration, or deliberately shared.

Authentication can become the workload by mistake. If every iteration logs in, the identity service receives more traffic than the business flow. Real clients may reuse sessions. Put login in setup or a per-VU lifecycle only when that matches behavior, and measure authentication separately when it has its own capacity requirement.

Cold starts can dominate the first stage. Do not delete them from the record. Label cold and warm intervals. A deployment-time test may intentionally include cold replicas. A steady capacity test may include a documented warm-up. Both are legitimate when named honestly.

Error aggregation can hide the transition. One overall percentage may pass even if every request in a short spike failed. Tag scenarios and operations, then apply thresholds and graphs to those subsets. Keep status codes and error classes. Controlled 429 responses, timeouts, connection errors, functional check failures, and server 500 responses are not interchangeable.

Percentiles across the entire run can also blur stage behavior. A long low-load period may dominate a brief overloaded stage. Stream tagged metrics to a backend that preserves timestamps, or separate profiles into runs when stage-level verdicts are essential. Do not infer the capacity knee from one end-of-test percentile.

Autoscaling changes the system during the experiment. Record desired and ready replicas, scale events, and resource limits. A ramp may be slower than the scaler and show a smooth result; a spike may outrun it. That difference is useful if both traffic shapes occur in production.

Retries amplify offered load. Client, proxy, and service retries can turn a moderate arrival increase into a downstream burst. Record retry policy and count attempts at each layer. A k6 iteration that eventually passes can still have generated several backend operations.

Automatic reruns are new experiments. The first run warms caches, grows pools, scales replicas, and consumes data. Preserve it. Restore the documented start state before another run. Never concatenate both JSON outputs and calculate one percentile unless the analysis plan explicitly calls for it.

Roll out profiles without turning CI into a load lab

Start with one representative operation and a low-cost smoke profile. Prove the request is valid, the check detects a bad response, tags are useful, data cleanup works, and the target allowlist stops mistakes. This belongs in ordinary CI if the environment can tolerate it.

An existing suite needs a migration order. Land bounded scenario and operation labels before changing the load shape so old and new runs can still be separated. Then preserve timestamped raw output and the deployment, generator, and fixture identities needed to interpret it. Next add the new profile as observation-only and run it from the approved generator pool. Validate the scheduled start shape against target ingress, accounting for each iteration's request count, before attaching latency or error verdicts. Add reviewed thresholds only after the functional check has been forced to fail against a known bad response and the cleanup path has survived an interrupted run.

Existing fixtures usually break before the service does. Shared accounts encounter authentication policy, carts collide, finite data pools empty, and cleanup jobs overlap the next run. CI time limits can also end the recovery window even though k6 itself behaved correctly. Detect those failures in the smoke profile, not at the top of a capacity ramp. Keep pull-request coverage focused on script validity and a small functional oracle. Move sustained and destructive shapes to a controlled schedule or manual approval, with a result status that distinguishes product failure from invalid experiment.

Add a short regression profile for reviewed boundaries. Keep it small enough to run consistently, but long enough for the mechanism it claims to observe. Use product-owned thresholds. A short pull-request job cannot prove endurance, long autoscaling behavior, or full production capacity, so do not label it that way.

Schedule gradual capacity, spike, and recovery profiles in an isolated environment with observability. Notify service owners. Protect test data and credentials. Define a stop procedure. Store script revision, deployment revision, environment configuration, stage schedule, generator inventory, and result locations.

The CI wiring should make destructive scale an explicit choice. This example runs only on manual dispatch and requires a named profile. The actual k6 script still owns the target allowlist and safe defaults.

name: controlled-load-test

on:
  workflow_dispatch:
    inputs:
      profile:
        description: Reviewed profile name
        required: true
        type: choice
        options:
          - gradual-checkout
          - spike-recovery

permissions:
  contents: read

jobs:
  run-profile:
    runs-on: ubuntu-latest
    environment: performance-test
    steps:
      - uses: actions/checkout@v4
      - name: Run the selected approved script
        env:
          BASE_URL: ${{ vars.PERFORMANCE_BASE_URL }}
          RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
        run: |
          case "${{ inputs.profile }}" in
            gradual-checkout)
              performance/run-k6.sh performance/gradual-checkout.js
              ;;
            spike-recovery)
              performance/run-k6.sh performance/spike-recovery.js
              ;;
          esac

Environment approval adds delay. Isolation costs infrastructure. Long retention costs storage. Detailed telemetry requires engineering work. Those costs are preferable to running ambiguous high load, but they should appear in the test plan.

Generator headroom has a concrete cost too. More VUs and more generator hosts consume compute, create more concurrent connections, and can drive a mistaken profile farther into overload before an operator intervenes. Rich stage-level labels improve diagnosis, but unbounded labels such as unique cart or user identifiers multiply time series and storage. Keep identifiers in protected run records and metric labels bounded to reviewed dimensions. The safer profile is more expensive to operate and maintain, so its owner needs a budget for environment time, fixture reset, and telemetry retention.

Performance QA owns the workload model, script behavior, and proof that the driver delivered it. The service owner owns target telemetry, business completion, and a safe stop condition. Platform or network engineering owns generator hosts and the path to the target. Product and reliability owners approve the outcome and recovery limits. The test-data owner supplies reset and privacy rules. When evidence crosses teams, the handoff must include script revision, stage schedule, generator inventory and location, deployment revision, fixture policy, intended starts, completed iterations, dropped iterations, the relevant HTTP timing components, target ingress, completed outcomes, error classes, scale events, and recovery state. It must identify the first timestamp where the series diverged.

A ramp profile does not catch a slow leak that appears only after many hours at ordinary traffic. Memory fragmentation, file-descriptor leakage, gradual database growth, and log-volume exhaustion can remain invisible during a short capacity experiment. That requires a separately bounded endurance test with its own cleanup and observation window. Extending every ramp until it becomes a soak would increase environment occupancy and data growth while making capacity comparisons harder to reproduce.

Review results with the stage schedule visible. Mark the first point where service behavior changes, the evidence that the generator remained healthy, and the recovery condition. If the experiment cannot answer its hypothesis, report it as inconclusive rather than forcing a pass or fail.

Know when a ramp is the wrong experiment

Do not use a slow ramp to claim burst resilience. It gives caches, pools, and autoscaling time to adapt. Run a bounded spike with a recovery window when sudden arrivals are the requirement.

Do not use a spike to find sustainable capacity. A service may absorb a brief burst using queues and then fail under a long plateau. Use gradual or stepped steady load for capacity, then a separate spike for burst behavior.

Do not run protocol-scale load through a browser unless browser concurrency is itself the question. Browser instances consume far more generator resources and introduce rendering variability. Use HTTP or protocol clients for service load and a small browser journey for frontend performance.

Do not treat thresholds copied from a tutorial as product requirements. Example values teach syntax. They do not know your users, service level objectives, environment, or business cost. Obtain reviewed criteria and record their owner.

Do not ramp an unisolated shared environment to discover its breaking point. Other teams and users become part of the experiment. Use a controlled target, a bounded safety limit, and an approved stop condition.

Do not declare success from a green average. Verify useful outcomes, tail behavior, stage-local errors, offered load, generator health, target telemetry, and recovery. A test that cannot prove what traffic arrived or what work completed has measured its own script more than the system.