A clean CI log does not prove your secrets are safe

A forked pull request changes a package install script, and the privileged workflow runs it before the first test starts. The log shows three asterisks where the staging key would have appeared, so the run looks safe. It is not safe: contributor-controlled code already received the credential.

Draw the trust boundary before testing redaction

A secret can fail in two different ways. The first is an access failure: code, an action, or a person receives a credential it should never be able to read. The second is a disclosure failure: the value reaches a log, artifact, cache, report, process argument, or external service. Redaction addresses one narrow disclosure route. It cannot repair an access failure.

Start with the event that created the run. A normal fork pull request and a privileged pull request event do not have the same security context. GitHub documents that repository secrets, apart from the restricted GITHUB_TOKEN, are not passed to workflows triggered from forks. GitHub also warns against using pull_request_target or workflow_run to check out and execute untrusted pull request code. Those facts belong together. A team can defeat the safe default by choosing a privileged event and then deliberately bringing attacker-controlled code into that job.

Next, identify the exact revision each step executes. The workflow file may come from the base branch while the checkout step fetches the contributor's head commit. Package managers can execute lifecycle hooks during installation. Test runners load configuration, reporters, fixtures, and plugins before they print the first test name. A review that begins at the test command has already skipped several execution points.

Then list every credential-bearing capability. Include explicit repository or environment secrets, the automatically created GITHUB_TOKEN, cloud identity obtained through OIDC, package registry configuration, SSH agents, service containers, and credentials already present on a self-hosted runner. Do not treat the absence of a secrets block as proof that the job is unprivileged. A token can be available through platform context or runner state.

Finally, follow retained output. GitHub's masking is performed on the runner and is designed to redact recognized values from workflow logs. The secure-use documentation says automatic redaction is not guaranteed for transformed values. It recommends registering generated or encoded sensitive values too. A Base64 string is encoding, not encryption. Splitting a value across fields, serializing it inside structured data, or writing it to a file may also evade a scanner that only searches for the original byte sequence.

This unsafe pattern makes the trust error visible. Do not run it with a real credential. It is shown so reviewers can recognize the combination during code review:

name: unsafe-privileged-test

on:
  pull_request_target:

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      - run: npm ci
        env:
          STAGING_API_KEY: ${{ secrets.STAGING_API_KEY }}
      - run: npm test

The dangerous line is not the test command. It is the combination of a privileged event, an untrusted checkout, arbitrary project execution, and a credential in the same job. Replacing npm test with a different runner does not change the boundary. Adding a mask command does not change it either.

Write the boundary as a small table for every privileged workflow: who starts it, which revision supplies the workflow, which revision supplies executed code, what the job can read, what it can write, and who can retrieve its output. If any cell says “pull request author controls this” while another says “production or staging credential available,” the design fails before dynamic testing begins.

Reproduce leak paths with a harmless canary

Never test this control by placing a production key in a deliberately hostile job. Use a synthetic canary with no authority. It should be unique enough that a match is meaningful, shaped like the class of value under test, and short-lived. A random marker that cannot authenticate anywhere is safer than a real service token with a reduced scope.

Keep a safe case identifier separate from the raw canary. For example, the report can say CANARY-2026-08-04-07 and retain a SHA-256 digest. The scanner needs the raw value during the controlled run, but its output should name the representation and file, not echo the matching bytes. Avoid shell tracing. A well-intentioned set -x can print expanded commands and environment-derived arguments.

The first worked example checks retained files. It looks for the raw canary plus common deterministic representations. This is not a universal secret scanner. It is a precise oracle for a canary that the test generated. Save the program as scripts/scan_canary.py and point it at an unpacked directory of logs and artifacts.

from __future__ import annotations

import argparse
import base64
import hashlib
import os
import sys
from pathlib import Path
from urllib.parse import quote_plus


def representations(value: str) -> dict[str, bytes]:
    raw = value.encode("utf-8")
    return {
        "raw": raw,
        "base64": base64.b64encode(raw),
        "hex": raw.hex().encode("ascii"),
        "url-encoded": quote_plus(value).encode("ascii"),
    }


def scan(root: Path, canary: str) -> list[tuple[Path, str]]:
    findings: list[tuple[Path, str]] = []
    needles = representations(canary)

    for path in root.rglob("*"):
        if not path.is_file():
            continue
        try:
            data = path.read_bytes()
        except OSError as error:
            print(f"could not read {path}: {error}", file=sys.stderr)
            continue

        for label, needle in needles.items():
            if needle and needle in data:
                findings.append((path, label))

    return findings


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("root", type=Path)
    args = parser.parse_args()

    canary = os.environ.get("CI_CANARY", "")
    if len(canary) < 24:
        print("CI_CANARY must be a synthetic value of at least 24 characters", file=sys.stderr)
        return 2

    canary_id = hashlib.sha256(canary.encode("utf-8")).hexdigest()[:12]
    findings = scan(args.root, canary)

    for path, representation in findings:
        print(f"canary={canary_id} representation={representation} file={path}")

    return 1 if findings else 0


if __name__ == "__main__":
    raise SystemExit(main())

This script demonstrates an important limit. A canary test only detects representations you defined. It will not identify arbitrary encryption, partial fragments, screenshots, OCR-visible text, or data sent over a network. Add a representation only when it maps to a credible failure path. Generating hundreds of mutations creates noise and can turn a focused regression into a weak general-purpose scanner.

The second worked example collects retained material from one completed GitHub Actions run. The GitHub CLI commands retrieve the log and artifacts into a case directory, then the Python scanner checks both. Run it from a trusted workstation or a separate analysis job whose access is restricted. Do not enable xtrace.

#!/usr/bin/env bash
set -euo pipefail

: "${RUN_ID:?set RUN_ID to the controlled workflow run id}"
: "${CI_CANARY:?set CI_CANARY to the synthetic canary value}"

case_dir="${TMPDIR:-/tmp}/ci-canary-${RUN_ID}"
mkdir -p "$case_dir/logs" "$case_dir/artifacts"

gh run view "$RUN_ID" --log > "$case_dir/logs/workflow.log"
gh run download "$RUN_ID" --dir "$case_dir/artifacts"

CI_CANARY="$CI_CANARY"   python3 scripts/scan_canary.py "$case_dir"

A nonzero result is evidence of retained disclosure, not evidence of the original access route. Correlate the file with the workflow step that created it. A report may contain a token because the application returned it, because the test printed request headers, or because the upload path included an environment dump. Those causes require different fixes.

The third example should exercise a different failure: a report generator receives the canary through the environment even though it never prints it. Create a controlled fixture that writes selected environment variables to a JSON file, run it only in the isolated canary repository, and verify the scanner fails on that artifact. This proves the artifact gate itself works. It does not justify handing the canary to untrusted code in a production repository.

A fourth useful case targets command arguments. Many tools include their complete invocation in failure output or process diagnostics. Pass a harmless identifier on the command line and the canary through standard input or an environment variable only if the target tool supports that route. GitHub's secret guidance recommends avoiding command-line secret arguments where possible because process listings and audit systems may capture them. The expected result is both functional success and no canary in the retained command record.

Read the evidence before blaming masking

Asterisks in one line prove only that one rendered value matched a registered mask. They do not prove that the job lacked access, that transformed output was caught, or that an uploaded file is clean. Treat masking as a defense against accidental log output, not as an authorization system.

The job log usually gives three useful clues. First, the event name and ref tell you why the run started and which context it used. Second, the checkout output identifies the fetched revision. Third, step boundaries show which action or command ran before the first suspicious output. Preserve those details with the workflow commit. A later edit to the YAML can make the same run look safer than it was.

GitHub's command log can show a value as masked while surrounding text reveals its type, length, account, or endpoint. That partial disclosure may still matter. A database URL with the password removed can expose internal hostnames. A cloud access key identifier can help an attacker even without the secret half. Define which metadata is allowed instead of declaring every line containing a mask safe.

Artifacts need their own inventory. Record each artifact name, producing step, included paths, retention setting, and audience. Download and unpack archives before scanning. Searching a zip file as a single binary blob can miss compressed content. Browser traces, screenshots, HAR files, JUnit XML, coverage HTML, and crash dumps deserve special attention because they collect data indirectly.

Caches are harder. A cache is not merely a faster artifact. Later runs may restore it under a broader trust context, and a cache key can sometimes be influenced by repository content. The GitHub secure-use guidance warns that privileged workflows share cache risk when they execute untrusted code. Prefer not to place credential-derived files in a cache at all. If a test must evaluate cache content, use an isolated repository and synthetic data, then destroy the cache through the platform's supported controls.

Network egress is the blind spot in many “clean output” conclusions. A malicious or compromised dependency does not need to print a credential. It can make an outbound request. Runner or cloud audit logs, an egress proxy, and service-side canary telemetry can show that attempt. Absence from GitHub logs therefore cannot establish absence of exfiltration. If the threat model includes hostile code, the primary fix is to remove the credential from that job.

Several near-misses look similar:

• An empty secret expands to an empty string because the event is not allowed to receive it. The job may fail authentication, but there was no credential to leak. Confirm secret availability without printing it, such as by testing whether a variable is empty and emitting only a boolean.
• A scanner flags a sample token committed in documentation. Compare the exact canary digest and file provenance. Pattern resemblance alone is not a confirmed leak.
• A masked line appears in a trusted deployment job. The output defect still needs correction, but it is not proof that a fork author executed code in that context.
• An artifact contains a public client identifier. Classification decides whether it is sensitive. Names such as “key” and “token” are hints, not security boundaries.
• A rerun is clean because the secret was rotated or the branch changed. Preserve the first attempt. Exposure is not canceled by a later pass.

The decisive evidence for an access failure is that a disallowed principal or code path could read or use the canary. The decisive evidence for a retained disclosure is an exact canary representation in output available beyond the intended process. Report both findings separately. That distinction tells the owner whether to redesign the workflow, repair output handling, or do both.

Separate look-alike failures by execution order

A masked field followed by an authentication failure can also come from a third-party action that ran before checkout and received a job-scoped value. That differs from contributor code reading the value during installation or tests, although both logs can end with the same asterisks and failed request. Redaction hides the value, not the emitting process.

Use the original run's step order as the discriminator. If the first service-side canary event or retained match predates checkout of the untrusted revision, that revision cannot explain it. Record the workflow commit, declared action reference, action step, and whether the credential was scoped to the whole job or a later step. If the occurrence follows executable project setup and the earlier action never received the value, the evidence points in the other direction. Preserve the resolved action revision when available, because an action name alone is weak provenance.

The scanner output has three fields worth reading. The canary field is the safe digest prefix that ties the finding to one controlled value. The representation field says whether the matched bytes were raw, Base64, hexadecimal, or URL-encoded. The file field names the retained object that must be traced back to its producer. A broken run prints at least one such finding and exits nonzero. A healthy run prints no finding and exits zero, but only after the collection record confirms that the expected log was retrieved and every expected archive was expanded.

Two results can mislead. The script reports an unreadable file on standard error and continues, so zero after that warning is not a clean verdict. A match can also belong to an old leaky fixture in the scan directory. Compare the digest prefix with the current case and verify that the file came from the current run and attempt. A path alone is not provenance.

Fix access first, then contain output

Split untrusted validation from privileged work. The pull request job should run with the minimum GITHUB_TOKEN permissions, no repository or environment secrets, and no access to a persistent privileged runner. A later deployment or integration job should consume only a reviewed, immutable result produced by trusted code. Do not pass an executable workspace from the untrusted job into the privileged one and assume the split is safe.

The following workflow is deliberately small. It runs contributor tests on the pull_request event without a configured secret. A separate manually invoked job uses an environment-protected synthetic canary and checks trusted code from the default branch. The canary job is a control test, not part of the contributor's test path.

name: trust-separated-ci

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  pull-request-tests:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci --ignore-scripts
      - run: npm test

  controlled-canary-check:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    environment: ci-canary-test
    env:
      CI_CANARY: ${{ secrets.CI_CANARY }}
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.repository.default_branch }}
      - name: Register the synthetic value for log masking
        shell: bash
        run: |
          echo "::add-mask::$CI_CANARY"
      - name: Exercise and scan controlled fixtures
        shell: bash
        run: |
          mkdir -p test-output
          printf 'mask check: %s\n' "$CI_CANARY"
          printf '%s\n' 'controlled report without credentials' > test-output/report.txt
          python3 scripts/scan_canary.py test-output

The ignore-scripts option in this example is not a universal fix. Some projects legitimately need install scripts, and test configuration itself remains executable. The core protection is the absence of privileged credentials from the untrusted job. If the suite requires package scripts, allow them only after reviewing the risk and still keep credentials out.

Reduce permissions explicitly. A read-only contents permission is a better default than an implicit broad token, but even a read token can expose private repository content. Give each job only the scopes it needs. Protect environment secrets with reviewers where the workflow warrants human approval. Prefer short-lived cloud identity over stored long-lived cloud keys when the provider and deployment design support it.

Contain output after the boundary is correct. Register synthetic or generated sensitive values for masking before any command can emit them. Disable shell tracing around credential handling. Keep secrets out of command-line arguments. Configure reporters to omit request and response headers that are not needed. Upload an allowlisted set of files instead of an entire workspace. Apply the shortest useful retention and restrict readers.

Rotation is part of the fix for a real incident. Editing the workflow prevents another exposure but does not invalidate the credential already observed. Revoke or rotate first. Delete or restrict exposed logs and artifacts according to the platform's supported procedure. Review audit records for downloads and suspicious use. Do not paste the real value into the issue created to track remediation.

Roll the control into an existing pipeline

Begin with inventory, not a broad canary injection. Search workflow files for secret references, environment use, privileged events, third-party actions, custom scripts, artifact uploads, cache operations, and self-hosted runner labels. Map each secret to the smallest set of jobs that actually require it. Remove unused values before writing tests for them.

Next, create a non-production canary environment. Give its value no service authority. If format validation requires a token-shaped string, issue it from a dedicated test system with a deny-all policy or immediate revocation. Record ownership, expiration, and destruction. The team running the test should be able to prove that disclosure cannot become service compromise.

Add static gates first. Review new privileged event use. Block direct checkout of pull request head code inside privileged jobs. Require explicit permissions. Pin external actions according to your organization's dependency policy. Static checks are fast and catch dangerous structure before a workflow runs, but they do not prove runtime output is clean.

Then add a controlled output regression. Start with one intentionally leaky fixture and one clean fixture. The leaky fixture proves the scanner fails. The clean fixture proves normal reports do not create false alarms. Scan logs and artifacts after the run through a separate trusted process. Do not let the scanner's success promote code automatically until its own failure behavior has been tested.

Roll out artifact allowlists one producer at a time. Browser suites often upload a directory because it is convenient, then quietly add traces, videos, environment snapshots, and downloaded files over time. Name the files needed for diagnosis. If a trace is valuable, redact sensitive application data before retention or restrict access rather than pretending the trace contains nothing sensitive.

For an established suite, land collection coverage before accepting a clean scan, trust separation before placing the canary in a protected path, producer allowlists before blocking promotion, and enforcement last. Collection works when the inventory contains every expected log and artifact. Separation works when the fork path sees no canary while the trusted control still executes. Enforcement works when downstream promotion cannot proceed after either a finding or a collection error. Combining these changes makes collector gaps, unsafe design, and real disclosure look like one failure.

Expect evidence collection on failed tests to break first. A producer can stop before its later upload runs, leaving the scanner an empty directory that looks clean. Treat missing expected input as an invalid verdict and arrange collection to run while preserving the original test failure. The cost is more retained output and I/O from failed runs. Moving integration tests out of pull requests also delays that coverage until the trusted stage, while expanding every archive adds runner time and storage traffic.

Watch retries during migration. A first attempt can leak and a second attempt can pass after a setup branch changes. Scan every attempt independently. Label the run number and attempt in the case record. Never merge logs and then report only the final status.

Assign owners for the two verdicts. The workflow owner handles trust and permissions. The producing test or application owner handles unsafe output. The platform or security owner handles retention, access review, and credential response. One ticket may involve all three, but a single vague “CI issue” label tends to leave the access failure unfixed.

Use one parent incident owned by the workflow owner, with linked remediation for the producing team and platform or security team. The handoff should contain the run and attempt, event and ref, workflow and checked-out commits, first producing step, canary digest prefix, representation, artifact path, output audience, retention state, and rotation decision. Keep the parent open until access, output handling, and retained copies each have an explicit verdict.

The cost is real. Isolated workflows add maintenance. Environment approval can delay a controlled test. Artifact scanning consumes storage and analysis time. Short-lived credentials require identity plumbing. Restricting diagnostic data can make ordinary failures harder to debug. State these costs in the rollout plan, then choose narrow controls that preserve the evidence engineers truly need.

Know when a canary test is the wrong tool

Do not inject a canary into a job whose design already gives hostile code a chance to read it. That experiment confirms a known architectural flaw while creating another value to manage. Remove the privileged capability first. Use code review and a static policy to prevent the unsafe combination from returning.

Do not use secret-pattern scanners to decide whether a value is confidential. Public package names, hashes, test fixtures, and publishable client keys can match generic rules. Classification must come from the credential owner and system design. The exact canary approach is useful because the expected value and channel are known.

Do not claim network safety from clean retained files. If untrusted code can run with a credential and unrestricted egress, it may send data away without leaving a local copy. Redesign the trust boundary, constrain egress where appropriate, and use service-side telemetry for synthetic canaries. Log scanning is one layer, not a proof of non-exfiltration.

This technique also does not catch an unrelated real credential that leaks during the same run. An exact canary scanner knows only the controlled value and its listed representations. A production token sourced from runner state, a different job, or a dependency can pass through the scan without matching. Keep broader secret detection and credential inventory as separate controls.

Do not put a production secret into a test so the marker is “realistic.” A canary needs the relevant shape and handling path, not real authority. If a third-party integration will only accept an active credential, use a dedicated tenant, minimum privileges, explicit approval, and immediate rotation. Often the safer answer is to test the boundary without calling that integration at all.

Do not disable useful logs globally because one field leaked. Remove or redact the sensitive field at its source, reduce artifact scope, and preserve enough context to diagnose ordinary failures. Blind pipelines cause teams to add unsafe debug output during incidents.

Finally, do not confuse a clean rerun with recovery. Once a credential reached unauthorized code or retained output, the event happened. The run can help verify the repair, but incident response still needs revocation, access review, and a regression that targets the original route.