Catch the pixels an AI image edit was never asked to change

A user asks the editor to remove a coffee cup, and the returned image also redraws the logo on the mug beside it. The result still looks polished, so a screenshot approval can miss the regression. The useful test is not whether the second image is attractive. It is whether the editor changed pixels and content that the request put outside its authority.

Write the contract before choosing a metric

An edit request describes a permitted change, not permission to regenerate the whole frame. That distinction gives a QA engineer something stronger than a general similarity score. The source image, the requested region, and the protected region form a spatial contract. The candidate may differ inside the requested region. Outside it, the acceptable difference depends on what the product promises.

Start by writing that promise in product language. A background-removal tool might promise that every foreground pixel remains unchanged after decoding. An inpainting feature might promise only that named objects outside a padded edit area retain their visible appearance. A generative restyle button may promise nothing about individual pixels, but it may still promise the same dimensions, alpha behavior, subject count, and placement of a protected logo. These are different products and should not share one threshold.

The mask needs the same care as the source. Record its width, height, origin, and meaning. In the examples below, a white mask pixel means that change is allowed and a black pixel means that the pixel is protected. That convention is local to the test harness. Do not assume a production editor uses the same convention. Convert the product's representation at the adapter boundary, save the converted mask as an artifact, and test the adapter with a tiny image whose coordinates are obvious.

Geometry checks come before visual checks. A candidate with a different width or height cannot be compared coordinate by coordinate. Neither can a mask produced for a preview thumbnail be applied directly to a full-resolution export. Rotation metadata creates another trap: two files can display with the same orientation in one viewer while exposing different pixel arrays to a decoder. Normalize orientation once, document where that happens, and retain the original metadata for diagnosis.

Compare decoded pixels rather than PNG file bytes. PNG is a lossless image format, but the datastream also contains filtering, compression, chunks, and optional ancillary information. Two encoders can represent the same pixel array with different bytes. A whole-file SHA-256 check is valuable when the requirement is byte-for-byte reproducible export, but it is a poor oracle for visible preservation. Decode first when the requirement concerns the image a user sees.

A browser-side adapter can inspect pixels through CanvasRenderingContext2D.getImageData(). With the rgba-unorm8 pixel format, the returned ImageData.data contains a one-dimensional RGBA sequence. That makes a tiny canvas useful for proving coordinate and mask conversion in a web client. It does not remove the need to compare the actual exported artifact, which may pass through additional processing after the canvas step.

Alpha needs an explicit rule too. An RGBA pixel contains color channels and an alpha channel, and PNG stores color values without premultiplying them by alpha. Raw RGB differences under a fully transparent pixel may have no visible effect. On the other hand, changing alpha while leaving RGB alone can expose or hide content. This comparator renders both images over black and white backgrounds before it measures visible channel differences. Using two contrasting backdrops makes transparency changes observable without pretending that one application background represents every display context.

Color management is a separate boundary. Converting an image to RGBA does not transform every source into one agreed color space. PNG can signal color through several chunks, including iCCP, cICP, sRGB, gAMA, and cHRM. The comparator below is deliberately conservative: it accepts PNG source and candidate files only, and it refuses to score them when their color-signaling chunks differ. Canonicalize such a pair with a color-managed step your team controls, then compare the canonical artifacts. A low-amplitude full-frame diff is not evidence of model drift until the comparison space is known.

There are at least three useful meanings of consistency. Spatial consistency asks whether changes stayed inside an allowed area. Contract consistency asks whether every output still obeys invariants such as dimensions, transparency, and protected content. Repeat-run consistency asks how often those invariants hold across multiple executions. Pixel identity across runs is a fourth and much stronger claim. Do not quietly substitute it for the other three simply because an exact comparison is easy to code.

A threshold is policy, not a fact discovered by the test library. Exact preservation may be right for a copy-paste compositor. It will be noisy for an editor that deliberately blends the edge of a mask. Add a reviewed halo around a feathered boundary, or classify that band separately from the protected interior. Any numerical examples below are illustrative policy values used to make the code executable. They are not measurements from a model, a customer workload, or The Testing Academy.

Finally, require the requested area to change when the operation should have a visible effect. A comparator that only protects the outside region will happily approve an untouched copy of the source. That oracle cannot catch a no-op, an upstream request drop, or an editor that returned the input after a timeout. A good contract has both sides: the protected area changed little enough, and the requested area changed enough to show that work happened. Semantic correctness still needs a separate check when any arbitrary change could satisfy the pixel rule.

Build an oracle that can fail for the right reason

The first implementation should produce evidence that survives a failed CI job. A single similarity score loses too much information. Record the count of protected pixels, the count that breached the channel tolerance, the fraction changed, the same values for the requested region, the largest protected-channel delta, and a bounding box around protected failures. Those fields answer different questions and make the next diagnostic step much faster.

The following module expects two PNG images and a grayscale mask with equal dimensions. It applies EXIF orientation through Pillow before comparison, refuses to score different PNG color metadata, converts the images to RGBA, composites them over black and white, and uses the larger per-channel difference from the two rendered views. A digest of the shared color-signaling chunks appears in the report. Mask values of 128 or greater are treated as editable. Lower values are protected. A production suite may need three mask classes instead, such as editable, feathered boundary, and strictly protected.

Save this as tests/image_contract.py. The CLI requires every policy value on purpose. Hiding tolerances in helper defaults makes it too easy for a threshold copied from one feature to govern another feature silently.

from __future__ import annotations

import argparse
import hashlib
import json
from dataclasses import asdict, dataclass
from pathlib import Path

from PIL import Image, ImageOps


@dataclass(frozen=True)
class ConsistencyResult:
    width: int
    height: int
    protected_pixels: int
    protected_changed: int
    requested_pixels: int
    requested_changed: int
    max_protected_delta: int
    protected_bbox: tuple[int, int, int, int] | None
    png_color_metadata_sha256: str | None

    @property
    def protected_changed_fraction(self) -> float:
        return self.protected_changed / self.protected_pixels

    @property
    def requested_changed_fraction(self) -> float:
        return self.requested_changed / self.requested_pixels


def load_oriented_rgba(path: Path) -> Image.Image:
    with Image.open(path) as opened:
        if opened.format != "PNG":
            raise ValueError(f"source and candidate images must be PNG: {path}")
        oriented = ImageOps.exif_transpose(opened)
        return oriented.convert("RGBA")


PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"
COLOR_CHUNK_TYPES = {
    b"cHRM",
    b"cICP",
    b"cLLI",
    b"gAMA",
    b"iCCP",
    b"mDCV",
    b"sBIT",
    b"sRGB",
}


def png_color_metadata_digest(path: Path) -> str | None:
    encoded = path.read_bytes()
    if not encoded.startswith(PNG_SIGNATURE):
        raise ValueError(f"source and candidate images must be PNG: {path}")

    offset = len(PNG_SIGNATURE)
    records: list[bytes] = []
    while offset + 12 <= len(encoded):
        length = int.from_bytes(encoded[offset : offset + 4], "big")
        chunk_type = encoded[offset + 4 : offset + 8]
        data_start = offset + 8
        data_end = data_start + length
        next_offset = data_end + 4
        if next_offset > len(encoded):
            raise ValueError(f"truncated PNG chunk in {path}")
        if chunk_type in COLOR_CHUNK_TYPES:
            records.append(chunk_type + length.to_bytes(4, "big") + encoded[data_start:data_end])
        offset = next_offset
        if chunk_type == b"IEND":
            payload = b"".join(sorted(records))
            return hashlib.sha256(payload).hexdigest() if records else None

    raise ValueError(f"PNG has no complete IEND chunk: {path}")


def flatten(image: Image.Image, background: tuple[int, int, int]) -> Image.Image:
    canvas = Image.new("RGBA", image.size, (*background, 255))
    return Image.alpha_composite(canvas, image).convert("RGB")


def evaluate(
    source_path: Path,
    candidate_path: Path,
    mask_path: Path,
    channel_tolerance: int,
) -> ConsistencyResult:
    if not 0 <= channel_tolerance <= 255:
        raise ValueError("channel_tolerance must be between 0 and 255")

    source = load_oriented_rgba(source_path)
    candidate = load_oriented_rgba(candidate_path)
    source_color_metadata = png_color_metadata_digest(source_path)
    candidate_color_metadata = png_color_metadata_digest(candidate_path)
    with Image.open(mask_path) as opened_mask:
        mask = opened_mask.convert("L")

    if source.size != candidate.size:
        raise ValueError(f"size mismatch: source={source.size}, candidate={candidate.size}")
    if source.size != mask.size:
        raise ValueError(f"mask mismatch: image={source.size}, mask={mask.size}")
    if source_color_metadata != candidate_color_metadata:
        raise ValueError(
            "PNG color metadata differs: canonicalize before comparison"
        )

    source_views = [flatten(source, (0, 0, 0)), flatten(source, (255, 255, 255))]
    candidate_views = [
        flatten(candidate, (0, 0, 0)),
        flatten(candidate, (255, 255, 255)),
    ]

    protected_pixels = protected_changed = 0
    requested_pixels = requested_changed = 0
    max_protected_delta = 0
    changed_bbox: tuple[int, int, int, int] | None = None

    source_pixels = [view.load() for view in source_views]
    candidate_pixels = [view.load() for view in candidate_views]
    mask_pixels = mask.load()
    for y in range(source.height):
        for x in range(source.width):
            delta = max(
                abs(source_pixels[view][x, y][channel]
                    - candidate_pixels[view][x, y][channel])
                for view in range(2)
                for channel in range(3)
            )
            if mask_pixels[x, y] >= 128:
                requested_pixels += 1
                requested_changed += delta > channel_tolerance
            else:
                protected_pixels += 1
                max_protected_delta = max(max_protected_delta, delta)
                if delta > channel_tolerance:
                    protected_changed += 1
                    if changed_bbox is None:
                        changed_bbox = (x, y, x + 1, y + 1)
                    else:
                        left, top, right, bottom = changed_bbox
                        changed_bbox = (
                            min(left, x),
                            min(top, y),
                            max(right, x + 1),
                            max(bottom, y + 1),
                        )

    if protected_pixels == 0:
        raise ValueError("mask contains no protected pixels")
    if requested_pixels == 0:
        raise ValueError("mask contains no requested pixels")

    return ConsistencyResult(
        width=source.width,
        height=source.height,
        protected_pixels=protected_pixels,
        protected_changed=protected_changed,
        requested_pixels=requested_pixels,
        requested_changed=requested_changed,
        max_protected_delta=max_protected_delta,
        protected_bbox=changed_bbox,
        png_color_metadata_sha256=source_color_metadata,
    )


def violations(
    result: ConsistencyResult,
    max_protected_fraction: float,
    min_requested_fraction: float,
) -> list[str]:
    if not 0.0 <= max_protected_fraction <= 1.0:
        raise ValueError("max_protected_fraction must be between 0 and 1")
    if not 0.0 <= min_requested_fraction <= 1.0:
        raise ValueError("min_requested_fraction must be between 0 and 1")
    errors: list[str] = []
    if result.protected_changed_fraction > max_protected_fraction:
        errors.append(
            "protected region changed: "
            f"{result.protected_changed}/{result.protected_pixels} pixels "
            f"({result.protected_changed_fraction:.6f}) exceeds "
            f"{max_protected_fraction:.6f}"
        )
    if result.requested_changed_fraction < min_requested_fraction:
        errors.append(
            "requested region changed too little: "
            f"{result.requested_changed}/{result.requested_pixels} pixels "
            f"({result.requested_changed_fraction:.6f}) is below "
            f"{min_requested_fraction:.6f}"
        )
    return errors


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("source", type=Path)
    parser.add_argument("candidate", type=Path)
    parser.add_argument("mask", type=Path)
    parser.add_argument("--channel-tolerance", type=int, required=True)
    parser.add_argument("--max-protected-fraction", type=float, required=True)
    parser.add_argument("--min-requested-fraction", type=float, required=True)
    args = parser.parse_args()

    result = evaluate(args.source, args.candidate, args.mask, args.channel_tolerance)
    errors = violations(result, args.max_protected_fraction, args.min_requested_fraction)
    report = asdict(result) | {
        "protected_changed_fraction": result.protected_changed_fraction,
        "requested_changed_fraction": result.requested_changed_fraction,
        "violations": errors,
    }
    print(json.dumps(report, indent=2))
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

Every assertion has a plausible falsifying change. Alter a protected pixel beyond tolerance and protected_changed_fraction rises. Return the source unchanged and requested_changed_fraction stays at zero. Resize the candidate or pass the wrong mask and evaluation stops before producing a misleading score. Give the mask no black or no white pixels and the harness rejects a contract that has lost one side of the comparison.

The two ratios should not be combined. A weighted total could let a large successful edit compensate for damage to a small protected logo. The release rule should preserve the direction of each requirement. In a strict export workflow, one changed protected pixel can fail even when millions of requested pixels changed correctly. In an inpainting workflow, a separately reviewed boundary band may permit limited drift while the interior protected region remains exact.

Channel tolerance and changed-pixel fraction solve different problems. The channel tolerance decides whether one decoded pixel differs enough to count. The fraction decides how much of a region may contain such pixels. Raising the first can hide faint damage everywhere. Raising the second can hide a small but severe change, such as a removed decimal point. Keep max_protected_delta and the bounding box in the report so a permissive fraction does not erase severity.

This oracle still does not know whether the coffee cup was removed. It only knows that something changed inside the authorized mask and that protected rendered pixels remained within policy. Add an operation-specific assertion for completion: alpha coverage for background removal, OCR for required text, a reviewed segmentation check for object removal, or a human decision for edits whose semantics resist deterministic checks. Keep that result separate so a team can distinguish containment failure from task-completion failure.

Exercise three failures that look similar in a gallery

A thumbnail gallery compresses evidence. A spill outside the mask, a no-op, and an alpha regression can all look like a vaguely disappointing edit at small size. They come from different boundaries and require different fixes. Synthetic fixtures are useful here because they let the test team prove the oracle itself before pointing it at expensive or nondeterministic outputs.

The next pytest file creates a 12 by 8 RGBA image. Its white rectangular mask authorizes a four by four edit. The passing case changes that rectangle only. The spill case changes the requested rectangle and one protected pixel. The no-op case returns the source. The alpha case performs the requested edit but changes the opacity of a protected pixel. The expected outcomes come from the stated spatial contract, not from values copied out of the comparator.

from pathlib import Path

import pytest
from PIL import Image, ImageDraw
from PIL.PngImagePlugin import PngInfo

from image_contract import evaluate, violations


def write_case(tmp_path: Path, mutation: str) -> tuple[Path, Path, Path]:
    source = Image.new("RGBA", (12, 8), (40, 80, 120, 255))
    source.putpixel((1, 1), (220, 30, 30, 128))

    mask = Image.new("L", source.size, 0)
    ImageDraw.Draw(mask).rectangle((4, 2, 7, 5), fill=255)

    candidate = source.copy()
    if mutation != "noop":
        ImageDraw.Draw(candidate).rectangle((4, 2, 7, 5), fill=(20, 190, 80, 255))
    if mutation == "spill":
        candidate.putpixel((10, 6), (255, 255, 0, 255))
    if mutation == "alpha":
        candidate.putpixel((1, 1), (220, 30, 30, 32))

    source_path = tmp_path / "source.png"
    candidate_path = tmp_path / f"candidate-{mutation}.png"
    mask_path = tmp_path / "mask.png"
    source.save(source_path)
    candidate.save(candidate_path)
    mask.save(mask_path)
    return source_path, candidate_path, mask_path


@pytest.mark.parametrize(
    ("mutation", "expected_fragment"),
    [
        pytest.param("good", None, id="requested-edit-only"),
        pytest.param("spill", "protected region changed", id="one-pixel-spill"),
        pytest.param("noop", "requested region changed too little", id="no-op"),
        pytest.param("alpha", "protected region changed", id="alpha-regression"),
    ],
)
def test_spatial_contract(
    tmp_path: Path,
    mutation: str,
    expected_fragment: str | None,
) -> None:
    source, candidate, mask = write_case(tmp_path, mutation)
    result = evaluate(source, candidate, mask, channel_tolerance=0)
    errors = violations(
        result,
        max_protected_fraction=0.0,
        min_requested_fraction=1.0,
    )

    if expected_fragment is None:
        assert errors == []
    else:
        assert len(errors) == 1, errors
        assert expected_fragment in errors[0]


def test_different_png_color_metadata_requires_canonicalization(
    tmp_path: Path,
) -> None:
    source = Image.new("RGBA", (2, 1), (40, 80, 120, 255))
    candidate = source.copy()
    mask = Image.new("L", source.size)
    mask.putdata([0, 255])

    source_path = tmp_path / "source-color.png"
    candidate_path = tmp_path / "candidate-color.png"
    mask_path = tmp_path / "mask-color.png"
    source.save(source_path)
    color_info = PngInfo()
    color_info.add(b"gAMA", (45_455).to_bytes(4, "big"))
    candidate.save(candidate_path, pnginfo=color_info)
    mask.save(mask_path)

    with pytest.raises(ValueError, match="PNG color metadata differs"):
        evaluate(source_path, candidate_path, mask_path, channel_tolerance=0)

The values in this fixture are constructed facts, not reported model performance. With an exact tolerance, the spill changes one protected pixel and the no-op changes none of the sixteen requested pixels. If a developer removes the protected-pixel mutation from the spill branch, that row stops failing. If the production comparator accidentally ignores alpha, the alpha row stops failing. If the requested-region check is deleted, the no-op row exposes the missing assertion. The final test gives identical RGBA samples different PNG gamma metadata, so deleting the metadata comparison makes that case stop failing. Those are useful mutation questions because each can break the test for a reason the contract cares about.

Running the spill artifact through the CLI returns exit status 1. For this synthetic 12 by 8 case, the JSON report contains protected_changed: 1, protected_pixels: 80, protected_bbox: [10, 6, 11, 7], and the exact violation protected region changed: 1/80 pixels (0.012500) exceeds 0.000000. Those values come directly from the constructed coordinate mutation and the executed script. They are not production measurements.

Failure one: the edit crosses a correct mask

Suppose the editor removes a person inside the selected polygon but also changes a sign twenty pixels away. The candidate dimensions match. The mask overlay aligns with the source. The protected diff contains a compact component around the sign rather than a uniform border around the selection. Replaying the same stored source, mask, and candidate produces the same component. That evidence supports a containment defect in the returned artifact or in a downstream compositing step.

Do not leap from that evidence to a claim about the model. First compare the artifact received from the editing service with the artifact delivered by the application. If the service artifact preserves the sign and the application export does not, the regression sits in decoding, resizing, color conversion, overlay composition, or encoding after the service call. If both artifacts contain the same protected change, the investigation can move upstream. Artifact identity at each boundary matters more than a generic trace status.

The fix may be to apply the original protected pixels after generation, expand the operation's internal context while clipping the final composite to the authorized mask, or reject an output that violates containment. Those options have different costs. Restoring original pixels creates a seam when lighting or texture inside the edit should blend across the edge. Clipping can expose a hard boundary. Regeneration adds latency and compute without guaranteeing the next attempt will pass. Product and design should choose the visible compromise; QA should make the compromise measurable.

Failure two: preservation passes because nothing happened

An unchanged source image has perfect protected-region consistency. That is why a preservation-only test is incomplete. A dropped request body, a failed worker that returns its input fallback, or a cache key that omits the edit instruction can all produce a byte-valid image with no protected drift. The requested-region minimum catches the shape of this failure, but it cannot prove that the requested object disappeared.

Evidence for a no-op is unusually clean. The requested diff count is zero or below the reviewed minimum. The protected count is also zero. The output may even have the same decoded pixels but different PNG bytes because it was re-encoded. Compare pixels first, then inspect application logs for the request identifier, the normalized mask artifact, and the returned artifact identifier. Do not use a changed file hash as proof that editing occurred.

The next oracle should match the operation. For background removal, verify the promised alpha state in background regions and preserve foreground pixels according to contract. For a crop-and-fill workflow, verify output geometry plus new content in the expanded canvas. For text replacement, run OCR as supporting evidence but retain a human-review lane for typography and small glyphs. A single universal image score will not carry these meanings safely.

Failure three: transparency changes while RGB looks stable

An editor can return the same red, green, and blue values with a different alpha channel. A viewer on a dark checkerboard may make the change obvious; a white product page may barely show it. Raw RGB comparison misses the defect because the color triplet did not move. Comparison after compositing makes the consequence visible. Black and white backdrops cover two extremes and expose many alpha differences, although they do not replace testing against a branded or textured background when that exact context is contractual.

The inverse near-miss also matters. Fully transparent pixels may contain different hidden RGB values while rendering identically on every backdrop. Failing those raw values can create noise with no user-visible consequence. There are exceptions. Hidden RGB matters if a later pipeline changes alpha, uses color channels as data, or exports to a format that discards transparency unexpectedly. Decide at the system boundary whether the contract concerns the current rendered result or the latent channel data.

An alpha regression can originate outside the editor. Premultiplication mistakes, an export path that drops transparency, and an incorrect blend order all change visible pixels. The PNG specification describes unassociated alpha in the file, while rendering systems often use compositing operations internally. Save the source RGBA sample, output RGBA sample, and flattened samples at a failing coordinate. That evidence tells an engineer whether color, alpha, or both moved.

Turn a red build into a useful image investigation

A failure should leave more than a ratio in terminal history. Produce a view that marks only protected pixels over tolerance, then list connected components. A component is a group of neighboring failing pixels. Its bounding box and area make patterns visible without pretending to explain their cause. One compact box over a logo differs from a one-pixel outline around the entire mask, and both differ from a full-frame haze.

The diagnostic script below uses the same mask convention as the oracle. It evaluates source and candidate over black and white, identifies protected pixels whose largest rendered RGB channel delta exceeds the chosen tolerance, writes a red overlay on the white rendering, and prints component boxes. Run it on retained artifacts after the contract command fails. The script does not classify the owner. It preserves evidence for a human investigation.

from __future__ import annotations

import argparse
import json
from collections import deque
from pathlib import Path

from PIL import Image

from image_contract import flatten, load_oriented_rgba, png_color_metadata_digest


def connected_components(binary: Image.Image) -> list[dict[str, object]]:
    width, height = binary.size
    pixels = binary.load()
    seen: set[tuple[int, int]] = set()
    components: list[dict[str, object]] = []

    for start_y in range(height):
        for start_x in range(width):
            start = (start_x, start_y)
            if pixels[start] == 0 or start in seen:
                continue
            queue = deque([start])
            seen.add(start)
            points: list[tuple[int, int]] = []
            while queue:
                x, y = queue.popleft()
                points.append((x, y))
                for neighbor in ((x - 1, y), (x + 1, y), (x, y - 1), (x, y + 1)):
                    nx, ny = neighbor
                    if 0 <= nx < width and 0 <= ny < height:
                        if pixels[neighbor] != 0 and neighbor not in seen:
                            seen.add(neighbor)
                            queue.append(neighbor)
            xs = [point[0] for point in points]
            ys = [point[1] for point in points]
            components.append(
                {
                    "area": len(points),
                    "bbox": [min(xs), min(ys), max(xs) + 1, max(ys) + 1],
                }
            )
    return sorted(components, key=lambda item: int(item["area"]), reverse=True)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("source", type=Path)
    parser.add_argument("candidate", type=Path)
    parser.add_argument("mask", type=Path)
    parser.add_argument("overlay", type=Path)
    parser.add_argument("--channel-tolerance", type=int, required=True)
    args = parser.parse_args()

    if not 0 <= args.channel_tolerance <= 255:
        raise ValueError("channel_tolerance must be between 0 and 255")

    source_rgba = load_oriented_rgba(args.source)
    candidate_rgba = load_oriented_rgba(args.candidate)
    if png_color_metadata_digest(args.source) != png_color_metadata_digest(args.candidate):
        raise ValueError("PNG color metadata differs; canonicalize before rendering a diff")
    source_views = [
        flatten(source_rgba, (0, 0, 0)),
        flatten(source_rgba, (255, 255, 255)),
    ]
    candidate_views = [
        flatten(candidate_rgba, (0, 0, 0)),
        flatten(candidate_rgba, (255, 255, 255)),
    ]
    candidate_display = candidate_views[1]
    with Image.open(args.mask) as opened_mask:
        mask = opened_mask.convert("L")
    if source_rgba.size != candidate_rgba.size or source_rgba.size != mask.size:
        raise ValueError("source, candidate, and mask dimensions must match")

    failures = Image.new("L", source_rgba.size, 0)
    failure_pixels = failures.load()
    for y in range(source_rgba.height):
        for x in range(source_rgba.width):
            if mask.getpixel((x, y)) >= 128:
                continue
            delta = max(
                abs(source_views[view].getpixel((x, y))[channel]
                    - candidate_views[view].getpixel((x, y))[channel])
                for view in range(2)
                for channel in range(3)
            )
            if delta > args.channel_tolerance:
                failure_pixels[x, y] = 255

    red = Image.new("RGB", source_rgba.size, (255, 0, 0))
    tinted = Image.blend(candidate_display, red, 0.65)
    overlay = Image.composite(tinted, candidate_display, failures)
    overlay.save(args.overlay)
    print(json.dumps({"components": connected_components(failures)}, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Read the output in a fixed order. First check dimensions and orientation. A doubled edge on every high-contrast object usually means the images are shifted, resized, or resampled relative to each other. Confirm landmark coordinates before changing a tolerance. A one-pixel rectangular outline matching the edit boundary points toward mask rasterization, feathering, or an inclusive-versus-exclusive coordinate bug. Inspect the saved mask at full resolution and print its values across the border.

Next inspect whether the protected change appears in the earliest output artifact. If a service response is stored separately, compare that decoded image to the application's final export. The earliest artifact containing the change narrows the owner. Avoid relying only on timestamps when retries can reorder logs. Join artifacts with a stable case identifier and record their content hashes so an engineer knows which exact files were compared.

Then look at the distribution of deltas. A few large components attached to recognizable objects suggest content was redrawn. Tiny differences across most of the frame suggest a global transform, lossy re-encoding, color handling, or resize. This is a diagnostic heuristic, not a verdict. Confirm the file format, dimensions, color metadata, and pipeline stages before labeling a defect as model drift.

JPEG deserves its own lane. It is lossy, and editing or merely re-encoding one region can alter blocks beyond the mask. An exact protected-pixel oracle that is appropriate for PNG will produce widespread failures on JPEG. Prefer a lossless internal comparison artifact if the product can provide one, then test the final JPEG with a policy designed for that delivery format. The cost is extra storage and a test hook that may not mirror the user download byte for byte.

Mask alignment is another near-miss that can imitate uncontrolled editing. A front end may collect a mask on a scaled preview, while a worker applies it to an orientation-normalized original. Rounding, letterboxing, or a missed EXIF transform moves the authorized area. Keep a synthetic coordinate test in the adapter: mark one known pixel near each corner, transform the mask through the real path, and assert its final coordinates. That test is cheaper and clearer than compensating with a broad halo.

Retries can conceal a consistency problem. If the product silently regenerates until one candidate passes an internal check, the user may receive a valid image but wait longer and incur more compute. Store attempt count and status as operational evidence, not as a pixel metric. A release rule can allow a limited retry policy while a separate reliability test reports how often the first candidate violates containment. Do not average a failed first attempt and a passing second attempt into one visual score.

For repeat-run analysis, retain each candidate as its own row. Evaluate the same invariants independently, then report the count of contract violations with a confidence interval or another reviewed statistical method once the sample design is large enough. Never claim that a handful of convenient runs measures a production failure rate. Seeds, if a product exposes them, are useful replay inputs only to the extent that the product documents their scope. A matching seed should not be described as a guarantee of identical pixels without evidence.

Roll the check out without freezing legitimate edits

Begin in report-only mode against a small set of reviewed cases. For a team introducing AI image editing consistency testing, this shadow period is where the spatial promise becomes an enforceable release rule. Choose cases that exercise different geometry: a central rectangular edit, a narrow object beside protected text, a feathered edge, transparency, and a full-resolution export. The purpose of this phase is not to make a dashboard green. It is to identify which failures belong to the product, which reveal a bad mask adapter, and which show that the written contract is too strict for the feature customers actually use.

Version each case as a bundle. Include the source image, normalized edit mask, requested operation, expected invariants, and any operation-specific evidence. Record hashes for source and mask files in a manifest. If either artifact changes, review it as a new fixture version rather than silently replacing history. A baseline approval should name the reviewer and the contract it approves, not merely say that the picture looks fine.

Promote checks in layers. Make format readability, dimensions, mask geometry, and missing artifacts blocking first because their interpretation is deterministic. Next block strict protected regions for operations that promise exact preservation. Keep feathered boundaries and probabilistic semantic checks in a review lane until the team has real failure examples. Finally, add repeated executions on a scheduled job if the release risk justifies their cost. This order gives developers useful failures before the suite starts consuming many generated images.

The workflow below is a concrete GitHub Actions example. It assumes the three scripts live under tests, reviewed image fixtures live under tests/fixtures/remove-cup, and a test dependency file installs Pillow and pytest. The numerical values are illustrative policy choices for this example, not observed tolerances. Replace them only through a review of the feature contract and retained artifacts.

name: image edit contract

on:
  pull_request:
    paths:
      - "src/image-editor/**"
      - "tests/image_contract.py"
      - "tests/test_image_contract.py"
      - "tests/fixtures/remove-cup/**"
  workflow_dispatch:

jobs:
  protected-region-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install test dependencies
        run: python -m pip install -r requirements-test.txt
      - name: Prove the oracle with synthetic mutations
        run: python -m pytest -q tests/test_image_contract.py
      - name: Check the reviewed edit artifact
        id: reviewed_artifact
        run: >-
          python tests/image_contract.py
          tests/fixtures/remove-cup/source.png
          tests/fixtures/remove-cup/candidate.png
          tests/fixtures/remove-cup/allowed-mask.png
          --channel-tolerance 4
          --max-protected-fraction 0.0005
          --min-requested-fraction 0.02
      - name: Render protected diff after a failure
        if: failure() && steps.reviewed_artifact.outcome == 'failure'
        run: >-
          python tests/render_protected_diff.py
          tests/fixtures/remove-cup/source.png
          tests/fixtures/remove-cup/candidate.png
          tests/fixtures/remove-cup/allowed-mask.png
          tests/fixtures/remove-cup/protected-diff.png
          --channel-tolerance 4
      - uses: actions/upload-artifact@v4
        if: failure() && steps.reviewed_artifact.outcome == 'failure'
        with:
          name: protected-region-evidence
          path: tests/fixtures/remove-cup/protected-diff.png

Run the synthetic mutation tests before the reviewed artifact check. They defend the oracle from accidental weakening. If someone reverses the mask convention, removes alpha-sensitive rendering, ignores changed PNG color metadata, or drops the no-op rule, at least one constructed case should fail. This is not proof that the production thresholds are right. It proves that the harness still enforces the rules its authors wrote.

Artifact generation must not happen implicitly inside a pull-request assertion unless the test is explicitly meant to call the editor. Mixing generation and comparison makes reruns expensive and diagnosis ambiguous. A deterministic fixture lane should compare checked-in or immutably stored artifacts. A separate integration lane can invoke the system under test, save its raw response and final export, and then call the same comparator. Label those lanes clearly so developers know whether a red result is reproducible from existing files.

The main cost is maintenance. Masks must follow intentional fixture changes. Thresholds need owners. Diff artifacts consume storage. Full-resolution comparison touches every pixel and retains four rendered RGB views plus the decoded source and candidate; the loop avoids extra Python lists, but large artifacts still consume meaningful memory. Repeat-run coverage also multiplies generation latency. A practical suite keeps a compact blocking set on pull requests, exercises more resolutions and operations nightly, and preserves long-running distribution checks for release candidates. Sampling saves time but reduces coverage, so rotate cases deliberately rather than letting the same attractive demo image become the entire test strategy.

Strict spatial clipping also has a product cost. It can preserve a logo perfectly while creating a visible seam beside it. A wider editable halo improves blending but grants the editor more authority. Raising tolerance reduces false failures but permits low-amplitude drift. Regenerating after a violation improves the delivered pass rate but adds latency and may hide an unstable first attempt. State these choices in the test policy. A threshold without its accepted failure mode is just a number waiting to be misread.

Do not auto-update a candidate because the new version looks plausible. That practice converts regression detection into change recording. When an expected edit changes intentionally, review the source, mask, candidate, diagnostic overlay, and operation-specific oracle together. Preserve the previous case long enough to compare behavior across the rollout. If a threshold changes, attach examples that previously failed and now pass, including the risk the team has chosen to accept.

Know when pixel consistency is the wrong requirement

Skip protected-pixel comparison when the feature explicitly regenerates the whole composition. Style transfer, relighting, pose changes, perspective changes, and broad outpainting can alter every pixel while meeting the user's request. A mask may still guide the operation, but it may not define a hard containment boundary. For those products, use semantic invariants that match the promise: required subjects remain present, prohibited content does not appear, text remains readable, composition stays within documented limits, or a qualified reviewer accepts the result.

Do not use a loose perceptual score as a disguised substitute for an absent contract. A score can rank images without saying which protected fact changed. If a serial number, legal label, face identity, or medical annotation must remain correct, test that property directly with an appropriate specialist system and human review. Pixel preservation can support the case, but it does not prove identity, meaning, or safety.

Avoid this method when images are not registered to the same coordinate system and registration is not itself part of the product. A one-pixel translation creates differences along nearly every edge. Automatically aligning the candidate before comparison may hide a real crop or placement defect. Only register images when the user-visible contract permits movement and the alignment algorithm is separately tested. Keep the applied transform in the report.

Whole-file hashing remains correct for a narrower requirement: a cached export must be the exact artifact previously approved, a download must arrive without byte corruption, or a deterministic encoder is under test. In those cases, hash the bytes. Do not call the result a visual consistency test. Pair it with decoded-image checks if both transport integrity and visible output matter.

Likewise, a generated thumbnail should not be compared against a full-resolution original with the protected-pixel oracle. Resampling deliberately combines neighboring pixels. Test the resizing algorithm against its own fixtures, then compare like-sized artifacts at the boundary where preservation is promised. If the application only exposes the thumbnail, write a contract for the thumbnail rather than pretending the unavailable master was tested.

Human review is the right release gate when the accepted change depends on context that the deterministic suite cannot encode safely. A reviewer can judge whether a removed reflection still looks natural, whether lighting continuity matters across the mask, or whether an identity feels preserved after a broad edit. Give that reviewer the source, candidate, mask overlay, and deterministic diagnostics. Do not reduce the decision to a thumbs-up beside an unmarked image.

Finally, do not collect more image artifacts than the investigation needs. User uploads can contain faces, documents, locations, and other sensitive material. Synthetic and licensed fixtures are preferable for routine CI. If production examples are necessary for incident work, apply the organization's retention, access, and redaction controls before they enter a test bucket. A visually excellent oracle is still a poor QA system if its evidence handling creates a new privacy failure.