A green image job can still ship the wrong picture

The image endpoint returns HTTP 200, and the asset has the requested dimensions. The picture still gives the product model six fingers and prints a sale price of “$199” instead of “$19.99.” Transport checks passed, but the output failed.

A useful evaluation therefore needs several small oracles tied to explicit requirements, not one mysterious “quality” number that lets a technically valid but unusable picture pass.

Split quality into questions you can answer

Begin with the use case, because quality changes with intent. A decorative blog illustration may tolerate invented background detail. A product listing cannot invent an accessory that is not included. A safety poster cannot improvise warning text. A profile avatar may allow broad stylistic variation while still prohibiting visible personal data.

Turn that intent into a contract with independent dimensions:

• Artifact validity covers format, decodability, dimensions, file size limits, and required color or transparency properties.
• Prompt adherence covers named subjects, counts, spatial relationships, actions, setting, and requested exclusions.
• Text correctness covers spelling, numbers, punctuation, language, and placement of text that must appear in the pixels.
• Visual integrity covers malformed anatomy, disconnected objects, impossible overlaps, repeated structures, and obvious generation artifacts.
• Policy and factual safety cover prohibited symbols, personal data, regulated claims, brand misuse, and details the system must not invent.
• Presentation fitness covers crop safety, focal area, composition, brand treatment, and whether the asset works in its target layout.
• Accessibility covers the surrounding product experience, including an appropriate text alternative when the image conveys information.

These dimensions should not be averaged before you look at them. A perfect composition score does not compensate for the wrong price. A beautifully rendered medication label with the wrong dosage is a failure, full stop. Mark requirements as blocking, reviewable, or informational before generating any images.

The evaluator also needs to know where each fact came from. “One blue backpack centered on a plain background” can be checked against the prompt. “The zipper must be on the left side because that is how SKU BP-17 is manufactured” comes from product data. “No faces may appear” may come from policy. Store those sources in the case manifest so a reviewer can challenge a label without reverse-engineering the prompt author's memory.

One image is not a test of a stochastic generator. It is a sample from a configuration that includes the prompt, model identifier, model version when exposed, generation parameters, and any preprocessing or post-processing. Save all of those fields that your provider actually returns. Do not claim that a seed guarantees reproducibility unless the provider's documentation for the pinned version says so. Even when a seed is accepted, infrastructure or model changes can alter output.

A useful result is a vector such as:

case_id: catalog-backpack-front
artifact:
  path: artifacts/catalog-backpack-front/output.png
  format: png
  required_width: 1024
  required_height: 1024
requirements:
  - id: subject-count
    rule: exactly one backpack is visible
    severity: blocking
  - id: product-color
    rule: the backpack body is navy blue
    severity: blocking
  - id: background
    rule: the background is plain light gray
    severity: review
  - id: no-included-extras
    rule: no bottle, laptop, or headphones are shown
    severity: blocking
review:
  rubric_version: "catalog-v3"
  minimum_reviewers: 2

That manifest is executable data, not prose buried in a ticket. It lets the harness reject a missing artifact, lets reviewers score the same requirements, and lets CI group failures by severity. The values above are an example contract, not results from an experiment.

Avoid adjectives without anchors. “Professional,” “high quality,” and “beautiful” invite reviewers to apply private standards. Replace them with observable questions: Is the whole product inside the crop? Is every required label readable at the delivery size? Are shadows consistent with the requested studio lighting? Does any object appear fused with another?

Catch broken artifacts before judging aesthetics

An evaluator should fail fast on files that downstream systems cannot use. Check that the response body is non-empty, the declared content type agrees with the accepted format, the decoder can read the file, and its dimensions match the delivery contract. If the product accepts only static PNG, an animated file or a JPEG renamed to .png is not a visual-quality debate.

The W3C PNG specification defines an eight-byte signature and requires the IHDR chunk first. That header contains width and height as four-byte unsigned integers. The following standard-library function checks that narrow contract. It deliberately does not pretend to be a complete PNG decoder or CRC validator:

from dataclasses import dataclass
from pathlib import Path
import struct


PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"


@dataclass(frozen=True)
class PngHeader:
    width: int
    height: int
    bit_depth: int
    color_type: int


def read_png_header(path: Path) -> PngHeader:
    header = path.read_bytes()[:29]
    if len(header) < 29:
        raise ValueError("file is too short to contain PNG signature and IHDR")
    if header[:8] != PNG_SIGNATURE:
        raise ValueError("file does not have the PNG signature")

    chunk_length = struct.unpack(">I", header[8:12])[0]
    if header[12:16] != b"IHDR" or chunk_length != 13:
        raise ValueError("first PNG chunk is not a 13-byte IHDR")

    width, height = struct.unpack(">II", header[16:24])
    if width == 0 or height == 0:
        raise ValueError("PNG dimensions must be non-zero")
    return PngHeader(
        width=width,
        height=height,
        bit_depth=header[24],
        color_type=header[25],
    )


if __name__ == "__main__":
    image = Path("artifacts/catalog-backpack-front/output.png")
    parsed = read_png_header(image)
    assert (parsed.width, parsed.height) == (1024, 1024)
    print(parsed)

Header inspection catches wrong formats and wrong dimensions early. It does not prove the compressed image data is intact, that every chunk is valid, or that a browser can render the file. Use a maintained decoder in the production harness for full decoding, and test the exact decoder used by your application if compatibility matters. The header check remains useful as a transparent diagnostic because it distinguishes “we received the wrong bytes” from “the pixels depict the wrong thing.”

Keep the original bytes. Re-encoding an image before evaluation can remove metadata, change color profiles, repair malformed files, or introduce compression differences. Evaluate the delivered artifact first, then derive thumbnails or judge inputs as named transformations. Record a digest for lineage, not as a quality score.

Byte hashes answer whether two files are identical. They do not answer whether two images are equally acceptable. Metadata ordering, compression settings, or a one-pixel change will alter a cryptographic digest. Conversely, a stable hash proves only identity to a previously captured file, not correctness. Snapshot approval can be valid for a deterministic post-processing layer, but it is a poor default oracle for a generative model.

Dimensions can also mislead. A 1024 by 1024 header does not guarantee that the subject is uncropped, that meaningful content fills the frame, or that text remains readable after the product displays a 160-pixel thumbnail. Test at the actual delivery size. A reviewer looking only at a full-resolution artifact may miss a label that disappears in the card component.

If images are published on the web, validate the surrounding markup separately. WCAG 2.2 requires a text alternative for non-text content, with exceptions based on purpose. The generation model cannot decide whether an image is informative, functional, or decorative in the final page. That decision belongs to the product context. A visually strong image with a useless filename as alt text is still an accessibility defect in the delivered experience.

Use worked cases that expose different defects

Consider an ecommerce prompt: “Studio photo of one navy commuter backpack, front view, plain light-gray background, no accessories.” The model returns a polished image with a water bottle in a side pocket. A generic similarity judge may reward the close match to “commuter backpack.” The catalog contract must fail because the bottle implies an included item.

This case needs object presence and absence labels. A reviewer records the backpack count, color, view, background, and each prohibited accessory independently. If an automated detector is used, its raw detections and confidence values remain evidence, not truth. Validate that detector against human labels on this product category. A detector calibrated on everyday photographs may behave differently on stylized or synthetic studio images.

The fix may be a stronger negative instruction, a product-specific reference image, post-generation selection, or a manual review gate. Every option costs something. More restrictive prompts can reduce composition variety. Generating several candidates increases latency and spend. Reference conditioning can overfit the pose or leak source-image traits. Manual review slows publication and requires staffing.

A second case is a localized event poster. The layout is attractive, the language is correct, but the date appears as “18/07” in one region where the approved copy says “Jul 18,” and the venue name loses an accent. This is not ordinary visual aesthetics. It is text fidelity.

Do not ask one reviewer for a holistic one-to-five score. Keep approved strings in the manifest, extract or manually transcribe visible text, and compare code points, numbers, and punctuation. OCR can accelerate the check, but OCR errors are a competing explanation. When OCR says “Jul I8,” inspect the pixels before blaming the generator. Preserve the crop passed to OCR and the engine output so the owner can see which system failed.

The safer production design may be to generate the background and render critical text with a normal layout engine. That sharply improves string control and accessibility options. The trade-off is architectural complexity: typography, wrapping, localization, and collision handling become application responsibilities. For prices, legal copy, dosage, dates, and contact details, that cost is usually easier to test than text painted by a generative model.

A third case is a warehouse evacuation diagram. All requested objects appear, yet an arrow points from the assembly area back toward the hazard zone. Image-prompt similarity can be high because the scene contains exits, arrows, and warning icons. The relationship is wrong.

Spatial rules need explicit labels such as “the green arrow begins at the current-location marker and ends at Exit B without crossing the red exclusion polygon.” A general vision judge may help triage, but a safety diagram should be built from structured geometry when exact topology matters. If the product insists on free-form generation, human approval is part of the release path, not an inconvenience to automate away.

These cases should remain separate report slices:

• Catalog fidelity catches invented or missing product details.
• Typography catches wrong characters, numbers, and language.
• Spatial safety catches incorrect relationships even when every object is present.

Combining them into one average loses the reason for failure. A model update may improve composition while making text worse. The release decision depends on where the model is used.

Candidate selection changes the system under test. If production generates four images and a ranking service chooses one, evaluating a hand-picked favorite measures neither the generator alone nor the delivered pipeline. Save every candidate, the selection rule, the selected identifier, and the final transformed asset. Score generator coverage across all candidates, then score delivery requirements on the selected result.

This distinction catches a useful near-miss. The generator may produce one compliant catalog image among four, while the selector repeatedly chooses a sharper image that includes a prohibited accessory. More generation will not fix the ranking rule. The reverse is also possible: a strong selector hides a falling candidate pass rate until cost or latency forces the product to generate fewer options. Report both candidate yield and selected-output quality, with the denominator stated.

Repeated generations should use a declared sampling plan. Running troublesome prompts more often is sensible for stress testing, but it changes the corpus weighting. Keep coverage runs, stress runs, and incident reproductions as separate reports. Otherwise a release can appear worse simply because QA added harder cases, or appear better because a retry policy quietly chose the first passing image.

Separate generator regressions from evaluator noise

The first clue is disagreement. If two trained reviewers apply the same requirement differently, the rubric may be ambiguous. If reviewers agree but an automated judge disagrees, the judge or its input transformation is suspect. If every evaluator agrees that the output is wrong, investigate generation.

Use blinded pairwise review for subjective changes. Show the baseline and candidate in randomized left-right order without model names. Ask a narrow question such as “Which image keeps the entire product visible and preserves the requested front view?” Include “tie” and “both fail” so reviewers are not forced to approve a bad candidate.

Absolute rubrics still matter for non-negotiable requirements. A candidate can beat a terrible baseline and remain unfit. Run blocking checks on each image before pairwise preference. Pairwise review then answers whether the candidate is better among outputs that already meet the floor.

Judge calibration is a test suite of its own. Build a reviewed set containing clear passes, clear failures, and difficult boundary cases. Run the exact judge prompt and image transformation used in CI. Track false approvals and false rejections by requirement type. Do not tune the rubric on the same cases used for the final comparison, because that makes the reported agreement optimistic.

The following program consumes independent JSONL ratings and exposes requirement-level disagreement. It uses no model API and can run against human or automated labels with the same schema:

from collections import defaultdict
import json
from pathlib import Path


def load_ratings(path: Path) -> list[dict[str, object]]:
    rows = []
    with path.open(encoding="utf-8") as stream:
        for line_number, line in enumerate(stream, start=1):
            if not line.strip():
                continue
            row = json.loads(line)
            required = {"case_id", "requirement_id", "reviewer_id", "passed"}
            missing = required.difference(row)
            if missing:
                raise ValueError(f"line {line_number} missing {sorted(missing)}")
            if type(row["passed"]) is not bool:
                raise TypeError(f"line {line_number} passed must be boolean")
            rows.append(row)
    return rows


def disagreements(rows: list[dict[str, object]]) -> list[tuple[str, str]]:
    votes: dict[tuple[str, str], set[bool]] = defaultdict(set)
    for row in rows:
        key = (str(row["case_id"]), str(row["requirement_id"]))
        votes[key].add(bool(row["passed"]))
    return sorted(key for key, values in votes.items() if len(values) > 1)


if __name__ == "__main__":
    ratings = load_ratings(Path("artifacts/image-ratings.jsonl"))
    for case_id, requirement_id in disagreements(ratings):
        print(f"{case_id}\t{requirement_id}\tREVIEW_REQUIRED")

Treat disagreement as a queue, not automatically as a model failure. Display the artifact, prompt, requirement, and each reviewer's rationale. Resolve whether the image is ambiguous, the requirement is vague, or one reviewer missed evidence. Update labels with an audit trail. Quietly taking a majority vote can institutionalize a broken rubric.

Near-miss failures often come from preprocessing. A judge may receive a center-cropped square while a human reviews the original landscape file. An upload service may convert a wide-gamut profile before the asset reaches production. A thumbnailer may cut off the required logo. Compare digests and dimensions at the generation, storage, transformation, judge, and delivery boundaries. The first boundary where they differ identifies the owning system.

Another near-miss is prompt assembly. The case manifest says “no text,” but a template adds “include the campaign title” after it. The generator is following the final prompt while the evaluator is scoring the source fragment. Save the fully assembled prompt and any reference inputs. A test result without the actual generation request is not reproducible evidence.

Separate a generator miss from a stale requirement

A missing visual element can produce the same blocking row through two unrelated defects. Imagine a catalog case whose evaluator requires a red warranty badge. The final image has no badge, so the report says the mandatory element is absent. The generator may have ignored a current requirement. The product record may instead have removed that badge while the evaluation manifest still carries an older rule. In the second case, the image can be correct for the current catalog and wrong only for the stale oracle.

The separating evidence is the business-source revision used on both sides of the test. Follow the product or campaign record into the case manifest, fully assembled generation request, reference inputs, and evaluator requirement. If the same current revision requires the badge, the assembled request contains that obligation, and the returned candidates omit it, generation owns the failure. If the current source and request no longer require a badge while the evaluator points to an earlier revision, update and re-review the case. If the manifest is current but the artifact was generated from another revision, the fixture or result join is broken rather than the rule itself.

The diagnostic view should place the case identifier, product identifier, business-source revision, case-manifest revision, generation request identifier, artifact digest, requirement identifier, requirement provenance, evaluator decision, and human decision together. A healthy failing row has aligned revisions and identifiers, then shows the required badge absent in the reviewed pixels. A stale-oracle row shows a requirement source older than the source used to assemble the request. A bad join shows a request or artifact identifier that belongs to another product even though the score row carries the expected case identifier.

Several values look reassuring without separating these causes. Human agreement that “no badge is visible” confirms the pixels but not whether a badge still belongs there. A detector confidence near zero confirms neither provenance nor product policy. A recent file timestamp can belong to an image generated from cached, outdated inputs. Even a matching product identifier is insufficient when generation and evaluation resolved it at different revisions. The revision chain, actual assembled request, and bound artifact must agree before the failure can be assigned to generation.

Do not silently relabel the row after discovering drift. Preserve the old decision, mark why the reference changed, and run the revised case against the same artifact. That audit trail distinguishes a corrected oracle from a model improvement. It also prevents a product-data rollback from making a previously removed requirement look newly valid without review.

Put the right failures in CI

Split the pipeline by cost. Pull requests can validate manifests, inspect saved artifacts, run deterministic requirements, and score a small reviewed smoke set. Scheduled jobs can generate more samples, call expensive judges, and open review queues. A release candidate can require human sign-off for high-risk slices.

This pytest file demonstrates deterministic artifact checks plus explicit reviewed labels. The thresholds are example policy values for the sample cases, not measurements or universal recommendations:

from pathlib import Path

import pytest

from png_contract import read_png_header


CASES = [
    {
        "id": "catalog-backpack-front",
        "path": Path("artifacts/catalog-backpack-front/output.png"),
        "size": (1024, 1024),
        "blocking_labels": {
            "subject-count": True,
            "product-color": True,
            "no-included-extras": True,
        },
    },
    {
        "id": "localized-poster-fr",
        "path": Path("artifacts/localized-poster-fr/output.png"),
        "size": (1536, 2048),
        "blocking_labels": {
            "approved-date": True,
            "venue-spelling": True,
        },
    },
]


@pytest.mark.parametrize("case", CASES, ids=lambda case: case["id"])
def test_image_contract(case: dict[str, object]) -> None:
    path = Path(case["path"])
    assert path.is_file(), f"missing generated artifact: {path}"

    header = read_png_header(path)
    assert (header.width, header.height) == case["size"]

    failed = [
        requirement
        for requirement, passed in dict(case["blocking_labels"]).items()
        if passed is not True
    ]
    assert not failed, f"blocking image requirements failed: {failed}"

In a real suite, reviewed labels should come from a versioned result file, not literals maintained beside the assertion. The compact example makes the gate visible: a failed blocking label cannot be averaged away. Add schema validation so absent labels do not become false or silently disappear.

A practical workflow preserves evidence on failure:

name: generated-image-evaluation

on:
  pull_request:
    paths:
      - "image_pipeline/**"
      - "evals/images/**"
      - "requirements.lock"
  workflow_dispatch:

jobs:
  image-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements.lock
      - run: python -m pip install -r requirements.lock
      - run: mkdir -p artifacts
      - run: python -m pytest -q tests/evals/test_image_contract.py --junitxml=artifacts/image-evals.xml
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: image-evaluation-evidence
          path: artifacts/

Spell out cache-dependency-path whenever the installed file is not one the action already looks for. Its defaults are **/requirements.txt and **/pyproject.toml, so a pipeline pinned by requirements.lock dies at setup and reports a failure that says nothing about image quality.

Store artifacts according to your data policy. Generated images can still contain faces, personal details copied from references, offensive material, or confidential prompt content. Restrict access and retention for failed outputs instead of publishing every artifact in a broadly visible CI system.

Roll out in four stages. First, validate manifests and file contracts without blocking model changes. Second, backfill reviewed labels for a small risk-balanced corpus. Third, block only clear deterministic violations while judge disagreements go to review. Fourth, add slice-level release rules after the team has observed evaluator stability.

For a suite that already publishes a holistic score, land compatible readers before changing writers. Existing dashboards often assume one image, one score, and one verdict per case. Candidate sets create several artifacts, requirement vectors create several verdicts, and an unscored case is neither a pass nor a failure. Add explicit handling for missing labels and review-pending rows first. Otherwise an older consumer may coerce an absent decision into a pass or count every candidate as a separate test case.

Keep the old score visible during the compatibility window, but do not use it to fill the new requirement fields. Build a crosswalk from every legacy case identifier to its new result envelope and run it over the same saved outputs without changing the generation request. Every old row must appear once as scored, unscored for a named reason, or awaiting review. Candidate children must remain attached to their parent case rather than inflating the case denominator. Resolve lost rows, duplicate rows, and attachment mismatches before any consumer takes its release decision from the new shape. Retire the old view only after retained failures can still be reopened through the new index.

The first operational break is often outside the evaluator. Snapshot browsers may assume a fixed output.png; cleanup jobs may retain only the selected image; report pages may generate duplicate links when four candidates share one case; and upload limits may discard the originals needed for triage. Update those consumers before relying on lineage. If a case generates four candidates, retaining all four plus the selected derivative stores more than the single delivered file the old suite kept. That extra storage, upload time, and restricted-access review surface are the concrete cost of proving whether generation or selection failed. Human confirmation adds queue time as well, particularly for releases that span several locales.

Ownership should follow the first bad artifact. The generation team owns a requirement missing from the returned candidates. The ranking team owns selection of a noncompliant candidate when a compliant one was available under the declared rule. The media pipeline owns a crop, conversion, or compositing regression. The evaluation team owns a wrong artifact binding or unsupported judge decision. Product, localization, accessibility, or policy owners define the requirement and decide ambiguous cases. A handoff needs the case and manifest revisions, fully assembled request, all candidate identifiers and digests, selected identifier, transformation chain, evaluated digest, requirement decision, reviewer rationale, and the first pair of images that differ. Sending only the final PNG forces the receiving team to reconstruct the pipeline without its evidence.

Watch the denominator. If generation fails and produces no image, that case belongs in the failure rate. Excluding it from visual scoring and reporting only the remaining images creates survivorship bias. Report request failures, invalid artifacts, unscored artifacts, review queues, and scored results as separate counts.

Version every moving part: dataset, prompt assembly, generation configuration, post-processing, rubric, judge prompt, judge model, and reviewer guide. A changed score without those identifiers is an anecdote. A row-level artifact with them is a regression you can investigate.

Know when not to automate the verdict

Do not automate aesthetic approval when the brand team has not defined what acceptable means. A judge will still produce a number, but the number merely hides an unresolved product decision. Use structured pairwise review to discover the criteria first.

Avoid generative images for content that must be exact and can be rendered from data. Prices, legal disclaimers, medication instructions, maps, wiring diagrams, and safety routes are better produced through deterministic layout or domain tools. Evaluation cannot make an unsuitable generation architecture safe.

Do not use a single reference image as a pixel target when variation is part of the feature. That rewards imitation and punishes valid diversity. References can clarify product identity, composition, or style, but scoring must remain tied to the requirements that matter.

Finally, do not let an automated vision judge make an irreversible policy decision without calibrated evidence and an appeal path. Use it to sort obvious passes, obvious failures, and uncertain cases. Human review costs time, but silent false approval costs trust, and the expensive mistake usually begins with a dashboard that reduced every kind of failure to one pleasant decimal.

This evaluation also does not prove that the right customer receives the right image. A cache-key or authorization defect can serve a fully compliant catalog asset from another tenant, and every pixel-level requirement for that asset may pass. Test tenant isolation, object ownership, and delivery authorization separately. Image quality evidence cannot detect a correct picture crossing the wrong security boundary.