ppsspp/docs/frametest.md

# Framedump rendering tests (frametests)

PPSSPP has a test system that replays GE frame dumps (`.ppdmp`) through
`PPSSPPHeadless` with various rendering settings, comparing the rendered
output against stored reference images. It is designed to catch rendering
regressions across GPU backends and configurations.

The system has three parts:

- **The test runner**: `frametests.py` (repo root). Test-set agnostic - it
  works against any test data tree pointed to by its configuration.
- **The configuration**: a JSON file describing where the test data lives and
  which rendering variants to run (e.g. `frametests/frametests.json`).
- **The test set**: the frame dumps and reference images, maintained as the
  `frametests` git submodule (repo `hrydgard/ppsspp-frametest-ci`, pinned by
  commit - its config `frametests.json` lives inside it). Different
  environments can use different test sets - GitHub CI uses the submodule,
  custom machines can use their own (possibly much larger) ones.

## Quick start

```bash
# Build PPSSPPHeadless (see headless/README.md), then:
python3 frametests.py frametests/frametests.json
```

The first run generates reference images for any dumps that don't have them
yet (status `NEW`) and compares the rest (status `PASS`/`FAIL`). A summary is
printed and a self-contained HTML report is written to
`frametests/out/report.html` (default output dir; gitignored).

Exit code is `0` if all tests passed, `1` if anything failed, `2` on
configuration/usage errors.

## Configuration

The configuration is a single JSON file. All relative paths are resolved
against the config file's directory, so the same script works with any test
set by just pointing `--config` at a different file. The config lives with
the test set it describes (e.g. `frametests/frametests.json`) and is not
committed to the repository.

```json
{
	"testRoot": "dumps",
	"refRoot": "ref",
	"outputRoot": "out",
	"headlessPath": "",
	"timeout": 60,
	"maxMse": 0.0,
	"variants": {
		"soft": {
			"args": "--graphics=software"
		}
	}
}
```

| Key             | Description                                                                 |
|-----------------|-----------------------------------------------------------------------------|
| `testRoot`      | Directory tree containing frame dumps (`.ppdmp` files, possibly wrapped in `.zip`). |
| `refRoot`       | Where reference images live; mirrors the `testRoot` tree.                   |
| `outputRoot`    | Where logs, diff images, the report, and generated references go (default `out`). |
| `headlessPath`  | Path to the `PPSSPPHeadless` binary. Empty = auto-detect (see below).        |
| `timeout`       | Per-test timeout in seconds.                                                 |
| `maxMse`        | Maximum allowed MSE for screenshot comparison (0 = exact match).            |
| `variants`      | Map of variant name to variant configuration (see below).                   |

### Variants

Each test runs once per variant. A variant has a **suffix** and a
**compare-suffix**:

- The **suffix** (default: the variant name) is appended to the variant's own
  output image filenames (`actuals`, `diffs`, `logs`).
- The **compare-suffix** (default: the suffix) names the reference image the
  variant compares against: `<name>-<compare-suffix>.png`.
- **Reference images are only generated for variants whose suffix matches
  their compare-suffix.** This lets several variants share a single reference
  set - e.g. OpenGL and Vulkan outputs should be nearly bitwise identical, so
  a `gl` variant can compare against the `soft` references:

```json
"variants": {
	"soft": { "args": "--graphics=software" },
	"gl":   { "args": "--graphics=opengl", "compare-suffix": "soft" },
	"vul":  { "args": "--graphics=vulkan", "compare-suffix": "soft" }
}
```

A plain string value (`"soft": "--graphics=software"`) is shorthand for
`{ "args": "...", "suffix": "soft", "compare-suffix": "soft" }`.

## How a test runs

For each dump (recursively under `testRoot`) and each variant:

- If the variant's reference image `<name>-<compare-suffix>.png` is
  **missing** and the variant generates references (suffix ==
  compare-suffix): the dump is rendered and the output saved as the new
  reference. Status: `NEW`. This is how references are created - run locally,
  then commit the generated images (also copied to `<outputRoot>/generated/`
  for convenience).
- If the reference is **missing** for a variant that doesn't generate
  references (suffix != compare-suffix): Status: `ERROR` - the shared
  reference needs to be generated by the matching variant first.
- If the reference **exists**: the dump is rendered, the output saved to
  `<outputRoot>/actuals/`, and compared against the reference using MSE
  (mean squared error over R, G, B per pixel, alpha ignored). A visual
  comparison image (actual / reference+diff) is saved to
  `<outputRoot>/diffs/` whenever a comparison runs. Status: `PASS` or `FAIL`
  (mismatch, crash, or timeout).

A `FAIL` with no MSE reported usually means the headless binary crashed
before producing a screenshot, and a reference image that can't be loaded
(corrupt) is reported as `ERROR`. The full emulator log of every non-passing
test is kept in `<outputRoot>/logs/`.

### Reference images

- PNG, 512×272 (480×272 display in a 512-wide framebuffer), stored top-down
  (row 0 = top of screen). The BMP output format is bottom-up per the BMP
  spec; the flip is applied only when writing BMPs.
- The alpha channel is forced to 255 when writing PNGs (games often use alpha
  for non-visual purposes, which would otherwise produce transparent-looking
  images); pass `--screenshot-keep-alpha` to the headless binary to preserve
  it (e.g. via a variant). The MSE comparison ignores alpha either way.
- Generated with `--graphics=software` they are fully deterministic: a
  subsequent run produces byte-identical output, so `maxMse` can be 0.
- If rendering code changes the output, existing references may need
  regenerating: delete the affected reference images and re-run to regenerate
  them.

### Headless flags used

The runner uses `--screenshot=<ref>` (compare), `--screenshot-save=<file>`
(save output; PNG if the path ends in `.png`, else BMP) and
`--screenshot-diff=<file>` (always write a visual comparison when comparing).
See `headless/README.md` for details.

## Command line options

```
usage: frametests.py [OPTIONS] [CONFIG.json]
```

| Option             | Description                                                           |
|--------------------|-----------------------------------------------------------------------|
| `CONFIG.json`      | Path to the configuration file (default: `frametests/frametests.json`). |
| `--filter=SUBSTR`  | Only run dumps whose relative path contains SUBSTR (case-insensitive). |
| `--strict`         | Treat missing reference images as failures (a configuration error).    |
| `--out-mode=all\|failures` | `all` keeps everything in the output dir; `failures` keeps only artifacts of non-passing tests (smaller CI artifacts). Default: `all`. |

The headless binary is located, in order of preference:

1. `headlessPath` from the config file.
2. The `PPSSPP_HEADLESS` environment variable.
3. Well-known paths relative to the current directory (e.g.
   `Windows/x64/Debug/PPSSPPHeadless.exe`, `build/PPSSPPHeadless`), newest
   by modification time.

## CI integration

In CI (`GITHUB_ACTIONS` is set) the runner behaves as if `--strict` was
passed and prints `::error` annotations for each failing test, so failures
show up inline in the GitHub Actions log.

The CI workflow extends the existing `test` job (Linux leg) in
`.github/workflows/build.yml`: the `Fetch tests` step inits the `frametests`
submodule (alongside `pspautotests`), then after the pspautotests run it
executes the frametests and uploads the report as an artifact:

```yaml
- name: Fetch tests
  run: git submodule update --init pspautotests frametests

- name: Execute frametests
  if: runner.os == 'Linux'
  run: python3 frametests.py frametests/frametests.json --out-mode=failures

- name: Upload frametest report
  uses: actions/upload-artifact@v7
  if: runner.os == 'Linux' && always()
  with:
    name: frametest-report
    path: frametests/out/
```

A missing reference image in CI means the test set is incomplete - the run
fails and the generated reference is available in
`frametests/out/generated/` (part of the uploaded artifact) so it
can be committed.

## Custom machines

The script is designed to run on machines with different GPUs and bigger test
sets than CI. Copy the repo, point at a custom config:

```bash
python3 frametests.py /path/to/my-config.json
```

The config can live anywhere and point at any test data tree; only the
script itself is shared. Add hardware-specific variants (`--graphics=vulkan`,
`--msaa=...`, etc.) to the machine's own config - no code changes needed for
new flag combinations, as long as the headless binary supports them.

## Adding a new framedump

1. Drop the dump into `frametests/dumps/` (either as a `.ppdmp` or zipped).
2. Run `python3 frametests.py frametests/frametests.json --filter=<name>` -
   the reference image(s) are generated and copied to `<outputRoot>/generated/`.
3. Commit the dump and the reference images under `frametests/ref/`, and push
   them to the `frametests` submodule repo (`hrydgard/ppsspp-frametest-ci`).
   Then bump the submodule pointer in PPSSPP so CI picks up the new test.

The CI test set (dumps and references) is maintained separately from the
runner; the `frametests/` submodule is just one of several possible test sets
- custom machines can use their own (possibly much larger) ones.

## Replaying a dump on a real PSP

The references above are PPSSPP's own renders, so they catch regressions, not
wrong output. To see what the hardware draws, replay the dump on a PSP:
`pspautotests/utils/ppdmp-playback` is a PSP program that runs a `.ppdmp` natively
(all dump versions PPSSPP writes), and its `run.py` drives it over PSPLink from
the host:

```bash
cd pspautotests
python3 utils/ppdmp-playback/run.py --out /tmp/shots \
	--headless ../build/PPSSPPHeadless "../frametests/dumps/Depth/21391 Coded arms ULUS10019_0001.zip"
```

It saves what the PSP displays as `NAME-psp.png`, and with `--headless` renders
the same dump in PPSSPPHeadless (`--graphics=software` by default), saves
`NAME-ppsspp.png` and prints the MSE between the two. Each frame stays on the
PSP's screen for 1.5 seconds (`--hold`). It needs the PSPSDK (it rebuilds the PRX) and a PSP in
PSPLink with `usbhostfs_pc` serving the pspautotests root, the same setup as
[pspautotests-hardware.md](pspautotests-hardware.md); like every hardware run,
one at a time. See its README for replaying a subset of the primitives.

The PSP render doesn't match PPSSPP's to the bit even where PPSSPP is right
(dithering, texture filtering and edge rules differ slightly), so compare MSEs
between versions rather than expecting zero: the Coded Arms dump above went from
108 to 33 when the software renderer got the GE's depth precision.