ppsspp/docs/frametest.md

# Framedump rendering tests (frametests)

PPSSPP has a test system that replays GE frame dumps (`.ppdmp`) through
`PPSSPPHeadless` with various rendering settings, comparing the rendered
output against stored reference images. It is designed to catch rendering
regressions across GPU backends and configurations.

The system has three parts:

- **The test runner**: `frametests.py` (repo root). Test-set agnostic - it
  works against any test data tree pointed to by its configuration.
- **The configuration**: a JSON file describing where the test data lives and
  which rendering variants to run (e.g. `frametests/frametests.json`).
- **The test set**: the frame dumps and reference images, maintained as the
  `frametests` git submodule (repo `hrydgard/ppsspp-frametest-ci`, pinned by
  commit - its config `frametests.json` lives inside it). Different
  environments can use different test sets - GitHub CI uses the submodule,
  custom machines can use their own (possibly much larger) ones.

## Quick start

```bash
# Build PPSSPPHeadless (see headless/README.md), then:
python3 frametests.py frametests/frametests.json
```

The first run generates reference images for any dumps that don't have them
yet (status `NEW`) and compares the rest (status `PASS`/`FAIL`). A summary is
printed and a self-contained HTML report is written to
`frametests/out/report.html` (default output dir; gitignored).

Exit code is `0` if all tests passed, `1` if anything failed, `2` on
configuration/usage errors.

## Configuration

The configuration is a single JSON file. All relative paths are resolved
against the config file's directory, so the same script works with any test
set by just pointing `--config` at a different file. The config lives with
the test set it describes (e.g. `frametests/frametests.json`) and is not
committed to the repository.

```json
{
	"testRoot": "dumps",
	"refRoot": "ref",
	"outputRoot": "out",
	"headlessPath": "",
	"timeout": 60,
	"maxMse": 0.0,
	"variants": {
		"soft": {
			"args": "--graphics=software"
		}
	}
}
```

| Key             | Description                                                                 |
|-----------------|-----------------------------------------------------------------------------|
| `testRoot`      | Directory tree containing frame dumps (`.ppdmp` files, possibly wrapped in `.zip`). |
| `refRoot`       | Where reference images live; mirrors the `testRoot` tree.                   |
| `outputRoot`    | Where logs, diff images, the report, and generated references go (default `out`). |
| `headlessPath`  | Path to the `PPSSPPHeadless` binary. Empty = auto-detect (see below).        |
| `timeout`       | Per-test timeout in seconds.                                                 |
| `maxMse`        | Maximum allowed MSE for screenshot comparison (0 = exact match).            |
| `variants`      | Map of variant name to variant configuration (see below).                   |

### Variants

Each test runs once per variant. A variant has a **suffix** and a
**compare-suffix**:

- The **suffix** (default: the variant name) is appended to the variant's own
  output image filenames (`actuals`, `diffs`, `logs`).
- The **compare-suffix** (default: the suffix) names the reference image the
  variant compares against: `<name>-<compare-suffix>.png`.
- **Reference images are only generated for variants whose suffix matches
  their compare-suffix.** This lets several variants share a single reference
  set - e.g. OpenGL and Vulkan outputs should be nearly bitwise identical, so
  a `gl` variant can compare against the `soft` references:

```json
"variants": {
	"soft": { "args": "--graphics=software" },
	"gl":   { "args": "--graphics=opengl", "compare-suffix": "soft" },
	"vul":  { "args": "--graphics=vulkan", "compare-suffix": "soft" }
}
```

A plain string value (`"soft": "--graphics=software"`) is shorthand for
`{ "args": "...", "suffix": "soft", "compare-suffix": "soft" }`.

## How a test runs

For each dump (recursively under `testRoot`) and each variant:

- If the variant's reference image `<name>-<compare-suffix>.png` is
  **missing** and the variant generates references (suffix ==
  compare-suffix): the dump is rendered and the output saved as the new
  reference. Status: `NEW`. This is how references are created - run locally,
  then commit the generated images (also copied to `<outputRoot>/generated/`
  for convenience).
- If the reference is **missing** for a variant that doesn't generate
  references (suffix != compare-suffix): Status: `ERROR` - the shared
  reference needs to be generated by the matching variant first.
- If the reference **exists**: the dump is rendered, the output saved to
  `<outputRoot>/actuals/`, and compared against the reference using MSE
  (mean squared error over R, G, B per pixel, alpha ignored). A visual
  comparison image (actual / reference+diff) is saved to
  `<outputRoot>/diffs/` whenever a comparison runs. Status: `PASS` or `FAIL`
  (mismatch, crash, or timeout).

A `FAIL` with no MSE reported usually means the headless binary crashed
before producing a screenshot, and a reference image that can't be loaded
(corrupt) is reported as `ERROR`. The full emulator log of every non-passing
test is kept in `<outputRoot>/logs/`.

### Reference images

- PNG, 512×272 (480×272 display in a 512-wide framebuffer), stored top-down
  (row 0 = top of screen). The BMP output format is bottom-up per the BMP
  spec; the flip is applied only when writing BMPs.
- The alpha channel is forced to 255 when writing PNGs (games often use alpha
  for non-visual purposes, which would otherwise produce transparent-looking
  images); pass `--screenshot-keep-alpha` to the headless binary to preserve
  it (e.g. via a variant). The MSE comparison ignores alpha either way.
- Generated with `--graphics=software` they are fully deterministic: a
  subsequent run produces byte-identical output, so `maxMse` can be 0.
- If rendering code changes the output, existing references may need
  regenerating: delete the affected reference images and re-run to regenerate
  them.

### Headless flags used

The runner uses `--screenshot=<ref>` (compare), `--screenshot-save=<file>`
(save output; PNG if the path ends in `.png`, else BMP) and
`--screenshot-diff=<file>` (always write a visual comparison when comparing).
See `headless/README.md` for details.

## Command line options

```
usage: frametests.py [OPTIONS] [CONFIG.json]
```

| Option             | Description                                                           |
|--------------------|-----------------------------------------------------------------------|
| `CONFIG.json`      | Path to the configuration file (default: `frametests/frametests.json`). |
| `--filter=SUBSTR`  | Only run dumps whose relative path contains SUBSTR (case-insensitive). |
| `--strict`         | Treat missing reference images as failures (a configuration error).    |
| `--out-mode=all\|failures` | `all` keeps everything in the output dir; `failures` keeps only artifacts of non-passing tests (smaller CI artifacts). Default: `all`. |

The headless binary is located, in order of preference:

1. `headlessPath` from the config file.
2. The `PPSSPP_HEADLESS` environment variable.
3. Well-known paths relative to the current directory (e.g.
   `Windows/x64/Debug/PPSSPPHeadless.exe`, `build/PPSSPPHeadless`), newest
   by modification time.

## CI integration

In CI (`GITHUB_ACTIONS` is set) the runner behaves as if `--strict` was
passed and prints `::error` annotations for each failing test, so failures
show up inline in the GitHub Actions log.

The CI workflow extends the existing `test` job (Linux leg) in
`.github/workflows/build.yml`: the `Fetch tests` step inits the `frametests`
submodule (alongside `pspautotests`), then after the pspautotests run it
executes the frametests and uploads the report as an artifact:

```yaml
- name: Fetch tests
  run: git submodule update --init pspautotests frametests

- name: Execute frametests
  if: runner.os == 'Linux'
  run: python3 frametests.py frametests/frametests.json --out-mode=failures

- name: Upload frametest report
  uses: actions/upload-artifact@v7
  if: runner.os == 'Linux' && always()
  with:
    name: frametest-report
    path: frametests/out/
```

A missing reference image in CI means the test set is incomplete - the run
fails and the generated reference is available in
`frametests/out/generated/` (part of the uploaded artifact) so it
can be committed.

## Custom machines

The script is designed to run on machines with different GPUs and bigger test
sets than CI. Copy the repo, point at a custom config:

```bash
python3 frametests.py /path/to/my-config.json
```

The config can live anywhere and point at any test data tree; only the
script itself is shared. Add hardware-specific variants (`--graphics=vulkan`,
`--msaa=...`, etc.) to the machine's own config - no code changes needed for
new flag combinations, as long as the headless binary supports them.

## Adding a new framedump

1. Drop the dump into `frametests/dumps/` (either as a `.ppdmp` or zipped).
2. Run `python3 frametests.py frametests/frametests.json --filter=<name>` -
   the reference image(s) are generated and copied to `<outputRoot>/generated/`.
3. Commit the dump and the reference images under `frametests/ref/`, and push
   them to the `frametests` submodule repo (`hrydgard/ppsspp-frametest-ci`).
   Then bump the submodule pointer in PPSSPP so CI picks up the new test.

The CI test set (dumps and references) is maintained separately from the
runner; the `frametests/` submodule is just one of several possible test sets
- custom machines can use their own (possibly much larger) ones.