ppsspp/docs/frametest.md

Framedump rendering tests (frametests)

PPSSPP has a test system that replays GE frame dumps (.ppdmp) through PPSSPPHeadless with various rendering settings, comparing the rendered output against stored reference images. It is designed to catch rendering regressions across GPU backends and configurations.

The system has three parts:

• The test runner: frametests.py (repo root). Test-set agnostic - it works against any test data tree pointed to by its configuration.
• The configuration: a JSON file describing where the test data lives and which rendering variants to run (e.g. frametests/frametests.json).
• The test set: the frame dumps and reference images, maintained as the frametests git submodule (repo hrydgard/ppsspp-frametest-ci, pinned by commit - its config frametests.json lives inside it). Different environments can use different test sets - GitHub CI uses the submodule, custom machines can use their own (possibly much larger) ones.

Quick start

# Build PPSSPPHeadless (see headless/README.md), then:
python3 frametests.py frametests/frametests.json

The first run generates reference images for any dumps that don't have them yet (status NEW) and compares the rest (status PASS/FAIL). A summary is printed and a self-contained HTML report is written to frametests/out/report.html (default output dir; gitignored).

Exit code is 0 if all tests passed, 1 if anything failed, 2 on configuration/usage errors.

Configuration

The configuration is a single JSON file. All relative paths are resolved against the config file's directory, so the same script works with any test set by just pointing --config at a different file. The config lives with the test set it describes (e.g. frametests/frametests.json) and is not committed to the repository.

{
	"testRoot": "dumps",
	"refRoot": "ref",
	"outputRoot": "out",
	"headlessPath": "",
	"timeout": 60,
	"maxMse": 0.0,
	"variants": {
		"soft": {
			"args": "--graphics=software"
		}
	}
}

Key	Description
testRoot	Directory tree containing frame dumps (.ppdmp files, possibly wrapped in .zip).
refRoot	Where reference images live; mirrors the testRoot tree.
outputRoot	Where logs, diff images, the report, and generated references go (default out).
headlessPath	Path to the PPSSPPHeadless binary. Empty = auto-detect (see below).
timeout	Per-test timeout in seconds.
maxMse	Maximum allowed MSE for screenshot comparison (0 = exact match).
variants	Map of variant name to variant configuration (see below).

Variants

Each test runs once per variant. A variant has a suffix and a compare-suffix:

• The suffix (default: the variant name) is appended to the variant's own output image filenames (actuals, diffs, logs).
• The compare-suffix (default: the suffix) names the reference image the variant compares against: <name>-<compare-suffix>.png.
• Reference images are only generated for variants whose suffix matches their compare-suffix. This lets several variants share a single reference set - e.g. OpenGL and Vulkan outputs should be nearly bitwise identical, so a gl variant can compare against the soft references:

"variants": {
	"soft": { "args": "--graphics=software" },
	"gl":   { "args": "--graphics=opengl", "compare-suffix": "soft" },
	"vul":  { "args": "--graphics=vulkan", "compare-suffix": "soft" }
}

A plain string value ("soft": "--graphics=software") is shorthand for { "args": "...", "suffix": "soft", "compare-suffix": "soft" }.

How a test runs

For each dump (recursively under testRoot) and each variant:

• If the variant's reference image <name>-<compare-suffix>.png is missing and the variant generates references (suffix == compare-suffix): the dump is rendered and the output saved as the new reference. Status: NEW. This is how references are created - run locally, then commit the generated images (also copied to <outputRoot>/generated/ for convenience).
• If the reference is missing for a variant that doesn't generate references (suffix != compare-suffix): Status: ERROR - the shared reference needs to be generated by the matching variant first.
• If the reference exists: the dump is rendered, the output saved to <outputRoot>/actuals/, and compared against the reference using MSE (mean squared error over R, G, B per pixel, alpha ignored). A visual comparison image (actual / reference+diff) is saved to <outputRoot>/diffs/ whenever a comparison runs. Status: PASS or FAIL (mismatch, crash, or timeout).

A FAIL with no MSE reported usually means the headless binary crashed before producing a screenshot, and a reference image that can't be loaded (corrupt) is reported as ERROR. The full emulator log of every non-passing test is kept in <outputRoot>/logs/.

Reference images

• PNG, 512×272 (480×272 display in a 512-wide framebuffer), stored top-down (row 0 = top of screen). The BMP output format is bottom-up per the BMP spec; the flip is applied only when writing BMPs.
• The alpha channel is forced to 255 when writing PNGs (games often use alpha for non-visual purposes, which would otherwise produce transparent-looking images); pass --screenshot-keep-alpha to the headless binary to preserve it (e.g. via a variant). The MSE comparison ignores alpha either way.
• Generated with --graphics=software they are fully deterministic: a subsequent run produces byte-identical output, so maxMse can be 0.
• If rendering code changes the output, existing references may need regenerating: delete the affected reference images and re-run to regenerate them.

Headless flags used

The runner uses --screenshot=<ref> (compare), --screenshot-save=<file> (save output; PNG if the path ends in .png, else BMP) and --screenshot-diff=<file> (always write a visual comparison when comparing). See headless/README.md for details.

--depth-save=<file> writes the current depth buffer next to the screenshot: a u32 width and height, then a u16 per pixel, or a float if the path ends in .f32. The software renderer gives the GE's own values. The hardware renderers keep depth at higher precision without the GE's rounding, so their u16 is floor(d * 65536) and usually within one of the software renderer's; the .f32 version keeps the fraction, which shows whether two draws of the same surface meet a depth test like >=.

Command line options

usage: frametests.py [OPTIONS] [CONFIG.json]

Option	Description
CONFIG.json	Path to the configuration file (default: frametests/frametests.json).
--filter=SUBSTR	Only run dumps whose relative path contains SUBSTR (case-insensitive).
--strict	Treat missing reference images as failures (a configuration error).
--out-mode=all|failures	all keeps everything in the output dir; failures keeps only artifacts of non-passing tests (smaller CI artifacts). Default: all.
--variants=NAME=ARGS,...	Run these variants instead of the config's.
--adb[=SERIAL]	Run on an Android device instead; see "Running on Android" below.

The headless binary is located, in order of preference:

1. headlessPath from the config file.
2. The PPSSPP_HEADLESS environment variable.
3. Well-known paths relative to the current directory (e.g. Windows/x64/Debug/PPSSPPHeadless.exe, build/PPSSPPHeadless), newest by modification time.

Running on Android

--adb runs every dump on an Android device over adb, with the ppsspp_headless executable from the legacy NDK build, and compares each screenshot with the host's Vulkan render of the same dump (through Tools/image_compare.py, so rasterizer noise doesn't count). The references in the config aren't used: the question is whether the device's drivers draw what a desktop GPU does.

cd android && NDK=/path/to/ndk ./ab.sh APP_ABI=arm64-v8a HEADLESS=1 SKIPAPP=1 -j8 && cd ..
python3 frametests.py frametests/frametests.json --adb

The first run pushes the binary, assets/ and the dumps to /data/local/tmp/ppsspp-frametests; later runs only push what changed. The variants default to vulkan=--graphics=vulkan --dump-compat,gl=--graphics=gles --dump-compat, and --host-args (default --graphics=vulkan --dump-compat) sets how the host renders its side. --dump-compat applies the game's compat.ini settings, which headless otherwise skips for dumps; without them some games look wrong on every backend (Coded Arms needs AccurateSplines). A failing test's diff image shows host, device and the differences side by side. Runs go to the device one at a time.

CI integration

In CI (GITHUB_ACTIONS is set) the runner behaves as if --strict was passed and prints ::error annotations for each failing test, so failures show up inline in the GitHub Actions log.

The CI workflow extends the existing test job (Linux leg) in .github/workflows/build.yml: the Fetch tests step inits the frametests submodule (alongside pspautotests), then after the pspautotests run it executes the frametests and uploads the report as an artifact:

- name: Fetch tests
  run: git submodule update --init pspautotests frametests

- name: Execute frametests
  if: runner.os == 'Linux'
  run: python3 frametests.py frametests/frametests.json --out-mode=failures

- name: Upload frametest report
  uses: actions/upload-artifact@v7
  if: runner.os == 'Linux' && always()
  with:
    name: frametest-report
    path: frametests/out/

A missing reference image in CI means the test set is incomplete - the run fails and the generated reference is available in frametests/out/generated/ (part of the uploaded artifact) so it can be committed.

Custom machines

The script is designed to run on machines with different GPUs and bigger test sets than CI. Copy the repo, point at a custom config:

python3 frametests.py /path/to/my-config.json

The config can live anywhere and point at any test data tree; only the script itself is shared. Add hardware-specific variants (--graphics=vulkan, --msaa=..., etc.) to the machine's own config - no code changes needed for new flag combinations, as long as the headless binary supports them.

Adding a new framedump

1. Drop the dump into frametests/dumps/ (either as a .ppdmp or zipped).
2. Run python3 frametests.py frametests/frametests.json --filter=<name> - the reference image(s) are generated and copied to <outputRoot>/generated/.
3. Commit the dump and the reference images under frametests/ref/, and push them to the frametests submodule repo (hrydgard/ppsspp-frametest-ci). Then bump the submodule pointer in PPSSPP so CI picks up the new test.

The CI test set (dumps and references) is maintained separately from the runner; the frametests/ submodule is just one of several possible test sets

• custom machines can use their own (possibly much larger) ones.

Comparing renderers

Two backends never match pixel for pixel: edges land a pixel apart, filtering and interpolation round differently, 16-bit framebuffers expand their colors differently. To find what really differs between, say, Vulkan and the software renderer (or a PSP capture), render the same dumps with each and compare the directories with Tools/image_compare.py:

python3 Tools/image_compare.py out_vulkan/ out_soft/ --sort --heatmaps heat/

It counts a difference only where a pixel is outside the range of the other image's 3x3 neighborhood (widened by a tolerance) and enough of its neighbors are too, so shifted edges and speckle drop out, and separately counts 8x8 blocks whose averages differ, for broad shifts. The heatmaps show both images and the counted pixels. On the GitHub dump set, Vulkan and OpenGL come out equivalent for 795 of 957 dumps, and what's left is mostly real (frames one of them renders black, missing effects).

Replaying a dump on a real PSP

The references above are PPSSPP's own renders, so they catch regressions, not wrong output. To see what the hardware draws, replay the dump on a PSP: pspautotests/utils/ppdmp-playback is a PSP program that runs a .ppdmp natively (all dump versions PPSSPP writes), and its run.py drives it over PSPLink from the host:

cd pspautotests
python3 utils/ppdmp-playback/run.py --out /tmp/shots \
	--headless ../build/PPSSPPHeadless "../frametests/dumps/Depth/21391 Coded arms ULUS10019_0001.zip"

It saves what the PSP displays as NAME-psp.png, and with --headless renders the same dump in PPSSPPHeadless (--graphics=software by default), saves NAME-ppsspp.png and prints the MSE between the two. Each frame stays on the PSP's screen for 1.5 seconds (--hold). It needs the PSPSDK (it rebuilds the PRX) and a PSP in PSPLink with usbhostfs_pc serving the pspautotests root, the same setup as pspautotests-hardware.md; like every hardware run, one at a time. See its README for replaying a subset of the primitives.

A zip holding several dumps plays its last .ppdmp, in PPSSPP and in the replayer, so extract that one when you need the file itself.

If a replay hangs, run.py resets the PSP and prints the last lines the replayer wrote. --progress=N prints a line every N dump commands and --trace-from=N every command from N on, and --cmds=N replays only the first N dump commands, which bisects a hang in a few runs. The replayer's Makefile has no header dependencies: after changing replay.h, delete the .o files, or a stale object corrupts the Replay object.

To find the draw where the two diverge, stop both at the same primitive: run.py --end=N on the PSP and --replay-end=N on PPSSPPHeadless. A draw into an offscreen buffer can be compared directly: run.py --display=ADDR,STRIDE,FMT shows and screenshots that buffer instead of the display (e.g. --display=04000000,512,3 for an 8888 target at the start of VRAM), and PPSSPPHeadless takes --screenshot-render-target to save the current render target.

PPSSPP's software renderer now matches the PSP bit for bit on nearly every dump here, so a difference is worth chasing. Hardware backends still differ slightly (filtering, edge rules), so compare their MSEs between versions instead.

To record a dump from a running game without the UI, use the WebSocket debugger's gpu.record.dump through Tools/wsdbg. Its reply comes later, so wait for it:

printf ':sleep 6\ngpu.record.dump\n:wait gpu.record.dump 60\n:quit\n' | \
	Tools/wsdbg/target/release/wsdbg --sync --compact --quiet --launch \
	build/PPSSPPHeadless game.iso --state=STATE.ppst --graphics=software --debugger-run=0 > out.txt

The reply's uri is a base64 data: URI holding the .ppdmp.