# Framedump rendering tests (frametests) PPSSPP has a test system that replays GE frame dumps (`.ppdmp`) through `PPSSPPHeadless` with various rendering settings, comparing the rendered output against stored reference images. It is designed to catch rendering regressions across GPU backends and configurations. The system has three parts: - **The test runner**: `frametests.py` (repo root). Test-set agnostic - it works against any test data tree pointed to by its configuration. - **The configuration**: a JSON file describing where the test data lives and which rendering variants to run (e.g. `frametests/frametests.json`). - **The test set**: the frame dumps and reference images, maintained as the `frametests` git submodule (repo `hrydgard/ppsspp-frametest-ci`, pinned by commit - its config `frametests.json` lives inside it). Different environments can use different test sets - GitHub CI uses the submodule, custom machines can use their own (possibly much larger) ones. ## Quick start ```bash # Build PPSSPPHeadless (see headless/README.md), then: python3 frametests.py frametests/frametests.json ``` The first run generates reference images for any dumps that don't have them yet (status `NEW`) and compares the rest (status `PASS`/`FAIL`). A summary is printed and a self-contained HTML report is written to `frametests/out/report.html` (default output dir; gitignored). Exit code is `0` if all tests passed, `1` if anything failed, `2` on configuration/usage errors. ## Configuration The configuration is a single JSON file. All relative paths are resolved against the config file's directory, so the same script works with any test set by just pointing `--config` at a different file. The config lives with the test set it describes (e.g. `frametests/frametests.json`) and is not committed to the repository. ```json { "testRoot": "dumps", "refRoot": "ref", "outputRoot": "out", "headlessPath": "", "timeout": 60, "maxMse": 0.0, "variants": { "soft": { "args": "--graphics=software" } } } ``` | Key | Description | |-----------------|-----------------------------------------------------------------------------| | `testRoot` | Directory tree containing frame dumps (`.ppdmp` files, possibly wrapped in `.zip`). | | `refRoot` | Where reference images live; mirrors the `testRoot` tree. | | `outputRoot` | Where logs, diff images, the report, and generated references go (default `out`). | | `headlessPath` | Path to the `PPSSPPHeadless` binary. Empty = auto-detect (see below). | | `timeout` | Per-test timeout in seconds. | | `maxMse` | Maximum allowed MSE for screenshot comparison (0 = exact match). | | `variants` | Map of variant name to variant configuration (see below). | ### Variants Each test runs once per variant. A variant has a **suffix** and a **compare-suffix**: - The **suffix** (default: the variant name) is appended to the variant's own output image filenames (`actuals`, `diffs`, `logs`). - The **compare-suffix** (default: the suffix) names the reference image the variant compares against: `-.png`. - **Reference images are only generated for variants whose suffix matches their compare-suffix.** This lets several variants share a single reference set - e.g. OpenGL and Vulkan outputs should be nearly bitwise identical, so a `gl` variant can compare against the `soft` references: ```json "variants": { "soft": { "args": "--graphics=software" }, "gl": { "args": "--graphics=opengl", "compare-suffix": "soft" }, "vul": { "args": "--graphics=vulkan", "compare-suffix": "soft" } } ``` A plain string value (`"soft": "--graphics=software"`) is shorthand for `{ "args": "...", "suffix": "soft", "compare-suffix": "soft" }`. ## How a test runs For each dump (recursively under `testRoot`) and each variant: - If the variant's reference image `-.png` is **missing** and the variant generates references (suffix == compare-suffix): the dump is rendered and the output saved as the new reference. Status: `NEW`. This is how references are created - run locally, then commit the generated images (also copied to `/generated/` for convenience). - If the reference is **missing** for a variant that doesn't generate references (suffix != compare-suffix): Status: `ERROR` - the shared reference needs to be generated by the matching variant first. - If the reference **exists**: the dump is rendered, the output saved to `/actuals/`, and compared against the reference using MSE (mean squared error over R, G, B per pixel, alpha ignored). A visual comparison image (actual / reference+diff) is saved to `/diffs/` whenever a comparison runs. Status: `PASS` or `FAIL` (mismatch, crash, or timeout). A `FAIL` with no MSE reported usually means the headless binary crashed before producing a screenshot, and a reference image that can't be loaded (corrupt) is reported as `ERROR`. The full emulator log of every non-passing test is kept in `/logs/`. ### Reference images - PNG, 512×272 (480×272 display in a 512-wide framebuffer), stored top-down (row 0 = top of screen). The BMP output format is bottom-up per the BMP spec; the flip is applied only when writing BMPs. - The alpha channel is forced to 255 when writing PNGs (games often use alpha for non-visual purposes, which would otherwise produce transparent-looking images); pass `--screenshot-keep-alpha` to the headless binary to preserve it (e.g. via a variant). The MSE comparison ignores alpha either way. - Generated with `--graphics=software` they are fully deterministic: a subsequent run produces byte-identical output, so `maxMse` can be 0. - If rendering code changes the output, existing references may need regenerating: delete the affected reference images and re-run to regenerate them. ### Headless flags used The runner uses `--screenshot=` (compare), `--screenshot-save=` (save output; PNG if the path ends in `.png`, else BMP) and `--screenshot-diff=` (always write a visual comparison when comparing). See `headless/README.md` for details. ## Command line options ``` usage: frametests.py [OPTIONS] [CONFIG.json] ``` | Option | Description | |--------------------|-----------------------------------------------------------------------| | `CONFIG.json` | Path to the configuration file (default: `frametests/frametests.json`). | | `--filter=SUBSTR` | Only run dumps whose relative path contains SUBSTR (case-insensitive). | | `--strict` | Treat missing reference images as failures (a configuration error). | | `--out-mode=all\|failures` | `all` keeps everything in the output dir; `failures` keeps only artifacts of non-passing tests (smaller CI artifacts). Default: `all`. | The headless binary is located, in order of preference: 1. `headlessPath` from the config file. 2. The `PPSSPP_HEADLESS` environment variable. 3. Well-known paths relative to the current directory (e.g. `Windows/x64/Debug/PPSSPPHeadless.exe`, `build/PPSSPPHeadless`), newest by modification time. ## CI integration In CI (`GITHUB_ACTIONS` is set) the runner behaves as if `--strict` was passed and prints `::error` annotations for each failing test, so failures show up inline in the GitHub Actions log. The CI workflow extends the existing `test` job (Linux leg) in `.github/workflows/build.yml`: the `Fetch tests` step inits the `frametests` submodule (alongside `pspautotests`), then after the pspautotests run it executes the frametests and uploads the report as an artifact: ```yaml - name: Fetch tests run: git submodule update --init pspautotests frametests - name: Execute frametests if: runner.os == 'Linux' run: python3 frametests.py frametests/frametests.json --out-mode=failures - name: Upload frametest report uses: actions/upload-artifact@v7 if: runner.os == 'Linux' && always() with: name: frametest-report path: frametests/out/ ``` A missing reference image in CI means the test set is incomplete - the run fails and the generated reference is available in `frametests/out/generated/` (part of the uploaded artifact) so it can be committed. ## Custom machines The script is designed to run on machines with different GPUs and bigger test sets than CI. Copy the repo, point at a custom config: ```bash python3 frametests.py /path/to/my-config.json ``` The config can live anywhere and point at any test data tree; only the script itself is shared. Add hardware-specific variants (`--graphics=vulkan`, `--msaa=...`, etc.) to the machine's own config - no code changes needed for new flag combinations, as long as the headless binary supports them. ## Adding a new framedump 1. Drop the dump into `frametests/dumps/` (either as a `.ppdmp` or zipped). 2. Run `python3 frametests.py frametests/frametests.json --filter=` - the reference image(s) are generated and copied to `/generated/`. 3. Commit the dump and the reference images under `frametests/ref/`, and push them to the `frametests` submodule repo (`hrydgard/ppsspp-frametest-ci`). Then bump the submodule pointer in PPSSPP so CI picks up the new test. The CI test set (dumps and references) is maintained separately from the runner; the `frametests/` submodule is just one of several possible test sets - custom machines can use their own (possibly much larger) ones. ## Replaying a dump on a real PSP The references above are PPSSPP's own renders, so they catch regressions, not wrong output. To see what the hardware draws, replay the dump on a PSP: `pspautotests/utils/ppdmp-playback` is a PSP program that runs a `.ppdmp` natively (all dump versions PPSSPP writes), and its `run.py` drives it over PSPLink from the host: ```bash cd pspautotests python3 utils/ppdmp-playback/run.py --out /tmp/shots \ --headless ../build/PPSSPPHeadless "../frametests/dumps/Depth/21391 Coded arms ULUS10019_0001.zip" ``` It saves what the PSP displays as `NAME-psp.png`, and with `--headless` renders the same dump in PPSSPPHeadless (`--graphics=software` by default), saves `NAME-ppsspp.png` and prints the MSE between the two. Each frame stays on the PSP's screen for 1.5 seconds (`--hold`). It needs the PSPSDK (it rebuilds the PRX) and a PSP in PSPLink with `usbhostfs_pc` serving the pspautotests root, the same setup as [pspautotests-hardware.md](pspautotests-hardware.md); like every hardware run, one at a time. See its README for replaying a subset of the primitives. A zip holding several dumps plays its last `.ppdmp`, in PPSSPP and in the replayer, so extract that one when you need the file itself. If a replay hangs, `run.py` resets the PSP and prints the last lines the replayer wrote. `--progress=N` prints a line every N dump commands and `--trace-from=N` every command from N on, and `--cmds=N` replays only the first N dump commands, which bisects a hang in a few runs. The replayer's Makefile has no header dependencies: after changing `replay.h`, delete the `.o` files, or a stale object corrupts the `Replay` object. To find the draw where the two diverge, stop both at the same primitive: `run.py --end=N` on the PSP and `--replay-end=N` on PPSSPPHeadless. A draw into an offscreen buffer can be compared directly: `run.py --display=ADDR,STRIDE,FMT` shows and screenshots that buffer instead of the display (e.g. `--display=04000000,512,3` for an 8888 target at the start of VRAM), and PPSSPPHeadless takes `--screenshot-render-target` to save the current render target. PPSSPP's software renderer now matches the PSP bit for bit on nearly every dump here, so a difference is worth chasing. Hardware backends still differ slightly (filtering, edge rules), so compare their MSEs between versions instead. To record a dump from a running game without the UI, use the WebSocket debugger's `gpu.record.dump` through `Tools/wsdbg`. Its reply comes later, so wait for it: ```bash printf ':sleep 6\ngpu.record.dump\n:wait gpu.record.dump 60\n:quit\n' | \ Tools/wsdbg/target/release/wsdbg --sync --compact --quiet --launch \ build/PPSSPPHeadless game.iso --state=STATE.ppst --graphics=software --debugger-run=0 > out.txt ``` The reply's `uri` is a base64 `data:` URI holding the `.ppdmp`.