# Framedump rendering tests (frametests) PPSSPP has a test system that replays GE frame dumps (`.ppdmp`) through `PPSSPPHeadless` with various rendering settings, comparing the rendered output against stored reference images. It is designed to catch rendering regressions across GPU backends and configurations. The system has three parts: - **The test runner**: `frametests.py` (repo root). Test-set agnostic - it works against any test data tree pointed to by its configuration. - **The configuration**: a JSON file describing where the test data lives and which rendering variants to run (e.g. `frametests/frametests.json`). - **The test set**: the frame dumps and reference images, maintained as the `frametests` git submodule (repo `hrydgard/ppsspp-frametest-ci`, pinned by commit - its config `frametests.json` lives inside it). Different environments can use different test sets - GitHub CI uses the submodule, custom machines can use their own (possibly much larger) ones. ## Quick start ```bash # Build PPSSPPHeadless (see headless/README.md), then: python3 frametests.py frametests/frametests.json ``` The first run generates reference images for any dumps that don't have them yet (status `NEW`) and compares the rest (status `PASS`/`FAIL`). A summary is printed and a self-contained HTML report is written to `frametests/out/report.html` (default output dir; gitignored). Exit code is `0` if all tests passed, `1` if anything failed, `2` on configuration/usage errors. ## Configuration The configuration is a single JSON file. All relative paths are resolved against the config file's directory, so the same script works with any test set by just pointing `--config` at a different file. The config lives with the test set it describes (e.g. `frametests/frametests.json`) and is not committed to the repository. ```json { "testRoot": "dumps", "refRoot": "ref", "outputRoot": "out", "headlessPath": "", "timeout": 60, "maxMse": 0.0, "variants": { "soft": { "args": "--graphics=software" } } } ``` | Key | Description | |-----------------|-----------------------------------------------------------------------------| | `testRoot` | Directory tree containing frame dumps (`.ppdmp` files, possibly wrapped in `.zip`). | | `refRoot` | Where reference images live; mirrors the `testRoot` tree. | | `outputRoot` | Where logs, diff images, the report, and generated references go (default `out`). | | `headlessPath` | Path to the `PPSSPPHeadless` binary. Empty = auto-detect (see below). | | `timeout` | Per-test timeout in seconds. | | `maxMse` | Maximum allowed MSE for screenshot comparison (0 = exact match). | | `variants` | Map of variant name to variant configuration (see below). | ### Variants Each test runs once per variant. A variant has a **suffix** and a **compare-suffix**: - The **suffix** (default: the variant name) is appended to the variant's own output image filenames (`actuals`, `diffs`, `logs`). - The **compare-suffix** (default: the suffix) names the reference image the variant compares against: `-.png`. - **Reference images are only generated for variants whose suffix matches their compare-suffix.** This lets several variants share a single reference set - e.g. OpenGL and Vulkan outputs should be nearly bitwise identical, so a `gl` variant can compare against the `soft` references: ```json "variants": { "soft": { "args": "--graphics=software" }, "gl": { "args": "--graphics=opengl", "compare-suffix": "soft" }, "vul": { "args": "--graphics=vulkan", "compare-suffix": "soft" } } ``` A plain string value (`"soft": "--graphics=software"`) is shorthand for `{ "args": "...", "suffix": "soft", "compare-suffix": "soft" }`. ## How a test runs For each dump (recursively under `testRoot`) and each variant: - If the variant's reference image `-.png` is **missing** and the variant generates references (suffix == compare-suffix): the dump is rendered and the output saved as the new reference. Status: `NEW`. This is how references are created - run locally, then commit the generated images (also copied to `/generated/` for convenience). - If the reference is **missing** for a variant that doesn't generate references (suffix != compare-suffix): Status: `ERROR` - the shared reference needs to be generated by the matching variant first. - If the reference **exists**: the dump is rendered, the output saved to `/actuals/`, and compared against the reference using MSE (mean squared error over R, G, B per pixel, alpha ignored). A visual comparison image (actual / reference+diff) is saved to `/diffs/` whenever a comparison runs. Status: `PASS` or `FAIL` (mismatch, crash, or timeout). A `FAIL` with no MSE reported usually means the headless binary crashed before producing a screenshot, and a reference image that can't be loaded (corrupt) is reported as `ERROR`. The full emulator log of every non-passing test is kept in `/logs/`. ### Reference images - PNG, 512×272 (480×272 display in a 512-wide framebuffer), stored top-down (row 0 = top of screen). The BMP output format is bottom-up per the BMP spec; the flip is applied only when writing BMPs. - The alpha channel is forced to 255 when writing PNGs (games often use alpha for non-visual purposes, which would otherwise produce transparent-looking images); pass `--screenshot-keep-alpha` to the headless binary to preserve it (e.g. via a variant). The MSE comparison ignores alpha either way. - Generated with `--graphics=software` they are fully deterministic: a subsequent run produces byte-identical output, so `maxMse` can be 0. - If rendering code changes the output, existing references may need regenerating: delete the affected reference images and re-run to regenerate them. ### Headless flags used The runner uses `--screenshot=` (compare), `--screenshot-save=` (save output; PNG if the path ends in `.png`, else BMP) and `--screenshot-diff=` (always write a visual comparison when comparing). See `headless/README.md` for details. ## Command line options ``` usage: frametests.py [OPTIONS] [CONFIG.json] ``` | Option | Description | |--------------------|-----------------------------------------------------------------------| | `CONFIG.json` | Path to the configuration file (default: `frametests/frametests.json`). | | `--filter=SUBSTR` | Only run dumps whose relative path contains SUBSTR (case-insensitive). | | `--strict` | Treat missing reference images as failures (a configuration error). | | `--out-mode=all\|failures` | `all` keeps everything in the output dir; `failures` keeps only artifacts of non-passing tests (smaller CI artifacts). Default: `all`. | The headless binary is located, in order of preference: 1. `headlessPath` from the config file. 2. The `PPSSPP_HEADLESS` environment variable. 3. Well-known paths relative to the current directory (e.g. `Windows/x64/Debug/PPSSPPHeadless.exe`, `build/PPSSPPHeadless`), newest by modification time. ## CI integration In CI (`GITHUB_ACTIONS` is set) the runner behaves as if `--strict` was passed and prints `::error` annotations for each failing test, so failures show up inline in the GitHub Actions log. The CI workflow extends the existing `test` job (Linux leg) in `.github/workflows/build.yml`: the `Fetch tests` step inits the `frametests` submodule (alongside `pspautotests`), then after the pspautotests run it executes the frametests and uploads the report as an artifact: ```yaml - name: Fetch tests run: git submodule update --init pspautotests frametests - name: Execute frametests if: runner.os == 'Linux' run: python3 frametests.py frametests/frametests.json --out-mode=failures - name: Upload frametest report uses: actions/upload-artifact@v7 if: runner.os == 'Linux' && always() with: name: frametest-report path: frametests/out/ ``` A missing reference image in CI means the test set is incomplete - the run fails and the generated reference is available in `frametests/out/generated/` (part of the uploaded artifact) so it can be committed. ## Custom machines The script is designed to run on machines with different GPUs and bigger test sets than CI. Copy the repo, point at a custom config: ```bash python3 frametests.py /path/to/my-config.json ``` The config can live anywhere and point at any test data tree; only the script itself is shared. Add hardware-specific variants (`--graphics=vulkan`, `--msaa=...`, etc.) to the machine's own config - no code changes needed for new flag combinations, as long as the headless binary supports them. ## Adding a new framedump 1. Drop the dump into `frametests/dumps/` (either as a `.ppdmp` or zipped). 2. Run `python3 frametests.py frametests/frametests.json --filter=` - the reference image(s) are generated and copied to `/generated/`. 3. Commit the dump and the reference images under `frametests/ref/`, and push them to the `frametests` submodule repo (`hrydgard/ppsspp-frametest-ci`). Then bump the submodule pointer in PPSSPP so CI picks up the new test. The CI test set (dumps and references) is maintained separately from the runner; the `frametests/` submodule is just one of several possible test sets - custom machines can use their own (possibly much larger) ones.