The test set lives in the 'frametests' git submodule (hrydgard/ppsspp-frametest-ci).
The test job previously only inited pspautotests, so the submodule was never
synced on CI and the (now removed) hashFiles gates skipped the frametests steps.
Use an HTTPS submodule URL so CI can fetch it without SSH keys.
Extend the existing 'test' job (Linux leg) with the frametests run after
the pspautotests, plus an artifact upload of the output so failures can
be examined. Both steps are gated on the test set being present in the
checkout, since dumps, references and config are maintained separately
from the runner.
Variants can now declare a compare-suffix (defaulting to the variant's own
suffix) to compare against another variant's reference images, enabling
shared references across e.g. opengl/vulkan when their output matches.
Reference images are only generated for variants whose suffix equals their
compare-suffix; other variants fail clearly if the shared reference is
missing. Rename the default output directory from frametest-out to out.
Games often use the framebuffer alpha channel for non-visual purposes,
so saved PNGs could look fully transparent in image viewers. Force alpha
to 255 when writing PNG screenshots unless --screenshot-keep-alpha is
passed. The MSE comparison ignores alpha either way.
- Add frametests.py: walks a dump tree, renders each dump per config variant
through PPSSPPHeadless, generates reference images when missing and compares
MSE when present, and writes a self-contained HTML report. The JSON config
(which lives with the test set, not in the repo) points at the data tree
and defines variants as suffix -> CLI args, e.g. 'soft': '--graphics=software'.
- Headless: --screenshot-save saves PNG when the path ends in .png; new
--screenshot-diff always writes a visual comparison when comparing;
screenshot comparison failures (mismatch or unloadable reference) now fail
the test instead of passing silently.
- Read back framebuffers top-down, flipping only for BMP output/input
(fixes upside-down PNG references). Sync libretro copy accordingly.
- Document the system in docs/frametest.md; add AGENTS.md reference.
Henrik Rydgård