ppsspp/docs/pspautotests.md

pspautotests

This test suite is located in the pspautotests submodule/subdirectory at the root of the repo. Its tests/ subdirectory contains tests arranged by category (audio/, cpu/, gpu/, threads/, etc.).

The runner script test.py at the repo root contains two lists: tests_good (known-passing regression tests) and tests_next (tests that don't yet pass — move things here → tests_good by fixing PPSSPP).

Tests are compiled as PRX binaries (a variation on ELF) which PPSSPP can load. They also have an .expected file alongside them, containing the reference output from running on a real PSP via PSPLink. The goal is for PPSSPPHeadless's output to match.

Prerequisites

• Initialize the submodule (if not done): git submodule update --init
• The PRX binaries and .expected files are already committed inside the submodule — no need to compile tests yourself just to run them.
• To build PPSSPPHeadless itself, see the next section.

Building PPSSPPHeadless

Windows (Visual Studio)

Open Windows/PPSSPP.sln in Visual Studio and build the PPSSPPHeadless project, or use MSBuild directly:

MSBuild.exe -noAutoResponse Windows/PPSSPP.sln -target:PPSSPPHeadless -property:Configuration=Debug -property:Platform=x64 -maxCpuCount

The built executable will be at Windows/x64/Debug/PPSSPPHeadless.exe.

CMake-based builds do exist but are not the recommended way on Windows — stick with the VS solution.

Linux / macOS

Use ./b.sh --debug, then find the executable under build/ (e.g. build/Debug/PPSSPPHeadless).

Running tests

Using test.py (recommended)

Make sure Python is available. On Windows the Microsoft Store alias may interfere; use py or the full path to python.exe.

• Run all good tests: py test.py -g
• Run all broken/next tests: py test.py -b
• Run all tests (good + next): py test.py
• Run a specific test (or space-separated list): py test.py -g cpu/cpu_alu/cpu_alu
• Run a group of tests by prefix match (add -m): py test.py -g -m audio/atrac

Direct headless invocation

Windows/x64/Debug/PPSSPPHeadless.exe --root pspautotests/tests/../ --compare --timeout-wall=5 --graphics=software pspautotests/tests/audio/atrac/addstreamdata.prx

Instead of a single PRX, you can pass a directory (e.g. pspautotests/tests/threads/mbx/...) to run all tests under it, recursively.

Key flags:

• --root — points to the directory above tests/ so the headless can find the expected directory layout.
• --compare — enables output comparison against .expected files.
• --timeout-wall=N — real seconds per test before killing it (default 5). --timeout-emulated=N is the same idea in emulated time, and both can be set at once. --timeout is the old name for --timeout-wall.
• --graphics=software — uses software GPU backend (required for headless; no real GPU available).

Tests that need the real firmware modules

A pspautotest is homebrew, so headless runs every library on our HLE for it (iForceEnableHLE in headless/Headless.cpp), even the ones the app runs from the firmware by default. Some tests only mean anything against the real module: video/mp4/avchold and video/mpeg/decodeend test the Media Engine through Sony's libmp4.prx, mp4msv.prx and mpeg.prx, which call our sceVideocodec HLE. Run on our HLE, avchold reports 0 samples, 0 decodes for every stream (our sceMp4 is stubs: the log says UNIMPL sceMp4Create), and it fails without exercising the decoder at all.

To run them as the hardware does, pass both:

• --disable-hle=N, the DisableHLEFlags bits (Core/ConfigValues.h) of the libraries to run for real, in decimal or 0x hex. sceMpeg is 0x10 and sceMp4 is 0x100, so the video tests take --disable-hle=0x110.
• --memstick= pointing at a memory stick with a firmware dump in PSP/NAND/flash0/kd. test.py otherwise hands headless the empty one beside the exe. Since the flag is explicit, headless exits with an error naming the missing module rather than falling back to HLE.

python test.py video/mp4/avchold video/mpeg/decodeend --graphics=software --memstick=<app memstick> --disable-hle=0x110

CI has no firmware dump, so these tests stay in tests_next, and a change to the video path needs this run done by hand.

What you'll see

Passing test:

pspautotests/tests/audio/atrac/addstreamdata.prx:
  audio/atrac/addstreamdata - passed!
1 tests passed, 0 tests failed, 0 tests missing.

Failing test:

pspautotests/tests/threads/mbx/refer/refer.prx:
O   hi 0 prio=00 next=OTHER  hi 1 prio=00 next=ITSELF  ...
E   hi 1 prio=00 next=ITSELF  ...
+   ...
0 tests passed, 1 tests failed, 0 tests missing.
Failed tests:
  threads/mbx/refer/refer

Lines prefixed with O are what PPSSPP produced (the Output), E is the corresponding line from the .expected file recorded on a real PSP, and + means a match.

The diff is line-by-line, so an O line followed by an E line at the same conceptual position means PPSSPP produced different output at that spot. A + line means both outputs agreed on that line.

The [x] and [r] markers - they're about scheduling

Most tests print their lines through checkpoint() in pspautotests/common/common.c, which tags every line with [x] or [r]:

O [x] While open: OK (allocated 12)
E [x] While open: OK (allocated 0)

The tag is not decoration. Each checkpoint() call starts a helper thread that does nothing but set a flag, then terminates and restarts it for the next one. If that thread got a chance to run before the next checkpoint, the line is tagged [r] - a reschedule happened while the code under test ran. If it never got scheduled, the line is tagged [x]. (With CHECKPOINT_ENABLE_TIME the tag becomes [x/1234], adding the microseconds since the previous checkpoint.)

So the marker records whether the syscall between the two checkpoints yielded to another thread, which is a big part of what these tests are checking.

That means a diff where only the marker differs, like

O [x]   GetCharInfo on open: 00000000
E [r]   GetCharInfo on open: 00000000

is not a value bug at all - the return value matched. It says our implementation of that call doesn't reschedule where the real one does (or the other way round). Fixing it means changing whether the HLE function yields - hleReSchedule, hleDelayResult and friends - not what it returns. Don't go hunting for a wrong value; there isn't one.

Workflow for fixing a test

1. Pick a test from tests_next in test.py.
2. Run it with headless to confirm failure and see what differs (O vs E lines).
3. Read the test source (.c/.cpp) and the .expected file to understand the API being tested. The .expected file was recorded from a real PSP — it's the ground truth. The test source reveals what syscalls are made and in what order. Sometimes the test deliberately corrupts state to probe kernel error handling.
4. Form a hypothesis: look for a systematic pattern in the diffs (wrong order, wrong error code, missing output). Cross-reference with multiple expected files that exercise the same API — they may reveal the PSP's real behavior from different angles.
5. Make changes to PPSSPP's HLE or other core code. Do not make super-targeted changes just to fix the test — instead, fix the underlying issue in a way that would also make sense on real PSP hardware.
6. Rebuild PPSSPPHeadless and re-run the test.
7. Run the full tests_good suite (py test.py -g) to check for regressions. A correct fix should not break any previously passing tests.
8. Rinse and repeat until it passes, then move it from tests_next to tests_good in test.py.

Tips from experience

• The diff output compares the full text output line-by-line. To see PPSSPP's raw output without the diff overlay, omit --compare:

  Windows/x64/Debug/PPSSPPHeadless.exe --root pspautotests/tests/../ --timeout-wall=5 --graphics=software path/to/test.prx
  

  There's another trick too, --print-equal-lines, which prints matching lines with a '=' prefix, so you can see the full output with context.
• Tests can show contradictory expected outputs at first glance. For example, the mbx/send test expected file shows FIFO message order (normal sends), while the mbx/refer test shows LIFO (after corruption) — because sceKernelReferMbxStatus on a real PSP updates firstMessage during traversal, changing the apparent head. Understanding the expected file's behavior often requires reading multiple related tests together.
• When searching for the underlying issue, trace through the HLE implementation with the test's syscall sequence. Check whether the kernel writes into PSP-visible memory — if so, test code can corrupt those values, and the PSP kernel may have specific handling for that.
• Pointer addresses differ between PSP and PPSSPP. A test that prints raw kernel pointers (heap addresses, TLS block addresses, etc.) will always have mismatched expected output because PSPLink shifts memory layout. Fix by changing the test to print offsets from a base address instead of absolute addresses. After changing the test, run it on hardware to generate a new .expected.
• The diff notation:
  • O line = present in PPSSPP's output but not in expected (PPSSPP-only).
  • E line = present in expected but not in PPSSPP output (expected-only).
  • + line = context line (shown around diffs for context).
  • = line (with --print-equal-lines) = matching line.
  • [r] / [x] prefix = rescheduling occurred / did not occur since last line.
• When a function is a stub (UNIMPL in log), the expected output is a good specification for what to implement. Look at multiple test cases in the expected file to understand the full range of valid and invalid inputs, return codes, and side effects.
• Time-dependent tests (RTC, timezone conversions) depend on the host machine's clock and timezone. The PSP's sceRtcParseDateTime was a stub — the expected file showed exactly which RFC 3339 / RFC 2822 formats are accepted and which are rejected (return -1).

Troubleshooting

• "Python was not found" on Windows — the Microsoft Store alias is interfering. Use py (the Python launcher) or the full path, e.g. "C:/Users/.../AppData/Local/Programs/Python/Python314/python.exe" test.py -g. Or disable the alias in Settings > Apps > Advanced app settings > App execution aliases.
• No PRX files found — run git submodule update --init from the repo root to fetch the pspautotests submodule.
• PPSSPPHeadless exits immediately / "CPU not started" — the test PRX may be missing or the --root path is wrong. Ensure --root points to the parent of tests/.
• MSBuild error MSB1008 — the MSBuild.rsp response file may be interfering. Always pass -noAutoResponse on Windows builds.