Document the headless traps that produce convincing but bogus measurements

Running a commercial game through headless to measure something has four ways to
report a clean pass for a run that proved nothing, and they compound: --log is
needed before anything is printed (and it goes to stderr, so grepping forces the
streams together), the memory stick defaults to one beside the exe rather than
the app's, which means no installed firmware and a silent LLE-to-HLE fallback,
an unrecognised parameter then hides in the merged output, and a count of zero
errors reads identically whether the code ran or never got there.

All four of these cost a round of wrong results while investigating ME memory.

Also records that the line-ending check has to be done on the bytes, since Git
Bash's grep normalises them: the obvious `grep -c $'\r$'` for a stray LF reports
every file clean, which is how a handful of LF lines got into two CRLF files
here without the usual whole-file-rewrite tell in git diff --stat.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Henrik RydgårdandClaude Opus 5 committed 2026-09-20 16:39:35 -06:00
1 parent 1a08f3c88c
commit d40953d09b
2 files changed
+41 -1

No files matched your search

+16 -1
View File
@@ -12,7 +12,7 @@ for it:
| Doc | When you need it |
|---|---|
| [docs/building.md](docs/building.md) | Build commands for every target (VS/MSBuild, CMake, UWP, legacy Android NDK, libretro), unit tests, pspautotests |
| [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, breakpoint reliability per CPU backend, debugging a game that works on hardware |
| [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, measuring a commercial game with headless, breakpoint reliability per CPU backend, debugging a game that works on hardware |
| [docs/DebuggerThreading.md](docs/DebuggerThreading.md) | `Core_RunOnCPUThread` / `g_frameMutex` / shutdown-lock rules - required reading before touching debugger code |
| [docs/HLEModules.md](docs/HLEModules.md) | Adding an HLE module or function, and the seven build files a new source file goes in |
| [docs/translations.md](docs/translations.md) | Translating UI strings with Tools/langtool |
@@ -48,6 +48,14 @@ for it:
`newline=''` silently converts the whole file, turning a two-line addition into a 5000-line diff.
Check `git diff --stat` before committing - a whole-file rewrite is obvious there and invisible in
the editor. Prefer the Edit tool, which does exact string replacement and can't do this.
**Don't verify this with `grep -c $'\r$'`** - Git Bash's grep normalises line endings, so it counts
a bare-LF line as having a CR and reports any file as clean. The opposite mistake to a whole-file
rewrite is inserting a handful of LF lines into a CRLF file (a Python `"""..."""` block written with
`newline=''` does exactly this), and that shows in `--stat` only as the lines you meant to add. Check
the bytes instead, e.g. `python -c "d=open(F,'rb').read(); print(d.count(b'\r\n'), d.count(b'\n'))"`,
and treat git's "LF will be replaced by CRLF the next time Git touches it" as the warning it is
rather than autocrlf noise.
6. **Don't feed Python to `bash -c` via a heredoc when the code needs a literal backslash in its
*output*.** Git Bash strips one level of escaping on the way in even with a quoted delimiter
(`<<'PY'`), so `"\\n"` reaches Python as `"\n"` and writes a real newline into the file - no error,
@@ -111,6 +119,13 @@ python test.py -g --graphics=software
around a hundred failures that mean nothing is wrong. The only meaningful result is `0 tests failed`.
(The debug-CRT "Detected memory leaks!" dump after the summary line is normal, not a failure.)
When the thing under test is a commercial game rather than a suite, headless needs `--log` before it prints
anything (to stderr), and defaults its memory stick to `<exe dir>/memstick` rather than the app's - so the
firmware you installed in the app isn't there, and LLE modules silently fall back to HLE. A run configured
differently from what you asked for, or one that never reached the code, produces the same all-zero counts as
a clean one, so assert the exit code and a positive "we got here" counter before believing any error count.
The traps in full: [docs/debugging.md](docs/debugging.md).
New unit tests are added to `availableTests`; large ones go in their own file in `unittest/`, which has
to be listed in **three** build files, not two: `CMakeLists.txt`, `unittest/UnitTests.vcxproj` (and its
`.filters`), and `android/jni/Android.mk`, which builds a unit test executable of its own. Miss the last
+25
View File
@@ -146,6 +146,31 @@ A working invocation, and the traps around it:
a lot of emulation speed (a 25-second run of a game playing video wrote 450k lines and ran several times slower
than real time, which on its own looks like the stall you're hunting).
## Measuring a commercial game with headless
Headless will happily run a game and print nothing, or run *a different configuration than the one you asked
for*, and both look like a clean pass if you are counting log lines. Four traps, each of which silently
produced a round of bogus results here:
- **`--log` is required for any log output at all**, and the log goes to **stderr**. Without it you get only
headless's own lines (`Loaded State`, `TIMEOUT`); `-d`/`-v` set the level but don't turn the printf logger
on. So folding stderr in isn't optional if you're grepping, which is what makes the third trap bite.
- **Headless's memory stick is not the app's.** It defaults to `<exe dir>/memstick` (`headless/Headless.cpp`),
so `Windows/<platform>/<config>/memstick`, while the app derives its own from `installed.txt` or Documents
(`InitMemstickDirectory` in `Windows/main.cpp`, which carries a TODO about sharing the derivation). Headless
*creates* that directory, empty `PSP/NAND/flash0` and all, so firmware installed through the app is invisible
and every LLE module quietly falls back to HLE - at which point you are measuring the HLE you were trying to
compare against. Pass `--memstick=` explicitly; it resolves relative to the CWD, not the exe.
- **Check the exit code.** An unrecognised parameter prints `Error: ...` to stderr and exits 1, which is
correct and easy to throw away: fold stderr into stdout (which the first trap forces), count grep hits, and a
run that never started reports the same all-zero line as a clean one.
- **Zero is not a pass.** Measuring by log-grepping needs a positive precondition asserted separately ("did
this run reach the code at all"), or "0 errors" also means "0 anything" - which is equally what a failed
boot, a savestate that never reaches the cutscene, and a silent HLE fallback produce.
The last three compound: the fix is to treat the run's exit code and a positive "we got here" counter as
preconditions, and only then believe the error counts.
## Debugging and breakpoint considerations
It might be worth trying the interpreter - all types of breakpoints are the most reliable with this CPU backend.