Merge pull request #22955 from hrydgard/android-unittest-runner

Android: Run the unit tests on a device, and make them pass there

AGENTS.md

@@ -12,7 +12,7 @@ for it:
 | Doc | When you need it |
 |---|---|
 | [docs/building.md](docs/building.md) | Build commands for every target (VS/MSBuild, CMake, UWP, legacy Android NDK, libretro), unit tests, pspautotests |
-| [docs/android-device-testing.md](docs/android-device-testing.md) | Running a build on a real Android device or a Quest over adb: installing the build you think you are, forcing activity recreates, measuring retained activities, VR |
+| [docs/android-device-testing.md](docs/android-device-testing.md) | Running a build on a real Android device or a Quest over adb: installing the build you think you are, forcing activity recreates, measuring retained activities, VR, running the C++ unit tests on a device (`Tools/android_unittest.py`) |
 | [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, measuring a commercial game with headless, comparing binaries' speed (`Tools/headless_bench.py`), finding a game's hot JIT blocks (`--jit-profile`), breakpoint reliability per CPU backend, debugging a game that works on hardware, finding and hashing a game function (like its own `memcpy`) for `ReplaceTables` when hardware backends miss CPU writes to VRAM |
 | [docs/DebuggerThreading.md](docs/DebuggerThreading.md) | `Core_RunOnCPUThread` / `g_frameMutex` / shutdown-lock rules - required reading before touching debugger code |
 | [docs/HLEModules.md](docs/HLEModules.md) | Adding an HLE module or function, and the seven build files a new source file goes in |

Common/MemArenaAndroid.cpp

@@ -102,10 +102,11 @@ bool MemArena::GrabMemSpace(size_t size) {
 	// Use ashmem so we don't have to allocate a file on disk!
 	const char* name = "PPSSPP_RAM";
 
-	// Since version 26 Android provides a new api for accessing SharedMemory.
-	if (System_GetPropertyInt(SYSPROP_SYSTEMVERSION) >= 26) {
-		fd = ashmem_create_region(name, size);
-	} else {
+	// Since version 26 Android provides a new api for accessing SharedMemory. It's looked up at runtime, so
+	// try it first rather than going by SYSPROP_SYSTEMVERSION, which executables without the app (unit
+	// tests, headless) don't know. Newer versions don't let us open /dev/ashmem ourselves.
+	fd = ashmem_create_region(name, size);
+	if (fd < 0) {
 		fd = legacy_ashmem_create_region(name, size);
 	}
 	// Note that it appears that ashmem is pinned by default, so no need to pin.

Tools/android_unittest.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+# Copyright (c) 2012- PPSSPP Project.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, version 2.0 or later versions.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License 2.0 for more details.
+
+# A copy of the GPL 2.0 should have been included with the program.
+# If not, see http://www.gnu.org/licenses/
+
+# Official git repository and contact information can be found at
+# https://github.com/hrydgard/ppsspp and http://www.ppsspp.org/.
+
+"""Runs the C++ unit tests on an Android device over adb.
+
+Builds ppsspp_unittest with the legacy NDK build (android/jni) if asked to, pushes it and the files
+some tests read to /data/local/tmp, and runs each test as its own process, so that one that crashes
+can't take the others' results with it. Every test's output is kept in --log-dir.
+
+    python Tools/android_unittest.py --build              # build for arm64-v8a, then run everything
+    python Tools/android_unittest.py Path VFS             # just these, with the binary already built
+    python Tools/android_unittest.py -s R5CNC19P5DP all   # pick a device
+
+The build needs the NDK: android/ab.cmd has its path on Windows, and android/ab.sh reads $NDK.
+Exits non-zero if any test failed.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+DEVICE_DIR = "/data/local/tmp/ppsspp_unittest_dir"
+# Files the tests open, relative to the working directory (see e.g. unittest/TestVFS.cpp).
+DATA_FILES = ["source_assets/ziptest.zip"]
+
+
+def adb(args, device, **kwargs):
+	cmd = ["adb"] + (["-s", device] if device else []) + args
+	return subprocess.run(cmd, **kwargs)
+
+
+def shell(command, device, **kwargs):
+	return adb(["shell", command], device, **kwargs)
+
+
+def build(abi):
+	android_dir = os.path.join(ROOT, "android")
+	if os.name == "nt":
+		cmd = ["cmd", "/c", os.path.join(android_dir, "ab.cmd"), "APP_ABI=" + abi, "UNITTEST=1"]
+	else:
+		cmd = ["./ab.sh", "APP_ABI=" + abi, "UNITTEST=1"]
+	print("Building: " + " ".join(cmd))
+	return subprocess.run(cmd, cwd=android_dir).returncode == 0
+
+
+def list_tests(device):
+	# Asking for a test that doesn't exist prints the list of the ones that do.
+	out = shell("cd %s && ./ppsspp_unittest no_such_test 2>&1" % DEVICE_DIR, device, capture_output=True, text=True).stdout
+	return [line.strip()[2:] for line in out.splitlines() if line.strip().startswith("* ")]
+
+
+def main():
+	parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+	parser.add_argument("tests", nargs="*", help="test names, or all (the default)")
+	parser.add_argument("--build", action="store_true", help="build ppsspp_unittest first")
+	parser.add_argument("--abi", default="arm64-v8a", help="the ABI to build and run (default arm64-v8a)")
+	parser.add_argument("-s", "--device", help="adb device serial, if there's more than one")
+	parser.add_argument("--log-dir", default=os.path.join(ROOT, "android", "unittest_logs"), help="where each test's output goes")
+	args = parser.parse_args()
+
+	if args.build and not build(args.abi):
+		print("Build failed")
+		return 1
+
+	binary = os.path.join(ROOT, "android", "libs", args.abi, "ppsspp_unittest")
+	if not os.path.exists(binary):
+		print("No %s, build it with --build" % binary)
+		return 1
+
+	# Push everything into a directory of our own, which is also the working directory of the tests.
+	shell("mkdir -p %s/source_assets" % DEVICE_DIR, args.device, check=True)
+	if adb(["push", binary, DEVICE_DIR + "/ppsspp_unittest"], args.device).returncode != 0:
+		return 1
+	for f in DATA_FILES:
+		adb(["push", os.path.join(ROOT, f), DEVICE_DIR + "/" + f], args.device, check=True, capture_output=True)
+	# Pushing over the old binary drops the executable bit.
+	shell("chmod 755 %s/ppsspp_unittest" % DEVICE_DIR, args.device, check=True)
+
+	tests = args.tests if args.tests and args.tests != ["all"] else list_tests(args.device)
+	if not tests:
+		print("Found no tests to run")
+		return 1
+
+	os.makedirs(args.log_dir, exist_ok=True)
+	failed = []
+	for test in tests:
+		result = shell("cd %s && ./ppsspp_unittest %s 2>&1" % (DEVICE_DIR, test), args.device, capture_output=True, text=True, errors="replace")
+		with open(os.path.join(args.log_dir, test + ".log"), "w", encoding="utf-8") as f:
+			f.write(result.stdout)
+		code = result.returncode
+		# adb passes the exit code through. 128 + signal for a crash, like 139 for SIGSEGV.
+		status = "ok" if code == 0 else ("CRASHED (%d)" % code if code > 128 else "FAILED (%d)" % code)
+		print("%-28s %s" % (test, status))
+		if code != 0:
+			failed.append(test)
+
+	print()
+	print("%d of %d passed. Logs in %s" % (len(tests) - len(failed), len(tests), args.log_dir))
+	if failed:
+		print("Failed: " + " ".join(failed))
+		print("To run just those again: python Tools/android_unittest.py " + " ".join(failed))
+	return 1 if failed else 0
+
+
+if __name__ == "__main__":
+	sys.exit(main())

android/jni/app-android.h

@@ -104,8 +104,10 @@ jobject Android_GetActivity(JNIEnv *env);
 // only room for 512 of them, and running out aborts the process.
 class JNILocalFrame {
 public:
+	// env is null in executables without a Java VM (headless, unit tests), where the storage functions
+	// using this then fail on the missing activity.
 	explicit JNILocalFrame(JNIEnv *env, int capacity = 16) : env_(env) {
-		pushed_ = env_->PushLocalFrame(capacity) == 0;
+		pushed_ = env_ && env_->PushLocalFrame(capacity) == 0;
 	}
 	~JNILocalFrame() {
 		if (pushed_) {

docs/android-device-testing.md

@@ -82,6 +82,25 @@ away within seconds, which reads as "logging stopped" rather than "logging was l
 
     adb logcat -c && adb logcat -s PPSSPP:V PpssppActivity:V   # in the background, then start the app
 
+## Running the C++ unit tests on a device
+
+CI builds the Android unit test executable but doesn't run it. To run it on a connected device:
+
+    python Tools/android_unittest.py --build        # build for arm64-v8a, push, run everything
+    python Tools/android_unittest.py Path VFS       # just these, with the binary already built
+
+It runs each test as its own process (a crash would otherwise swallow the buffered output of every test
+before it), keeps each test's output in `android/unittest_logs/`, and prints the command to re-run only the
+failures. Worth doing before a release, and after changing `Common/File` or anything with an
+`#if PPSSPP_PLATFORM(ANDROID)` branch, which desktop unit tests compile out.
+
+Two things to know when reading its results:
+
+- The executable has no Java VM, so content URIs fail like missing files there. They're only exercised
+  for real inside the app.
+- From Git Bash, set `MSYS_NO_PATHCONV=1` before any `adb push`/`adb shell` of your own, or
+  `/data/local/tmp` gets rewritten into a Windows path and the push silently lands somewhere else.
+
 ## VR (Quest)
 
 - The VR flavor is a separate package: `./gradlew :android:assembleVrDebug`, installing as