diff --git a/AGENTS.md b/AGENTS.md index 4e5928c6a6..2165cc6c32 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,6 +32,7 @@ for it: | [docs/cpu-equivalence.md](docs/cpu-equivalence.md) | Making the CPU cores agree bit for bit on every host: the `CPUEquivalence` fuzzer, what it found, and the differences still open (several waiting on a hardware test) | | [docs/dnas-patches.md](docs/dnas-patches.md) | Game patches installed on a real PSP (Killzone: Liberation's DLC): installing one over PSPLink, and the Fuse-ID-keyed PGD and type 3 PRX that `sceKernelLoadModuleDNAS` decrypts | | [docs/kernel-hle-review.md](docs/kernel-hle-review.md) | Findings from a review pass over `Core/HLE/sceKernel*.cpp`, and what was verified clean | +| [docs/shader-compilation.md](docs/shader-compilation.md) | Shader/pipeline compile stutter: where the time goes on MoltenVK (Metal's MSL compile, not glslang), the caches involved, how game-specific shaders are, and how to measure a cold start | | [docs/metal-backend.md](docs/metal-backend.md) | What a native Metal backend would take, and why programmable blending is the reason to want one | ## General instructions diff --git a/docs/metal-backend.md b/docs/metal-backend.md index a7f2944bab..7e56afe70c 100644 --- a/docs/metal-backend.md +++ b/docs/metal-backend.md @@ -55,6 +55,8 @@ HLSL. Far less work, and the right thing to start with - but be clear that it is MoltenVK's shader half, so glslang and SPIRV-Cross stay as runtime dependencies and only the Vulkan API emulation is shed. Metal PSO creation from MSL source is also slower than Vulkan pipeline creation from SPIR-V, so the existing asynchronous pipeline compilation machinery matters more, not less. +Keeping glslang and SPIRV-Cross costs little: under MoltenVK they take about 1.5 ms per shader against +roughly 100 ms for Metal's own MSL compile ([shader-compilation.md](shader-compilation.md)). Option A is an optimisation that can be done later, or never. diff --git a/docs/shader-compilation.md b/docs/shader-compilation.md new file mode 100644 index 0000000000..643a0c63d9 --- /dev/null +++ b/docs/shader-compilation.md @@ -0,0 +1,119 @@ +# Shader compilation stutter + +Where the time goes when a game compiles its shaders and pipelines, mainly on the Mac (Vulkan through +MoltenVK), and how to measure it. Measured in October 2026 on an M1 with MoltenVK 1.4.2. + +## The path a shader takes + +1. The shader generators (`GPU/Common/VertexShaderGenerator.cpp`, `FragmentShaderGenerator.cpp`) + write GLSL from a `VShaderID` / `FShaderID`. +2. glslang compiles it to SPIR-V (`GLSLtoSPV()` in `Common/GPU/Vulkan/VulkanContext.cpp`), on a + dedicated thread per shader (`CompileShaderModuleAsync()` in `GPU/Vulkan/ShaderManagerVulkan.cpp`). +3. `vkCreateGraphicsPipelines()` runs on the thread pool, which `VulkanRenderManager::CompileThreadFunc()` + feeds, one task per vertex/fragment shader pair. On MoltenVK this is where the rest happens: + SPIRV-Cross turns SPIR-V into MSL, Metal compiles the MSL into an `MTLLibrary`, and then builds the + `MTLRenderPipelineState`. MoltenVK's `vkCreateShaderModule()` only copies the SPIR-V. +4. The render thread waits for the pipeline in `VulkanQueueRunner` (`BlockUntilReady()`) when it + reaches the first draw that uses it. That wait is the stutter. + +## Where the time goes + +Wipeout Pure from a savestate, starting with no PPSSPP cache and an empty Metal cache. MoltenVK's own +counters (see below): + +| Stage | Time each | Count | +|---|---|---| +| glslang, GLSL to SPIR-V | ~1 ms | 27 | +| SPIRV-Cross, SPIR-V to MSL | 0.5 ms | 40 | +| Metal, MSL to `MTLLibrary` | 85-110 ms (max ~470) | 43 | +| Metal, `MTLFunction`s to a pipeline state | 5 ms | 63 | + +Nearly all the cost is Metal compiling MSL, and it happens about once per distinct shader, not once +per pipeline. Pipelines that differ only in blend, cull, depth or stencil state cost about 5 ms each +once their shaders' libraries exist. On the Mac, then: + +- Speeding up the shader generators or glslang gains nothing worth having. +- Cutting the pipeline count (more dynamic state, e.g. cull mode and depth/stencil through + `VK_EXT_extended_dynamic_state`) helps little. In this run it would have taken 58 pipelines down to + 43 that differ in anything Metal bakes in, and the expensive step doesn't repeat for those anyway. +- Fewer distinct shaders (moving ID bits into uniforms) is the lever that matters, with the runtime + cost that ubershaders bring, mostly in fragment shaders. + +### The shared `VkPipelineCache` + +We used to create every pipeline with the same `VkPipelineCache`, and MoltenVK serializes the MSL +compiles of pipelines that share one: its "retrieve shader library from the cache" time, which includes +waiting for another thread's compile, averaged 142 ms. The 3.7 s of compiling blocked the render thread +for 2.9 s in total, so there was almost no parallelism. The cache is only held in memory (we don't save +its data), and on MoltenVK all it saves is the SPIR-V to MSL step, so on Apple platforms we now pass +`VK_NULL_HANDLE` (`CreateVulkanPipeline()` in `GPU/Vulkan/PipelineManagerVulkan.cpp`). + +| Cold Wipeout Pure, two runs each | With the cache | Without | +|---|---|---| +| Render thread blocked on pipelines, total | 2.9 s | 1.2 s | +| Longest single stall | 1.0-1.1 s | 0.3 s | +| Average wait inside MoltenVK's cache lookup | 142 ms | 11 ms | + +`MVK_CONFIG_SHOULD_MAXIMIZE_CONCURRENT_COMPILATION=1` made no measurable difference. + +## The caches + +- **PPSSPP's per-game cache**, `PSP/SYSTEM/CACHE/.vkshadercache`. It holds the shader IDs and + pipeline keys the game used, plus their SPIR-V, and `GPU_Vulkan::LoadCache()` recompiles all of it + while the game boots. With it, Wipeout's 62 pipelines took 140 ms in total. +- **macOS's Metal cache**, keyed by the MSL source. With it warm, even a run *without* PPSSPP's cache + took about 100 ms for all 62 pipelines. So on the Mac, a second boot is cheap with or without our + cache, and a shader one game compiled is cheap in the next game too. Only the first time a shader + is ever seen (and possibly after an OS or app update) is it slow. + +### How game-specific are the shaders? + +Less than expected. Across the 37 `.vkshadercache` files of one Mac's memory stick (at the current +`CACHE_VERSION`), for the seven biggest: + +| Game | VS | Also in another game | FS | Also in another game | +|---|---|---|---|---| +| UCES00002 (Ridge Racer) | 12 | 92% | 47 | 51% | +| UCES01250 | 13 | 85% | 36 | 67% | +| UCES00001 (Wipeout Pure) | 14 | 93% | 25 | 80% | +| UCES00465 | 12 | 100% | 22 | 86% | +| UCES00842 (God of War: Chains of Olympus) | 11 | 82% | 21 | 76% | +| ULES00502 (GTA: Vice City Stories) | 10 | 90% | 20 | 70% | +| ULES01376 (Tekken 6) | 11 | 73% | 20 | 65% | + +All 37 games together use only 48 distinct vertex shader IDs. The IDs depend on the device's GPU +features and the user's settings, and some of those caches come from short runs, so a list shipped with +the app would match less well than one gathered on the device. A per-device list of shaders seen in +several games, precompiled at boot, looks worth trying. + +## Ideas not done yet + +- Don't block the render thread on a pipeline that isn't ready: skip the draw, or keep the previous + pipeline, for the few frames until it is. Visible (objects pop in), so probably an option, but it + would remove the remaining stalls on every backend. +- Precompile cross-game shaders at boot, from the list above. +- A native Metal backend could use `MTLBinaryArchive`. Not measured. See + [metal-backend.md](metal-backend.md). + +## Measuring + +`--shader-cache=false` turns off loading and saving PPSSPP's per-game cache for one run (headless or +the app; it isn't saved to the ini). On the Mac that isn't a cold start, since Metal's cache still +hits. Headless is an unbundled binary and shares Metal's cache with other command-line tools, in +`$(getconf DARWIN_USER_CACHE_DIR)com.apple.metal/`. For a real cold start, move the GPU's folders +there aside (on the M1, `16777235_322` and `32024`, the two big ones) and put them back afterwards. +`MTL_SHADER_CACHE_DISABLE` and similar environment variables don't turn the cache off. + +```bash +build-release/PPSSPPHeadless "$HOME/PSP/Wipeout Pure.cso" --memstick=$HOME/.config/ppsspp \ + --state=$HOME/.config/ppsspp/PSP/PPSSPP_STATE/UCES00001_1.01_0.ppst \ + --graphics=vulkan --timeout-wall=30 --timeout-emulated=20 --log --shader-cache=false +``` + +- Each pipeline's creation time is logged at INFO level, as `Pipeline (x/N) time on ...`. +- MoltenVK's breakdown: set `MVK_CONFIG_PERFORMANCE_TRACKING=1` and + `MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE=2`. The summary prints when the device is destroyed, + at the end of the run. +- The render thread's stalls aren't logged. Time `BlockUntilReady()` in `VulkanQueueRunner.cpp` + temporarily to see them; the sum of pipeline times overstates the stall, since compiles overlap. +- Run each configuration at least twice. Single cold runs varied by about 15%.