Docs: Write down where shader compile time goes on MoltenVK

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -32,6 +32,7 @@ for it:
 | [docs/cpu-equivalence.md](docs/cpu-equivalence.md) | Making the CPU cores agree bit for bit on every host: the `CPUEquivalence` fuzzer, what it found, and the differences still open (several waiting on a hardware test) |
 | [docs/dnas-patches.md](docs/dnas-patches.md) | Game patches installed on a real PSP (Killzone: Liberation's DLC): installing one over PSPLink, and the Fuse-ID-keyed PGD and type 3 PRX that `sceKernelLoadModuleDNAS` decrypts |
 | [docs/kernel-hle-review.md](docs/kernel-hle-review.md) | Findings from a review pass over `Core/HLE/sceKernel*.cpp`, and what was verified clean |
+| [docs/shader-compilation.md](docs/shader-compilation.md) | Shader/pipeline compile stutter: where the time goes on MoltenVK (Metal's MSL compile, not glslang), the caches involved, how game-specific shaders are, and how to measure a cold start |
 | [docs/metal-backend.md](docs/metal-backend.md) | What a native Metal backend would take, and why programmable blending is the reason to want one |
 
 ## General instructions

docs/metal-backend.md

@@ -55,6 +55,8 @@ HLSL. Far less work, and the right thing to start with - but be clear that it is
 MoltenVK's shader half, so glslang and SPIRV-Cross stay as runtime dependencies and only the Vulkan
 API emulation is shed. Metal PSO creation from MSL source is also slower than Vulkan pipeline creation
 from SPIR-V, so the existing asynchronous pipeline compilation machinery matters more, not less.
+Keeping glslang and SPIRV-Cross costs little: under MoltenVK they take about 1.5 ms per shader against
+roughly 100 ms for Metal's own MSL compile ([shader-compilation.md](shader-compilation.md)).
 
 Option A is an optimisation that can be done later, or never.
 

docs/shader-compilation.md

@@ -0,0 +1,119 @@
+# Shader compilation stutter
+
+Where the time goes when a game compiles its shaders and pipelines, mainly on the Mac (Vulkan through
+MoltenVK), and how to measure it. Measured in October 2026 on an M1 with MoltenVK 1.4.2.
+
+## The path a shader takes
+
+1. The shader generators (`GPU/Common/VertexShaderGenerator.cpp`, `FragmentShaderGenerator.cpp`)
+   write GLSL from a `VShaderID` / `FShaderID`.
+2. glslang compiles it to SPIR-V (`GLSLtoSPV()` in `Common/GPU/Vulkan/VulkanContext.cpp`), on a
+   dedicated thread per shader (`CompileShaderModuleAsync()` in `GPU/Vulkan/ShaderManagerVulkan.cpp`).
+3. `vkCreateGraphicsPipelines()` runs on the thread pool, which `VulkanRenderManager::CompileThreadFunc()`
+   feeds, one task per vertex/fragment shader pair. On MoltenVK this is where the rest happens:
+   SPIRV-Cross turns SPIR-V into MSL, Metal compiles the MSL into an `MTLLibrary`, and then builds the
+   `MTLRenderPipelineState`. MoltenVK's `vkCreateShaderModule()` only copies the SPIR-V.
+4. The render thread waits for the pipeline in `VulkanQueueRunner` (`BlockUntilReady()`) when it
+   reaches the first draw that uses it. That wait is the stutter.
+
+## Where the time goes
+
+Wipeout Pure from a savestate, starting with no PPSSPP cache and an empty Metal cache. MoltenVK's own
+counters (see below):
+
+| Stage | Time each | Count |
+|---|---|---|
+| glslang, GLSL to SPIR-V | ~1 ms | 27 |
+| SPIRV-Cross, SPIR-V to MSL | 0.5 ms | 40 |
+| Metal, MSL to `MTLLibrary` | 85-110 ms (max ~470) | 43 |
+| Metal, `MTLFunction`s to a pipeline state | 5 ms | 63 |
+
+Nearly all the cost is Metal compiling MSL, and it happens about once per distinct shader, not once
+per pipeline. Pipelines that differ only in blend, cull, depth or stencil state cost about 5 ms each
+once their shaders' libraries exist. On the Mac, then:
+
+- Speeding up the shader generators or glslang gains nothing worth having.
+- Cutting the pipeline count (more dynamic state, e.g. cull mode and depth/stencil through
+  `VK_EXT_extended_dynamic_state`) helps little. In this run it would have taken 58 pipelines down to
+  43 that differ in anything Metal bakes in, and the expensive step doesn't repeat for those anyway.
+- Fewer distinct shaders (moving ID bits into uniforms) is the lever that matters, with the runtime
+  cost that ubershaders bring, mostly in fragment shaders.
+
+### The shared `VkPipelineCache`
+
+We used to create every pipeline with the same `VkPipelineCache`, and MoltenVK serializes the MSL
+compiles of pipelines that share one: its "retrieve shader library from the cache" time, which includes
+waiting for another thread's compile, averaged 142 ms. The 3.7 s of compiling blocked the render thread
+for 2.9 s in total, so there was almost no parallelism. The cache is only held in memory (we don't save
+its data), and on MoltenVK all it saves is the SPIR-V to MSL step, so on Apple platforms we now pass
+`VK_NULL_HANDLE` (`CreateVulkanPipeline()` in `GPU/Vulkan/PipelineManagerVulkan.cpp`).
+
+| Cold Wipeout Pure, two runs each | With the cache | Without |
+|---|---|---|
+| Render thread blocked on pipelines, total | 2.9 s | 1.2 s |
+| Longest single stall | 1.0-1.1 s | 0.3 s |
+| Average wait inside MoltenVK's cache lookup | 142 ms | 11 ms |
+
+`MVK_CONFIG_SHOULD_MAXIMIZE_CONCURRENT_COMPILATION=1` made no measurable difference.
+
+## The caches
+
+- **PPSSPP's per-game cache**, `PSP/SYSTEM/CACHE/<DISCID>.vkshadercache`. It holds the shader IDs and
+  pipeline keys the game used, plus their SPIR-V, and `GPU_Vulkan::LoadCache()` recompiles all of it
+  while the game boots. With it, Wipeout's 62 pipelines took 140 ms in total.
+- **macOS's Metal cache**, keyed by the MSL source. With it warm, even a run *without* PPSSPP's cache
+  took about 100 ms for all 62 pipelines. So on the Mac, a second boot is cheap with or without our
+  cache, and a shader one game compiled is cheap in the next game too. Only the first time a shader
+  is ever seen (and possibly after an OS or app update) is it slow.
+
+### How game-specific are the shaders?
+
+Less than expected. Across the 37 `.vkshadercache` files of one Mac's memory stick (at the current
+`CACHE_VERSION`), for the seven biggest:
+
+| Game | VS | Also in another game | FS | Also in another game |
+|---|---|---|---|---|
+| UCES00002 (Ridge Racer) | 12 | 92% | 47 | 51% |
+| UCES01250 | 13 | 85% | 36 | 67% |
+| UCES00001 (Wipeout Pure) | 14 | 93% | 25 | 80% |
+| UCES00465 | 12 | 100% | 22 | 86% |
+| UCES00842 (God of War: Chains of Olympus) | 11 | 82% | 21 | 76% |
+| ULES00502 (GTA: Vice City Stories) | 10 | 90% | 20 | 70% |
+| ULES01376 (Tekken 6) | 11 | 73% | 20 | 65% |
+
+All 37 games together use only 48 distinct vertex shader IDs. The IDs depend on the device's GPU
+features and the user's settings, and some of those caches come from short runs, so a list shipped with
+the app would match less well than one gathered on the device. A per-device list of shaders seen in
+several games, precompiled at boot, looks worth trying.
+
+## Ideas not done yet
+
+- Don't block the render thread on a pipeline that isn't ready: skip the draw, or keep the previous
+  pipeline, for the few frames until it is. Visible (objects pop in), so probably an option, but it
+  would remove the remaining stalls on every backend.
+- Precompile cross-game shaders at boot, from the list above.
+- A native Metal backend could use `MTLBinaryArchive`. Not measured. See
+  [metal-backend.md](metal-backend.md).
+
+## Measuring
+
+`--shader-cache=false` turns off loading and saving PPSSPP's per-game cache for one run (headless or
+the app; it isn't saved to the ini). On the Mac that isn't a cold start, since Metal's cache still
+hits. Headless is an unbundled binary and shares Metal's cache with other command-line tools, in
+`$(getconf DARWIN_USER_CACHE_DIR)com.apple.metal/`. For a real cold start, move the GPU's folders
+there aside (on the M1, `16777235_322` and `32024`, the two big ones) and put them back afterwards.
+`MTL_SHADER_CACHE_DISABLE` and similar environment variables don't turn the cache off.
+
+```bash
+build-release/PPSSPPHeadless "$HOME/PSP/Wipeout Pure.cso" --memstick=$HOME/.config/ppsspp \
+    --state=$HOME/.config/ppsspp/PSP/PPSSPP_STATE/UCES00001_1.01_0.ppst \
+    --graphics=vulkan --timeout-wall=30 --timeout-emulated=20 --log --shader-cache=false
+```
+
+- Each pipeline's creation time is logged at INFO level, as `Pipeline (x/N) time on ...`.
+- MoltenVK's breakdown: set `MVK_CONFIG_PERFORMANCE_TRACKING=1` and
+  `MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE=2`. The summary prints when the device is destroyed,
+  at the end of the run.
+- The render thread's stalls aren't logged. Time `BlockUntilReady()` in `VulkanQueueRunner.cpp`
+  temporarily to see them; the sum of pipeline times overstates the stall, since compiles overlap.
+- Run each configuration at least twice. Single cold runs varied by about 15%.