diff --git a/AGENTS.md b/AGENTS.md index 56992fe53c..fe33b221b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,7 +34,7 @@ for it: | [docs/kernel-hle-review.md](docs/kernel-hle-review.md) | Findings from a review pass over `Core/HLE/sceKernel*.cpp`, and what was verified clean | | [docs/shader-compilation.md](docs/shader-compilation.md) | Shader/pipeline compile stutter: where the time goes on MoltenVK (Metal's MSL compile, not glslang), the caches involved, how game-specific shaders are, and how to measure a cold start | | [docs/metal-backend.md](docs/metal-backend.md) | The Metal backend: shader translation, the render manager (recording, replay, pass merging), programmable blending, how it was tested against Vulkan, and what isn't done | -| [docs/async-savestates.md](docs/async-savestates.md) | Design for moving savestate compression and file I/O off the emu thread (not implemented yet): where the time goes, the save/load/rewind pipelines, ordering, memory rules | +| [docs/async-savestates.md](docs/async-savestates.md) | How savestate compression and file I/O run on a worker thread: the save/load/rewind pipelines, ordering, memory rules, what stays synchronous, and testing slots with the `game.state.*` debugger commands | ## General instructions diff --git a/Core/Debugger/WebSocket/GameSubscriber.cpp b/Core/Debugger/WebSocket/GameSubscriber.cpp index c925b724ec..5a84b1524b 100644 --- a/Core/Debugger/WebSocket/GameSubscriber.cpp +++ b/Core/Debugger/WebSocket/GameSubscriber.cpp @@ -95,6 +95,9 @@ private: // // Parameters: // - path: string, path to the .ppst file (only for load and save). +// - slot: number, a savestate slot of the running game, from 0, instead of path. Slots work like the +// save and load keys: a slot save keeps the slot's previous state as its undo, and a slot load +// keeps the state it replaces for load undo (if state undo is enabled). // - wait: optional boolean, respond when the operation has finished rather than when queued. // // Response (same event name) once queued, or an error if the network or achievements hardcore @@ -106,8 +109,16 @@ void WebSocketGameState::StateOperation(DebuggerRequest &req, StateOp op) { const char *event = events[(int)op]; std::string path; bool wait = false; - if ((op == StateOp::LOAD || op == StateOp::SAVE) && !req.ParamString("path", &path)) + uint32_t slot = 0; + const bool useSlot = (op == StateOp::LOAD || op == StateOp::SAVE) && req.HasParam("slot"); + if (useSlot) { + if (!req.ParamU32("slot", &slot)) + return; + if (slot >= (uint32_t)g_Config.iSaveStateSlotCount) + return req.Fail("Invalid slot"); + } else if ((op == StateOp::LOAD || op == StateOp::SAVE) && !req.ParamString("path", &path)) { return; + } if (!req.ParamBool("wait", &wait, DebuggerParamType::OPTIONAL)) return; @@ -139,10 +150,21 @@ void WebSocketGameState::StateOperation(DebuggerRequest &req, StateOp op) { } switch (op) { case StateOp::LOAD: - queued = SaveState::Load(Path(path), -1, callback); + if (useSlot) { + // Refusals go to the callback. + SaveState::LoadSlot(SaveState::GetGamePrefix(g_paramSFO), (int)slot, callback); + queued = true; + } else { + queued = SaveState::Load(Path(path), -1, callback); + } break; case StateOp::SAVE: - queued = SaveState::Save(Path(path), -1, callback); + if (useSlot) { + SaveState::SaveSlot(SaveState::GetGamePrefix(g_paramSFO), (int)slot, callback); + queued = true; + } else { + queued = SaveState::Save(Path(path), -1, callback); + } break; case StateOp::REWIND: canRewind = SaveState::CanRewind(); diff --git a/docs/WebSocketDebugger.md b/docs/WebSocketDebugger.md index 0754ede886..9d38871071 100644 --- a/docs/WebSocketDebugger.md +++ b/docs/WebSocketDebugger.md @@ -271,7 +271,7 @@ file - this is just an index. | Category | Events | File | |---|---|---| -| Game/version | `game.reset`, `game.status`, `game.state.load` (queue loading a `.ppst` file, applied at the end of the next frame the CPU runs; with `wait: true` the response comes once it's done, or as an error if it failed), `game.state.save` (the same for saving one), `game.state.rewind` (the same for the newest rewind snapshot, which it keeps, so only a rewind within 0.8 seconds of the last steps further back; needs `RewindSnapshotInterval` set, and the `wait` response's `metadata` says how far back that is in emulated time), `game.state.verify` (the same for checking that saving the current state and reading it back agree), `game.speed.get/set` (emulation speed - unlimited fast-forward, or a percentage of 60 FPS; see below), `version` | `GameSubscriber.cpp` | +| Game/version | `game.reset`, `game.status`, `game.state.load` (queue loading a `.ppst` file given as `path`, or a `slot` of the running game, applied at the end of the next frame the CPU runs; with `wait: true` the response comes once it's done, or as an error if it failed), `game.state.save` (the same for saving one), `game.state.rewind` (the same for the newest rewind snapshot, which it keeps, so only a rewind within 0.8 seconds of the last steps further back; needs `RewindSnapshotInterval` set, and the `wait` response's `metadata` says how far back that is in emulated time), `game.state.verify` (the same for checking that saving the current state and reading it back agree), `game.speed.get/set` (emulation speed - unlimited fast-forward, or a percentage of 60 FPS; see below), `version` | `GameSubscriber.cpp` | | CPU core | `cpu.stepping`, `cpu.resume`, `cpu.status` (reports `ticks` plus `us`, emulated microseconds, and `clockHz` - use `us` to line up with wall-clock timings, since games change the clock frequency and the ticks-per-second ratio isn't fixed), `cpu.getAllRegs`, `cpu.getReg`, `cpu.setReg`, `cpu.evaluate` | `CPUCoreSubscriber.cpp` | | Stepping | `cpu.stepInto`, `cpu.stepOver`, `cpu.stepOut`, `cpu.runUntil`, `cpu.runUntilTime` (run until a point in emulated time - `us` absolute or `relativeUs` from now - and break there; this is how to get a scripted repro reproducibly "N seconds into the game" instead of polling `cpu.status` in a loop), `cpu.nextHLE` | `SteppingSubscriber.cpp` | | Breakpoints | `cpu.breakpoint.add/update/remove/list`, `memory.breakpoint.add/update/remove/list`, `cpu.regBreakpoint.add/update/remove/list` (break when a register is written to, by any instruction anywhere - currently GPRs only; interpreter-only, no effect under a JIT backend) | `BreakpointSubscriber.cpp` | diff --git a/docs/async-savestates.md b/docs/async-savestates.md index 2a3d2740de..a4f616ba2f 100644 --- a/docs/async-savestates.md +++ b/docs/async-savestates.md @@ -1,8 +1,9 @@ # Async savestates -The design for taking savestate file work off the emulation thread, so saving and loading don't -stall the game long enough to glitch the audio. Not implemented yet. Written 2026-10-19; file -references will drift. +How savestate compression and file I/O were taken off the emulation thread, so saving and loading +don't stall the game long enough to glitch the audio. The worker and the operation queue are in +`Core/SaveState.cpp`, rewind in `Core/SaveStateRewind.cpp`, and the file format code in +`Common/Serialize/Serializer.cpp`. Written 2026-10-19; file references will drift. **Keep memory use low.** A state is about 38 MB uncompressed, and much of the hardware we run on is short of memory. So don't waste it: no full-size buffers beyond what an operation needs, freed when @@ -40,7 +41,7 @@ doesn't (worker). 1. Emu thread, in `Process()`: `SaveStart::DoState` into a fresh buffer (the snapshot). The waits and `FlushPendingDrawing()` stay here, and so does the `saveStateGeneration` bump. -2. Worker: compress with streaming zstd straight into `.tmp` (see [Memory](#memory)), then for a slot save the rotation that `SaveSlot`'s +2. Worker: compress (see [Memory](#memory)) and write the file, to `.tmp` for a slot save, then for a slot save the rotation that `SaveSlot`'s `renameCallback` does today (old file to `.undo`, `.tmp` to the real name), including the `g_Config.Save()` for the undo bookkeeping. Then set `g_filesStale`. 3. Completion: the callback, with "Saved State" or the failure. **The message appears when the file @@ -85,44 +86,47 @@ Operations still take effect in the order they were asked for: - **Worker jobs run in queue order**, so a save to a slot followed by a load of it writes before it reads. As a shortcut, a load whose path has a write pending takes that save's uncompressed snapshot directly and skips the file round trip. -- **UI-thread file operations on slots wait for pending worker jobs on the same slot**: `UndoSaveSlot`, - `UndoLoad`, `DeleteSlot`, and `SaveSlot`'s screenshot rotation. `LoadSlot`'s existence check - counts a slot with a write pending as existing. +- **UI-thread file operations on slots wait for the worker** (`WaitForWorker()`, for all jobs, since + it's rare and short): `UndoSaveSlot` and `DeleteSlot`. `UndoLoad` is a load, so it's queued behind + the load undo write like anything else. `LoadSlot`'s existence check counts a slot with a save + pending as existing. The thumbnail screenshot isn't the worker's, so its rotation doesn't wait. ## What stays synchronous -There's one code path. Synchronous means enqueue and wait for completion. On the emu thread that -wait runs the worker phase inline instead of blocking on the worker, so it can't deadlock against a -queue that needs the emu thread. +There's one code path. Waiting for it is `WaitForWorker()`, which simply blocks: worker jobs never +need the emu thread, so it can't deadlock. - **libretro**: `retro_serialize`/`retro_unserialize` are in memory with no file I/O, and stay as - they are. + they are. So does the freeze-frame feature (`SaveToRam`/`LoadFromRam` every frame). - **Verify** stays synchronous. -- **Shutdown and game switch** (`SaveState::Shutdown()`): flush pending writes (a save that was asked - for is never lost), cancel pending reads and call their callbacks with a failure. -- **App exit**, including Android activity destruction, calls `SaveState::FlushPendingWrites()` with - a timeout. +- **Shutdown and game switch** (`SaveState::Shutdown()`, from the kernel's shutdown, which app exit + goes through too): wait for the worker, so a save that was asked for is never lost. Then run the + completions' bookkeeping (the undo slot in the config) without their callbacks, which can refer to + an EmuScreen that's going away, and drop whatever is still queued, as before. - Loads at boot (auto-load, `--state`, opening a `.ppst`) and the debugger's `game.state.*` with - `wait` already work through callbacks, and need no special handling. The game already runs at - least a frame before a boot-time load applies today. + `wait` already work through callbacks, and need no special handling. The game already ran at least + a frame before a boot-time load applied. ## Rewind -Rewind snapshots already compress on a thread. What's still on the emu thread: +Rewind snapshots used to compress on a `std::thread` per snapshot, which the next snapshot joined +(stalling if it was still running), and the whole restore ran on the emu thread: wait for the +compressor, decode (rebuild the full ~38 MB state from the base and the changed blocks), apply. The +snapshot itself (2-5 ms, mostly copying RAM and VRAM, already a `ParallelMemcpy`) stays on the emu +thread. -- The snapshot (2-5 ms, mostly copying RAM and VRAM, already a `ParallelMemcpy`). -- `compressThread_.join()` at the start of each snapshot, which stalls if the previous compress is - still running. That gets more likely once the deltas are compressed with zstd. -- The whole restore: wait for the compressor, decode (rebuild the full ~38 MB state from the base and - the changed blocks, essentially a 38 MB copy), apply. +Now: -Changes: - -- Compression moves to the savestate worker, replacing the `std::thread` per snapshot. If the - previous snapshot's job is still running, skip the snapshot rather than wait: snapshots are best - effort. -- **Restore becomes an async load from memory**: the worker decodes, the emu thread applies. No - prefetching (see [Memory](#memory)). +- The emu thread owns the ring itself (which slots hold what); compressing and decoding are jobs on + the savestate worker, and the emu thread leaves the buffers a job uses alone until it's done. +- **Compression runs on the worker.** If the previous snapshot's job is still running, the snapshot + is skipped and retried next frame rather than waited for: snapshots are best effort. +- **The deltas get zstd on top** (level 1: higher levels shrink them a little for a lot more time). + The changed blocks still compress about 3x, so the ring takes about a third of the memory. +- **Restore is an async load from memory**: `StartRestore()` picks the state and has the worker + decode it (queued behind the newest snapshot's compression, if that's still going), and + `FinishRestore()` applies it on the emu thread. No snapshots are taken in between. No prefetching + (see [Memory](#memory)). ## Memory @@ -130,10 +134,11 @@ What each operation holds at its peak, against today: | | Today | Async | |---|---|---| -| Save | snapshot (38 MB) + `ZSTD_compressBound` output buffer (about 38 MB) | snapshot (38 MB) + streaming output buffer (a few hundred KB) | -| Load | whole compressed file (about 12 MB) + state (38 MB) | state (38 MB) + streaming input buffer | +| Save | snapshot (38 MB) + `ZSTD_compressBound` output buffer (about 38 MB) | snapshot (38 MB) + the compressed output as it grows (10-12 MB). The snapshot is freed before writing | +| Load | whole compressed file (about 12 MB) + state (38 MB) | state (38 MB) + a small streaming input buffer | | Load from a slot | the save, then the load | at apply: the loaded state + the load undo snapshot (76 MB), briefly. The loaded state is freed right after the apply, before the undo is compressed | | Rewind restore | a `static` 38 MB decode buffer, never freed | the same buffer, freed after the apply | +| Rewind ring | block diffs | the same, zstd-compressed: about a third | Rules: @@ -143,10 +148,11 @@ Rules: a memory hog in the settings). - **At most two save snapshots at a time** (see [Save](#save)), and one load buffer at a time: a second load queued behind a first doesn't start reading until the first is applied. -- **Stream compression in both directions.** `ZSTD_compressStream2` writing to the file in chunks, - and `ZSTD_decompressStream` reading from it, so a compressed copy of the state never sits in - memory in full. The file format doesn't change: the header's sizes are known before writing - (`UncompressedSize`) or patched in afterwards (`ExpectedSize`, by seeking back). +- **Stream compression in both directions.** Saving, `ZSTD_compressStream2` into a buffer that grows + with its output, instead of allocating the worst case up front. It isn't streamed straight to the + file, since the header holds the compressed size and seeking back to patch it in isn't safe on + every kind of storage (Android's content URIs). Loading, `ZSTD_decompressStream` reads the file in + chunks, so the compressed data never sits in memory in full. The file format doesn't change. - **Rewind adds nothing.** The worker reuses rewind's existing buffers. There's no prefetching (decoding a likely target before the rewind is pressed), which would need a spare full-size buffer. @@ -157,14 +163,23 @@ Rules: the write pass. - The apply is mostly `DoState` logic rather than copying; leave it unless it shows up. -## Implementation order +## Results -1. The worker, job and completion queues, ordering, flush and cancel on shutdown, with every job - still waited for inline (no behavior change). -2. Async saves: plain and slot saves, the undo rotation, duplicate detection. -3. Async loads, with the load undo snapshot moved to apply time. -4. Rewind compression on the worker, then async restore. -5. zstd on rewind deltas (the changed blocks only), which slots into step 4's worker job. +Measured the same way as above (Release headless, desktop x64), on the emu thread: -Each step can be tested with headless, wsdbg and the `game.state.*` debugger commands. A debugger -query for pending savestate operations would let tests wait for the worker to drain. +| | Before | After | +|---|---|---| +| Save | 49-75 ms | 2.4-2.5 ms (the snapshot) | +| Load | 28-40 ms | 12-15 ms (the apply, plus the warnings and callback around it) | +| Rewind restore | decode + apply | 12-13 ms (the apply) | +| A frame waiting on the worker | | under 0.05 ms | + +Rewind deltas went from 4-13 MB to 1.3-3.8 MB each, at 10-24 ms of worker time. + +## Testing + +Headless, wsdbg and the `game.state.*` debugger commands cover all of it. `game.state.save` and +`game.state.load` take a `slot` instead of a `path`, which goes through the slot code like the save +and load keys: undo rotation, load undo, repeats. Use a scratch `--memstick` for that, not the app's, +or the tests overwrite real slots. wsdbg's `--sync` moves on when a request is acknowledged, not when +a `wait` result arrives, so put a `:sleep` between steps whose order matters, and one at the end.