The core and the headless tools no longer use mutexes, condition variables,
spinlocks, std::atomic, std::thread or volatile flags. Threads hand off through
libretro-common's retro_atomic and rthreads plus a few small pieces on top of
them in Common/Thread: Thread, MpscQueue, EventCounter, ParkingLot waits and
ReaderGate. Every wait blocks until it's woken; nothing polls on a timer.
State that only one thread touches just loses its lock. The PSP filesystem,
sceIo and the savedata dialog now belong to the emulation thread: async reads
and writes run when they're queued and complete later in emulated time, and
the IO and savedata threads are gone. The friend finder only talks to the
adhoc server and hands what it gets to the emulation thread. GPU init always
runs on the thread that starts the game.
The Vulkan render and compile queues, pipeline and render pass caches, delete
list and SPIR-V cache, the GL render queue, the software renderer's jit caches,
the file loader caches, the web debugger, UPnP, input, savestates, logging and
the GE debugger all hand off through queues and atomics. The IR jit's spinning
profiler thread goes.
A CPU thread with nothing to run now sleeps until work is queued instead of
spinning. Headless with --debugger waits for a client while stopped, and ends
the run once every client that connected has left. Web server shutdown keeps
servicing the CPU queue until the server has finished.
The libretro core is the frontend, and the headless tools are the test and
debugging hosts. The Windows, UWP, Android, iOS, macOS and SDL apps, the UI
framework and imgui go, along with what only they used: the Win32 debugger
hooks (g_frameMutex, the shutdown lock), JNI thread attachment, the Android
text drawer, RAIntegration, Discord, recent files, game installs, tilt input,
the audio resamplers, the control mapper, memstick moving and the remote ISO
and upload server modes. The web server keeps the debugger. The MSVC, Gradle
and ndk-build projects go too.
Socket waits now block in select() on the socket plus a wake or cancel
socket (Common/Net/Cancel), so a stop or cancel ends them at once.
- HTTP: cancel flags become net::CancelToken; Connect waits once on all
addresses, receives wait instead of spinning, no DNS retry sleep.
- Adhoc: the friend finder and adhoc server block on their sockets with
a wake socket and a deadline; matching runs on a CoreTiming tick
instead of two threads (savestate section bumped).
- sceNetInet select/poll check without blocking and park the HLE thread,
re-checked from a CoreTiming event until ready or the emulated
timeout.
- Vendor the aemu_postoffice client (was a submodule) with no sleeps:
waits on the socket, close via shutdown and an eventcount, cancelable
connects, and lock-free session slot claims.
- PortManager waits without a timeout; PSP_InitWait replaces the
headless boot poll.
- TimeUtil: sleep_us/sleep_ms only outside the core; sleep_precise,
sleep_random, yield and the Windows timer period are gone.
- Unit test for the socket waits.
The core no longer sleeps to pace itself: the frontend runs retro_run at
its own rate, so the frame throttle, the idle-vblank delay and the real
clock sync event (and its core option) are gone, and a stepping CPU hands
straight back instead of waiting 2 ms. Waits that polled now park on a
lock-free eventcount through Common/Thread/ParkingLot and are woken by
whoever changes the state: a paused GL emu thread, which also queues a
marker behind its last frame so pausing runs the render queue up to it
instead of polling; async I/O results; the software renderer when its
queue is full; Vulkan pipeline compiles; and debugger requests once
stepping ends. Boot joins the loader thread, cache shutdown joins its
read-ahead thread, and the slow-I/O simulation is removed.
The path browser, the HTTP and WebSocket servers, the Windows console and
the game info cache are left out of the core build. libretro/test gains
lrgltest, which drives the GL backend on a surfaceless EGL context through
every path that pauses or stops the emu thread; run.sh and the image
checks CI job run it on llvmpipe.
Deflate, zlib and gzip streams go through rinflate and rdeflate, zstd
through rzstd, CHD images through rchd, zip archives through
rzip_archive and PNG through rpng. The bundled libraries, their build
glue and the system-library options are gone; adler32 and uncompress
keep zlib's ABI for the prebuilt FFmpeg archives that still call them.
libretro-common is one CMake library that the core, the headless tool
and the unit tests all link.
The reads take shorter paths on the way. CHD decodes straight from the
loader's reads and a range of blocks at a time, and a file read at any
alignment decodes into place; CSO keeps one inflate stream instead of
one per block; a plain ISO serves a file read with one read of its exact
bytes; a stored zip member is read in place and a deflated one through
restart points rather than a copy of the whole image in memory; zip
members decode into the caller's buffer; sequential reads through the
frontend's VFS skip the seek, which also keeps its buffering.
libretro/test reads one image as ISO, CSO, CHD and zip through the block
devices, boots each, and checks the codecs against Python-made vectors;
a CI job runs it.
since 7e6405291 only full 64 KB reads get cached, so any read touching the
shorter last block of a file came back short. count blocks with
blocks_.size() too, so failed reads can't inflate the count until
MakeCacheSpaceFor loops forever. also stop read-ahead from asking the
backend for blocks past the end of the file.
Identify_File and friends compare against lowercase extensions, the way
Path::GetFileExtension returns them, but ZipFileLoader passed the name
from the zip through unchanged. So a file stored as e.g. "DUMP.PPDMP"
inside a zip failed to load, while "dump.ppdmp" worked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JvJR8oJNSCimCM9KXVLjfq
blocksActuallyRead rounded bytesRead up to the next whole block
unconditionally, intending to handle the legitimate case where the
very last block of the file is naturally shorter than BLOCK_SIZE
(cache_ is deliberately over-allocated for that). But it applied the
same rounding to any short read, including a genuine failure or a
dropped connection mid-file (this loader can sit on top of the whole
Remote ISO chain via CachingFileLoader/HTTPFileLoader when "Cache full
ISO in RAM" is enabled) - marking a block as fully cached when only a
few of its bytes were actually written. Since cache_ is malloc'd (not
zeroed), every later read of that block would serve uninitialized heap
memory as if it were real file data.
Only round up when the short read's end position exactly matches the
true end of the file.
SaveIntoCache checked `readBytes != 0` instead of comparing against the
full expected length, so any nonzero-but-short read from the backend
(e.g. a Remote ISO connection dropping mid-file) was treated as a
complete success: in the multi-block path this marked *all* requested
blocks (up to 16) as fully cached and wrote the uninitialized tail of
the read buffer to the on-disk cache file, and in both paths the
short/uninitialized data was also copied straight into the caller's
output buffer and counted in the return value - so a read failure was
reported (and permanently cached) as success. Only treat a block as
read once the backend actually delivered the full blockSize_ for it,
and stop before caching or returning anything for blocks it didn't.
Also fixes two latent bugs in the same functions, unreachable in the
current call graph (DiskCachingFileLoader is only ever driven by
CachingFileLoader, which always issues block-aligned reads) but wrong
if ever called otherwise:
- The multi-block loop reused the batch's initial `offset` (the
position within the *first* block) for every subsequent block
instead of resetting it to 0, which would both read from the wrong
place in `wholeRead` and mis-copy less than a full block for i > 0.
- ReadBlockData() applied `offset` to the destination pointer instead
of the file seek position, which would both read the wrong bytes
from disk and write up to `offset` bytes past the end of the
caller's buffer.
LoadCacheIndex's sanity check on persisted block indices used `>`
instead of `>=` against maxBlocks_ (blockIndexLookup_ only has
maxBlocks_ entries, valid indices 0..maxBlocks_-1), so a corrupted
cache file's index entry with block == maxBlocks_ exactly would pass
validation and then index one past the end of blockIndexLookup_.
SaveIntoCache() discarded backend_->ReadAt()'s return value entirely
and unconditionally marked the requested block(s) as cached. A short
or failed read from the backend (e.g. a Remote ISO connection dropping
mid-file, now that LocalFileLoader/RetryingFileLoader correctly report
failures as 0 rather than a huge count) would still get stored as a
"valid" cached block, permanently serving its uninitialized tail as if
it were real file data on every later read, with no retry.
Only insert a block once we've confirmed the backend actually
delivered the full BLOCK_SIZE for it.
ReadAt()'s contract is to return the number of bytes/units actually
read. On every platform branch, an OS-level read failure (ReadFile
returning FALSE, or pread/read returning -1) was fed straight into a
division by `bytes` without checking for it first:
- Windows explicitly returned (size_t)-1.
- Elsewhere, the signed -1 from pread/read was implicitly converted to
size_t (via the usual arithmetic conversions with the unsigned
`bytes`) before the division, producing a huge bogus count instead
of a small one.
Every caller in the caching chain (CachingFileLoader,
RamCachingFileLoader, RetryingFileLoader, ZipFileLoader's libzip
source callback) loops on "did we get at least what we asked for",
which a huge return value trivially satisfies - so a local I/O error
(removable media ejected, a content-URI permission problem mid-read,
etc.) would be reported as a fully successful read of whatever
uninitialized memory happened to be in the destination buffer.
A malicious or MITM'd server could send a Content-Range header matching
the requested range but a larger entity body, overflowing the caller's
fixed-size buffer via output.Take. Clamp the copied size to the requested
range.
Turns out these were needed after all. For some reason, on Windows and
Mac, <algorithm> gets auto-included by something else so I don't notice
when it's missing, and MSVC's include dependency tracker doesn't see it
either.
sleep_ms() should generally be avoided when possible. This can be used to try
to track down unnecessary sleeps by adding some logging.
This commit on its own doesn't actually add any logging.
* Rename LogType to Log
* Explicitly use the Log:: enum when logging. Allows for autocomplete when editing.
* Mac/ARM64 buildfix
* Do the same with the hle result log macros
* Rename the log names to mixed case while at it.
* iOS buildfix
* Qt buildfix attempt, ARM32 buildfix
libretroadmin