A drained MpscQueue node goes onto a free stack shared by every queue of the same type, and a
producer takes that stack whole when its own thread's list runs dry, so once warmed up a push
no longer allocates. That takes the heap allocation back off memory block notifications, the
CPU queue, Vulkan deletes and the GL render queue. A unit test runs several producers against
one consumer over two rounds.
The NpSignin dialog's delays run on emulated time and are saved with the state, so the dialog
behaves the same at any speed and under run-ahead. The time since the last save and savestate
goes; only the standalone frontends' exit prompts read it.
Readback buffers are allocated when the readback is recorded rather than on the render thread,
so only the thread that records frames touches VMA. The allocator is created externally
synchronized, and VMA is built with no-op mutexes and plain counters instead of std::mutex and
std::atomic.
Headless runs Vulkan offscreen in builds without SDL too, and CI runs the headless tests on
lavapipe, skipping the pixel tests a hardware backend doesn't match the software references on.
sleep_ms and sleep_us go, with the thread manager test waiting on an event instead, and so does
the unused portable-file-dialogs header.
The core and the headless tools no longer use mutexes, condition variables,
spinlocks, std::atomic, std::thread or volatile flags. Threads hand off through
libretro-common's retro_atomic and rthreads plus a few small pieces on top of
them in Common/Thread: Thread, MpscQueue, EventCounter, ParkingLot waits and
ReaderGate. Every wait blocks until it's woken; nothing polls on a timer.
State that only one thread touches just loses its lock. The PSP filesystem,
sceIo and the savedata dialog now belong to the emulation thread: async reads
and writes run when they're queued and complete later in emulated time, and
the IO and savedata threads are gone. The friend finder only talks to the
adhoc server and hands what it gets to the emulation thread. GPU init always
runs on the thread that starts the game.
The Vulkan render and compile queues, pipeline and render pass caches, delete
list and SPIR-V cache, the GL render queue, the software renderer's jit caches,
the file loader caches, the web debugger, UPnP, input, savestates, logging and
the GE debugger all hand off through queues and atomics. The IR jit's spinning
profiler thread goes.
A CPU thread with nothing to run now sleeps until work is queued instead of
spinning. Headless with --debugger waits for a client while stopped, and ends
the run once every client that connected has left. Web server shutdown keeps
servicing the CPU queue until the server has finished.
The libretro core is the frontend, and the headless tools are the test and
debugging hosts. The Windows, UWP, Android, iOS, macOS and SDL apps, the UI
framework and imgui go, along with what only they used: the Win32 debugger
hooks (g_frameMutex, the shutdown lock), JNI thread attachment, the Android
text drawer, RAIntegration, Discord, recent files, game installs, tilt input,
the audio resamplers, the control mapper, memstick moving and the remote ISO
and upload server modes. The web server keeps the debugger. The MSVC, Gradle
and ndk-build projects go too.
Socket waits now block in select() on the socket plus a wake or cancel
socket (Common/Net/Cancel), so a stop or cancel ends them at once.
- HTTP: cancel flags become net::CancelToken; Connect waits once on all
addresses, receives wait instead of spinning, no DNS retry sleep.
- Adhoc: the friend finder and adhoc server block on their sockets with
a wake socket and a deadline; matching runs on a CoreTiming tick
instead of two threads (savestate section bumped).
- sceNetInet select/poll check without blocking and park the HLE thread,
re-checked from a CoreTiming event until ready or the emulated
timeout.
- Vendor the aemu_postoffice client (was a submodule) with no sleeps:
waits on the socket, close via shutdown and an eventcount, cancelable
connects, and lock-free session slot claims.
- PortManager waits without a timeout; PSP_InitWait replaces the
headless boot poll.
- TimeUtil: sleep_us/sleep_ms only outside the core; sleep_precise,
sleep_random, yield and the Windows timer period are gone.
- Unit test for the socket waits.
The core no longer sleeps to pace itself: the frontend runs retro_run at
its own rate, so the frame throttle, the idle-vblank delay and the real
clock sync event (and its core option) are gone, and a stepping CPU hands
straight back instead of waiting 2 ms. Waits that polled now park on a
lock-free eventcount through Common/Thread/ParkingLot and are woken by
whoever changes the state: a paused GL emu thread, which also queues a
marker behind its last frame so pausing runs the render queue up to it
instead of polling; async I/O results; the software renderer when its
queue is full; Vulkan pipeline compiles; and debugger requests once
stepping ends. Boot joins the loader thread, cache shutdown joins its
read-ahead thread, and the slow-I/O simulation is removed.
The path browser, the HTTP and WebSocket servers, the Windows console and
the game info cache are left out of the core build. libretro/test gains
lrgltest, which drives the GL backend on a surfaceless EGL context through
every path that pauses or stops the emu thread; run.sh and the image
checks CI job run it on llvmpipe.
Deflate, zlib and gzip streams go through rinflate and rdeflate, zstd
through rzstd, CHD images through rchd, zip archives through
rzip_archive and PNG through rpng. The bundled libraries, their build
glue and the system-library options are gone; adler32 and uncompress
keep zlib's ABI for the prebuilt FFmpeg archives that still call them.
libretro-common is one CMake library that the core, the headless tool
and the unit tests all link.
The reads take shorter paths on the way. CHD decodes straight from the
loader's reads and a range of blocks at a time, and a file read at any
alignment decodes into place; CSO keeps one inflate stream instead of
one per block; a plain ISO serves a file read with one read of its exact
bytes; a stored zip member is read in place and a deflated one through
restart points rather than a copy of the whole image in memory; zip
members decode into the caller's buffer; sequential reads through the
frontend's VFS skip the seek, which also keeps its buffering.
libretro/test reads one image as ISO, CSO, CHD and zip through the block
devices, boots each, and checks the codecs against Python-made vectors;
a CI job runs it.
Returning to the CPU flushed all queued drawing, also when a list only reached its stall address.
Wipeout Pure builds its lists as it goes and stalls about a hundred times a frame, so the main thread
spent a third of its time finishing the drawing at those flushes. A stalled list isn't done, so now
only a finished one flushes there, which makes Wipeout Pure 15% faster.
What relied on that flush now flushes itself: memory copies, sets, readbacks and uploads that overlap
the queued drawing, and save states, before memory is saved or loaded (FlushPendingDrawing). Syncs and
the display already did. Queued drawing still reads textures from PSP memory when it runs, so a game
that rewrites a texture between stalls without syncing would see the new one; on hardware the GE has
usually drawn with the old one by then.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Texture pack installs took the destination directory from the archive's
textures.ini, and PKG update installs from the PBOOT's DISC_ID, both
unchecked, so a crafted archive or package could write outside
PSP/TEXTURES or PSP/GAME. Add IsSafePathComponent() to PathUtil and check
both, replacing the WebServer's own upload filename check with it too.
DISC_ID from any PARAM.SFO also names configs, cheats, textures, exdata
and savestates, so ReadSFO now drops one that isn't a safe name and the
game gets a fake ID like homebrew. The same goes for a frame dump's ID.
Combines #22419 and #22420 by Acts1631.
Co-Authored-By: Acts1631 <acts1631kjv@proton.me>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sceKernelStartModule, sceKernelStopModule and the self-stop calls now
check their option first: with SDK 2.08 or later its size must be 0x14,
and attribute bits outside 0x00F06000 are an error.
The option's attribute is added to the module's thread attribute rather
than ignored (or, when stopping, replacing it), and the default attribute
is 0 rather than the module's own attribute. A nonzero mpidstack puts the
stack in that partition, which has to be user-accessible for a user
module and kernel-only for a kernel module.
A failed thread creation is now returned instead of asserting.
Hardware test: modules/startoptions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
modulemgr only defaults a module_start/module_stop thread's stack to 0x40000
for user modules. Kernel modules get 0x1000. We gave every
module 0x40000, which mostly cost kernel memory when booting the VSH.
User modules, and so games, are unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Matching packet handlers added untrusted payload lengths to fixed header
sizes before comparing them with the datagram length. Large values could
overflow and permit out-of-bounds reads. Compare against the remaining
packet size instead, and reject negative ACCEPT sibling counts.
Lets a dump replay stopped with --replay-end be compared against the PSP on
an offscreen buffer. The framebuffer commands after the draw limit are
dropped too, so it shows the buffer the last drawn primitive went to.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
--depth-save writes the depth buffer next to the screenshot (u32 width, u32
height, then u16 depths), and --replay-end stops a frame dump replay after
that many primitives. Both match pspautotests' ppdmp-playback, so a dump's
depth can be compared with the PSP's at any point in the frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
hleDelayResult and Sas's delay put the wait on whatever thread was
current, which in an interrupt handler is the idle thread it runs on.
With both idle threads waiting there's nothing left to schedule, the
same crash as sceDisplaySetMode in Gods Eater Burst's vblank handler.
Where pspautotests intr/delays shows what hardware returns there,
match it: sceIoOpen and __sceSasCore fail with ILLEGAL_CONTEXT,
sceMpegCreate with 80618008, and sceAtracReinit can deinit but not set
up the codec (807f00fd).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
wsdbg --launch learns the port from our output, which without --log was
nothing. Also set the port before reporting the server as running, so
WebServerWaitForStartup() callers can read it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gods Eater Burst calls sceDisplaySetMode from its vblank handler. Its
implicit vblank wait went to the idle thread the handler runs on. Since
vblank waiters are released ~48us after the vblank (02b2b7b699), the next
frame's handler runs, on the other idle thread, before the first one is
released, so both idle threads end up waiting and the scheduler asserts
with an empty ready queue on leaving the interrupt.
On hardware sceDisplaySetMode returns 0 without waiting there, and with
interrupts or dispatch disabled (pspautotests intr/waits).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Achievement badges and game icons don't change, so re-downloading them
daily was wasted. InsertIcon now takes a max age, and each entry saves
its expiry time. Leaderboard avatars get a day, everything else 30 days.
Bumps the cache version.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With RAIntegration, Initialize leaves login to the DLL load, but WinMain
only calls InitializeRAIntegration once and Shutdown unloads the DLL, so
turning achievements off and on left you logged out until restart.
Remember the window and load the DLL again on its thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The deprecated unsized rc_client_deserialize_progress parses with no
bound, so a truncated or corrupt state could make it read past the
buffer. data_size is known, so pass it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rc_client answers a second login with RC_INVALID_STATE, synchronously,
and login_password_callback then cleared g_isLoggingIn while the token
login was still running. IsBlockingExecution stopped blocking, and a game
started then ran without achievements.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On invalid credentials the token was kept, and Idle() retried it every
10 seconds for the rest of the session, silently after the first attempt.
Clear it, as was already done for an expired token, so the settings show
the login form. On access denied, stop retrying.
Show a rejection even when it comes from a background retry, since it
won't repeat, and have LoginProblems return the server's message.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rc_client_get_rich_presence_message returns the length it needed rather
than what it wrote, so a message of 512 characters or more made the pause
screen read past its buffer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rc_client keeps raw pointers to itself in the callback data of award,
leaderboard submit and ping requests, so destroying it with one in flight
(disabling achievements mid-game) had the late callback write to freed
memory. Count requests per client, retire a busy client instead of
destroying it, and destroy it when its last request completes.
Request timeouts in naett are per operation and the plain HTTP client has
none for the body, so a retired client is also destroyed after 60 seconds
regardless. Its late callbacks are then dropped, leaking their data.
Clients are tracked by ID so a new client at a reused address can't get an
old one's callbacks. At app exit, pending requests are cancelled without
callbacks, so everything is destroyed immediately.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
naett reports a connection failure as -1, which rcheevos reads as
RC_API_SERVER_RESPONSE_CLIENT_ERROR, a non-retryable client error. An
unlock or leaderboard submission during a network dropout was then
treated as a server rejection and dropped, on every platform using naett.
Map any non-HTTP result to RC_API_SERVER_RESPONSE_RETRYABLE_CLIENT_ERROR.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rc_client doesn't call back for a game load aborted by unload, or for
anything pending when the client is destroyed. g_isIdentifying or
g_isLoggingIn then stayed set, and IsBlockingExecution kept every later
boot on a black screen until restart. Quitting during identification, or
disabling achievements during a token login, was enough.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
UWP compiled naett with the WinHTTP backend but never reported
SYSPROP_SUPPORTS_HTTPS, so achievement traffic and badge icons went over
plain http. When the media server redirected to https, icons failed.
A failed icon download also left the URL marked pending, so it was never
retried for the rest of the session. Mark it failed instead, and let
IconCache retry it after 30 seconds.
The set-completed badge also skipped RemoveHttpsIfNeeded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The release was counted down on the WebSocket thread, one step per poll
of host time however many vblanks had passed, so how long a scripted
press lasted depended on how fast the emulator ran, and scripted runs
went different ways. sceCtrl now releases it after that many vblank
samples, on the emulator thread; the debugger only reports when it's done.
Also: wsdbg's :screenshot works in headless with Vulkan.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On hardware the last part of a savedata shutdown runs at priority 0x20,
whatever the dialog's own thread priorities, so a caller at 0x20 gets the
CPU back first and sees SHUTDOWN, and one at 0x21 or worse only sees NONE
(pspautotests utility/savedata/shutdownstatus). We ended it at the access
thread's priority, so Freak Out, which calls ShutdownStart from 0x20 and
waits for SHUTDOWN, sat at 'Please press START' forever. NFL Street 3,
which calls it from 111 and then InitStart straight away, still gets NONE.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Taking the highest unused slot instead could steal one that a state event
restores later, as VBlankWake did to MicBlockingResume, which then had
nowhere to go. Also name the event in the assert.
AGENTS.md: When a savestate fails to load, suspect the branch first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Recorded on hardware (pspautotests threads/tls/timeout), a Tlspl
allocation follows the same timeout rule as the other waits, including
failing at once for 0 and 1us without writing the timeout back, which the
shared rule it moved to in the last commits didn't give it yet. Before
that it waited the raw timeout, ~30us short.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Priority-ordered waiting lists were sorted with a comparator wrapper per
object (msgpipe, fpl, vpl), or searched with a copy of the same function
(mutex, mbx). HLEKernel::SortWaitingThreadsByPriority() and
FindBestPriorityWaiter() now do both for any waiting list, of thread ids
or of structs with a threadID.
HLEKernel::ClearWaitingThreads() replaces the identical cancel/delete
loops in semaphores, event flags, fpl and vpl.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Semaphores, event flags, mutexes, lwmutexes, mbx, msgpipes, fpl, vpl,
tlspl and WaitThreadEnd each had their own CoreTiming event, handler
registration and savestate entry for wait timeouts, and their own function
to schedule one. Now one event (WaitThreadEnd's, renamed) times out all of
them, keyed by thread, and dispatches on the thread's wait type to a
timeoutFunc registered alongside the begin/end callback functions.
__KernelWaitCurThreadWithTimeout() starts such a wait, and the HLEKernel
helpers have overloads that use the shared event.
Old savestates still load: each object's section reads its old event id
and points it at the shared handler, so a timeout pending in the state
goes off as before. Checked with a state saved mid-wait by the previous
build, and with four games.
The one behaviour change: tlspl timeouts now follow the same hardware
rule as the others, where they used the raw timeout.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A state with more event types than are registered now was refused, so no
event could ever be removed or merged. Loading now keeps the extra slots:
modules that still know an old event restore it to a handler, and the rest
stay placeholders that do nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An interrupt with no handler to run, a vblank with none registered for
example, switched the running thread off to idle and left it there until
some later event rescheduled: ~775us of every frame in a game that spins
without a vblank handler. It now reschedules at once. Taking an interrupt
also clears the ll bit directly, which that switch had been doing.
Interrupt handlers can now carry a cost before they run and after the
last queued one returns. Alarms use it: on hardware a thread that keeps
running loses ~70us to an alarm handler, and a thread the handler wakes
runs ~50us after it (pspautotests threads/scheduling/alarmcosts), so
17us in and 40us out. sceKernelSetAlarm's 40us is split evenly around the
deadline, keeping the handler ~1040us after a 1000us alarm.
A handler's return value re-arms its alarm counting from the previous
deadline, so a repeating alarm doesn't drift by those costs, unless
that's already past, as after interrupts were suspended for a while.
The vblank's own cost (~62us of CPU on hardware) isn't charged yet: with
it, a waiter ~90us after the vblank still reads hcount 1 on hardware, but
line 2 here. Hardware evidently raises the interrupt ~40us before the
line count wraps. That's noted where the waiters are released.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On hardware a thread waiting for vblank returns ~53us after it, where we
had it back in ~5us, and the first of four waiters runs after ~85us: each
waiter beyond the first adds ~9us (pspautotests threads/scheduling/
vblankwake). The waiters are now released by a separate event 48us + 9us
per extra waiter after the vblank. Which vblank a wait is for is still
decided at the vblank, so a thread that starts waiting in between still
waits a whole frame (sceDisplay section version 8).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Timed on a PSP, each way of handing the CPU to another thread (the call
and the switch together):
hardware before now
rotate to an equal thread 7 14 7
signal, better thread runs 10 17 10
it waits again, back to caller 10 19 12
wakeup, better thread runs 8 13 6
it sleeps again, back to caller 7 12 6
start a better thread, entry 30 28 30
thread ends, back to its waiter 21 13 20
notify, better thread's callback 14 13 14
A switch between two threads now costs 1150 cycles instead of 2700.
Starting a better thread costs 2000 cycles more, ending a thread 3300,
and setting up a callback 1800.
Also splits a wait timeout's ~30us into the deadline being taken 12us
into the call and the timeout going off 18us after it. That only changes
the time left written back, which threads/semaphores/wait and
threads/fpl/cancel pin between them. intr/vblank is re-recorded so it
no longer depends on the phase of the frame.
threads/callbacks/combos now passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On hardware, notifying a callback of a thread in a CB wait takes it out of
the wait at once, even though the callback only runs when the thread would
get the CPU. A semaphore signalled in between doesn't end the wait: the
callback runs first, then the wait resumes and takes it (pspautotests
threads/callbacks/combos). We left the thread on the wait list until the
callback started, so the signal ended the wait and the callback didn't
run.
The notify now pauses the wait, as starting a callback used to. If the
callbacks are canceled before the thread's turn comes, the wait just
resumes (Thread savestate section version 7).
threads/callbacks/combos goes in the to-do list: a callback returning to
the thread that notified it still takes ~13us where hardware takes ~9,
part of the context switch cost.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every wait with a timeout behaves the same on hardware (pspautotests
threads/scheduling/waittimeouts). The deadline is taken, and the alarm set
up a moment later. If the deadline has passed by then, the wait fails with
WAIT_TIMEOUT at once, without yielding or writing the timeout back. That's
usual for 0us, half the time for 1us, and rare after; AllocateVpl does more
first. Otherwise it ends max(t, 205us) + ~35us after the call. Each object
had its own guess (24/245, 25/250, 20/250 and so on), and only MsgPipe had
the immediate case.
__KernelWaitTimesOutAtOnce() and __KernelWaitTimeoutUs() now do it for
semaphores, event flags, mutexes, lwmutexes, mbx, msgpipes, fpl, vpl and
WaitThreadEnd. The latency past the deadline isn't counted in the time
left written back.
Outcomes that hardware decides by the clock's phase (these, and
sceKernelDelayThread returning at once) go with the likelier one. Ones
between 50% and certain are instead spread evenly over calls, so a polling
loop can't lock into never yielding (sceKernelThread section version 7).
This replaces the pseudo-random choice for delays.
Also adds threads/scheduling/readyqueue, which already passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On hardware sceKernelSetAlarm takes about 40us, and the handler never runs
sooner than about 215us after the alarm is set, however short it asked
for (pspautotests threads/scheduling/alarmcosts). Also clamps huge
sysclock alarms before converting to cycles; LONG_LONG_MAX used to
overflow and go off at once, which the late-firing events had hidden.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each slice is sized to end at the next queued event, but scheduling a
sooner one didn't touch it, so the new event waited for the old slice to
run out. An alarm set by a thread that kept running went off 175-440us
late. GE enqueues worked around this with hleCoreTimingForceCheck(); now
every caller gets it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
That failure comes after the codec is set up and the frame has been tried
on the ME, so the thread waits for both, unlike the other SetData errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
libatrac3plus picks the codec parameter from the frame size and the
header's joint stereo flag, and the channel count plays no part. We
guessed joint stereo from the frame size and channel count instead, so
LocoRoco 2's MuiMui house music never started: the game writes a 2-channel
normal-stereo header for every track it streams, and that 0xC0 track holds
one mono sound unit per frame. Taken as joint stereo, its first frame failed
during setup, and the game retried forever.
Now the joint stereo flag comes from the track header, and the decoder's
channel count from the parameter it maps to, so that track decodes as mono
into both output channels, as on hardware. Low-level decoding, which has
no header, still goes by the frame size. Atrac2 saves the flag; older
states fall back to the guess.
Fixes#8647.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
