Config.h is edited about 100 times a year and each edit recompiled 198
files; now 172. SmallDataConvert.h pulls in arm_neon.h and was included by
headers that don't use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crosshair turns red, and lock-on works, when stencil bit 3 is set at
the centre of the frame, which the game tests by reading the pixel from
VRAM with the CPU (#17936). Hook that and read back the stencil of that
row into the alpha bytes in RAM. The alpha the hardware backends keep
doesn't follow stencil everywhere, so reading color isn't enough.
Adds PerformReadbackStencilToMemory for this (8888 only for now).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Buzz! Master Quiz converts its question videos (sceMpegBaseCscAvcRange)
straight into the middle of the frame it has drawn, at 84,69. The
framebuffer upload only matched writes starting at a framebuffer's
address, so the hardware backends never showed the video. Pass the
written width along and draw just that rectangle into the framebuffer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On native Windows (Ryzen 9 9950X3D), lingering only paid in the transform
helpers. With no drawing-thread linger and 100 us for the helpers, wall
time stayed within 1% of 200/300 us in Daxter, GTA VCS, Wipeout Pure,
Tekken 6, Burnout Dominator, LocoRoco and Ridge Racer, with 9-27% less
CPU time. No linger at all cost up to 7% (Daxter), almost all of it from
the transform helpers.
The linger times are now per platform; Linux keeps the WSL2 values.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Where the mip level comes from q (slope mode, or auto with UV planes) and q is
the same over the whole triangle, its pixels all have one level. The staged
spans went through ApplyTexturing for each four pixels anyway, to work out that
level and call the quad sampler; a variant of DrawStagedSpans now works it out
once and calls the quad sampler directly. It's a separate variant, as the extra
code in the shared one cost other games about 1% of instructions on an M1.
LocoRoco takes 7.5% fewer instructions and 5% less wall time, Wipeout 2% fewer
instructions; the others are unchanged. The frame dump set renders the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DrawSprite is only fast in its single pixel paths; for any other pixel state
(fog, depth test, other blending) it sampled and drew a pixel at a time, while
DrawRectangle draws four in vector lanes. Such sprites now go to DrawRectangle.
On an M1: Tekken 6 and LocoRoco take 4-5% fewer instructions, Burnout 1%. The
frame dump set renders the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The light color products went through Vec4<int>'s multiply, a 32-bit
lane multiply that SSE2 emulates, though their factors are at most 10
bits. MulSmall uses CrossSIMD's Mul16 where Vec4<int> has a register.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GEDot3 and GERowSum called ldexpf for each result, and the UV plane
setup ldexp and ilogb for every triangle and tile, about 2% of the CPU
time between them. Adding to the exponent field gives the same values
while they stay normal (ScaleByPow2Exact, which calls ldexpf otherwise),
and the planes' values and shifts always are. 2-4% fewer instructions
in every game measured.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pixel and sampler ID hashes and equality, the functions describing
the IDs for the JIT's code pointers, and SamplerID's linear and fetch
bits, which only told the JIT cache's entries apart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The four-pixel bilinear sampler read its sixteen texels one at a time,
each with its CLUT lookup and index transform, and loaded them as
vectors afterwards. For CLUT4 and CLUT8 with a 32-bit palette and no
index shift (by far the most common), and for 8888, the texels are now
gathered in lanes: the index bytes, the nibble, the mask and start, then
the palette entries. CrossSIMD gets Vec4S32::Gather and GatherU8 for it,
one load per lane. LocoRoco executes 5.5% fewer instructions, Burnout 3%.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Measured on an M1: lingering cost wall time at every length tried, as
waking a thread is cheap on macOS, and seven or eight drawing threads
beat six. So Apple platforms use eight and don't linger, and Android
doesn't linger either (spinning costs battery). Windows and Linux keep
six and lingering, which was measured under WSL2 only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
StartRun waited until all of a run's vertices were transformed before
the first primitive was assembled and binned. Now each chunk records the
run it was done for, and the draws take each vertex as soon as its chunk
is done (RunNeed), doing other chunks themselves while it isn't, so the
emulation thread bins while the helpers transform. Whatever reuses the
decode buffer or the transform state finishes the run first
(FinishRun), and so does its last draw.
God of War takes 3.5 s for five emulated seconds (from 3.6 s), Wipeout
3.2 s (from 3.3 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Vec4<int> and Vec4<float>'s AssignToAll went through the four value
constructor, which stores the values one by one, so whatever loaded the
vector next stalled: the edge functions' setup per triangle and tile,
among others. They now splat in a register on SSE and NEON.
ReadSpanDepth loads its four depths as one vector where they're stored
together, rather than through an array, and ComputePlane is inline, as
returning its plane through memory stalled the UV planes' setup the same
way.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ForTargetTiles divided by the framebuffer stride in 64 bits twice per
texture row, and by the bytes per pixel and tile sizes, which are powers
of two. The offsets fit 32 bits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The edge functions' start built a constant vector through Vec4<int>'s
four value constructor, which stores the values one by one, and the UV
ramps went through a stack array: each reloaded as a vector, a store
forwarding stall per triangle and tile. The constant is now loaded from
a static array, and the ramps {0, d, 2d, 3d} are made from d in lanes.
Wipeout takes 3.4 s for five emulated seconds (from 3.5 s), God of War
3.6 s (from 3.7 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A run is faster than its draws one by one even when it's too short for
the helpers to join (16 vertices beat 32, 64 and 128 in God of War),
and a single large draw benefits from the helpers too (Wipeout). Chunks
are now 32 vertices. God of War takes 3.7 s for five emulated seconds
(from 3.9 s), Wipeout 3.5 s (from 3.54 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tile's head and busy flag, which the drawing threads write, shared a
cache line with its tail, which the emulation thread writes for every
item it hands the tile. And a drawing thread waiting for work read
every tile's head and tail again and again. Now the tail has a line of
its own, and a waiting thread watches one counter that DistributeItems
bumps, looking at the tiles only when it changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ApplyGESkinning computes skinned positions and normals the way the GE
does and overwrites what the decoder skinned in float, so that was
wasted, 4% of the emulation thread in God of War. A new decoder option,
callerSkins, leaves out the weights, position and normal steps for
weighted vertices outside through mode and only reserves the float
position and normal. The software renderer's transform unit gets its
own decoders with it; bounding boxes, splines and the debugger still
use the ones that skin. Checked to give the same output with the vertex
decoder JIT as without the option.
God of War takes 4.0 s for five emulated seconds (from 4.1 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BinQueue's head, tail and size were atomics, so each push was two
locked increments, but only the emulation thread uses them: the drawing
threads read items, which the tiles publish. And each BinItem was built
on the stack in pieces and then copied with wider loads, a store
forwarding stall. Now the indices are plain and items are built in
place (PushItem). AddTriangle was 8% of the emulation thread's time in
God of War, which now takes 4.1 s for five emulated seconds (from 4.2 s),
Wipeout 3.5 s (from 3.7 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There are 24 of them. SoftDirty(-1), which some paths set, now means
just those (plus unused bits up to 31) rather than 64 bits of which
40 never got cleared.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
God of War draws thousands of short strips per frame, each continuing
the previous one's vertices with nothing changed in between. At the
first draw of such a run, SoftGPU now looks ahead in the display list
for the draws that follow it, past only commands that can't change
anything (NOPs, unchanged state, VADDR to where the vertices continue),
and the transform unit decodes the whole run at once and then skins and
transforms it in chunks that two helper threads take alongside the
emulation thread. The draws then read their vertices from that, and the
UV and normal they carry to the next draw are updated as reading their
last vertex would.
The helpers wait a moment for the next run before their tasks end, as
the drawing threads do: waking them costs the emulation thread a system
call. The thresholds were picked under WSL2, where that costs more than
on bare metal, so they're worth measuring again there.
God of War takes 4.2 s for five emulated seconds (from 4.5 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SSE2 has no 32-bit lane multiply, so Vec4S32::Mul is emulated with two
pmuludq and shuffles, and the bilinear sampler spent a tenth of its time
there. The texel row offsets, the texture functions, fog and blending
only multiply values that fit 16 bits, which Mul16 does in one pmaddwd.
The bilinear lerp of two channels packed as 16-bit halves uses the new
MulHalves (pmullw), which multiplies both at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ApplyGESkinning took a vertex at a time with its components in three of
four lanes. Now four vertices go through each bone together, a lane
each, with the bone matrix entries, the weights and the positions taken
apart once for the GE's multiply (GEMulOperand). A bone whose weight is
zero in some lanes adds zeros there, leaving the sums as skipping it
would. The per-component order of the adds is the GE's as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A draw's vertices now go through the position stage of ReadVertex (the
clip coordinates, the viewport, the screen rounding and the GE's fog)
four at a time, a vertex per lane, before the rest. GERowSumLanes is
GERowSum4 with a lane per vertex instead of per matrix row, sharing the
positions' parts between the five rows, and GERecip4 is GERecip for four
w's. A group with a lane those don't handle (non-finite coordinates, a
w that's zero, denormal or too large) goes through ReadVertex as before.
Wipeout takes 3.8 s for five emulated seconds (from 4.2 s), God of War
4.7 s (from 5.0 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A drawing task ended as soon as it found no work, so the emulation
thread woke threads again and again, each a system call it paid for
(about 15 us under WSL2). A thread now waits up to 200 us for more,
unless the emulation thread is waiting for the drawing to finish.
More drawing threads also slow the emulation thread, which is what
limits these games, more than they help it: in God of War and Wipeout
on a 16 core Ryzen, six beat four, eight and the 31 the thread manager
has. With both, God of War takes 5.0 s for five emulated seconds (from
5.2 s with four threads, 7.4 s with all), Wipeout 4.2 s (from 4.7 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GERowSum4 split every matrix entry into exponent, fraction and sign for
each vertex. GERowSumRows does that once when the transform state is
computed, for the clip matrix, the world matrix's normal rows, the fog
row and the texture matrix.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ApplyTexturing's per-pixel path built each pixel's color from four
strided channel values, which the compiler stored in pieces and then
loaded as a vector, a store forwarding stall per pixel. On x86 and
ARM64, where the sampler takes its color in a register, the channel
rows are transposed in registers instead, and back afterwards.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GEUVSpan stored the four 64-bit plane values, and the reciprocal table's
entries, to arrays one by one and loaded them as vectors, a store
forwarding stall each time, which took most of its time on x86.
CrossSIMD gets LoadS64Low (the low halves of four 64-bit values) and
LoadPairs (four pairs of 32-bit values split into two vectors) to do it
without memory. Wipeout runs about 2% faster.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With 2048, God of War filled the queue often enough that the emulation
thread spent a fifth of its drawing time waiting for room. With 8192 it
stays ahead of the drawing threads: about 6% less wall time there (with
eight drawing threads), and the same in Wipeout. Larger made no
difference.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ReadVertex took the GE's viewport arithmetic one coordinate at a time,
with a reciprocal of w for each, and stored the position, the UVs and
the whole vertex in pieces that were then reloaded with wider loads,
each a store forwarding stall. x, y and z now go through one GERecip and
the four-lane multiply and add (the scalar path stays for non-finite
input and zero or denormal w), the position is loaded as a vector, and
the vertex is written straight into its destination.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The C++ paths draw and sample four pixels at a time in vector lanes,
while a JIT function only does one, and with a JIT function in use the
span and quad paths were skipped. On x86 the JITs had become slower:
Wipeout took 13% more wall time and 25% more CPU with them, God of War
3% more. They were also a second implementation of every GE rule, and
ARM never had them.
The SoftwareRendererJit setting goes with them. RasterizerRegCache.h
keeps only the types the pixel and sampler functions take, as
RasterizerTypes.h.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A sprite's u gradient comes from the setup reciprocal of its whole area, so it's
exact only for a power of two area, whatever the height: the Δy only cancels on
paper. Checked against exp87's sizes and exp256, where 64x128 CLUT8 sprites with
u = 1 + 2x read the next pixel's low byte on a PSP and 64x272 ones their own
high byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The GE doesn't divide by a q <= 0: it samples the last texel in u and v,
whatever s and t are, in clamp and repeat modes alike (gpu/texmtx/negq). The
hardware backends divided, which mirrors a projected texture behind its
projector: Fate/Extra's character shadows ran up the school's walls (#18667).
The software renderer sampled (0, 0) instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the span drawing and sampling faster, the setup a triangle repeats
for each tile it touches weighs more, so the rule is now eight tiles per
thread instead of sixteen (64x32 on eight threads). Against sixteen, in
wall time on five games: God of War -2.6%, Wipeout -1.7%, LocoRoco -5.2%,
Tekken 6 -5.6%, Burnout -3.3%, with 2.5-8% less CPU. Four per thread used
the least CPU but was slower than eight in wall time in most of them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Staged spans without mip levels pass no level arrays, and the sampler goes
straight to its one level path. 4% fewer instructions in God of War.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The texels come out corner by corner, so no transpose, and R with B, G with
A, share 32-bit lanes as 16-bit halves: no lerp of 8-bit values needs more
than 16 bits, and masking after each shift truncates both halves as before.
2.5% fewer instructions in God of War.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Without mip levels and with one filter for both, a span's texturing is the
quad sampler at level 0: staged spans call it directly instead of going
through ApplyTexturing, which also skips the level calculation. God of War
retires 7% fewer instructions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SampleLinearQuadT filtered each pixel's four texels as 16-bit channels in a 64-bit integer, then
transposed the results into a vector per channel for the texture function. Now the lerps run on a
channel of all four pixels at a time, producing that layout directly. God of War about 6% faster.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fallback (which RISC-V uses) did its integer arithmetic in plain int,
where signed overflow and left shifts of negative values are undefined,
and converted out of range floats with a cast, which is undefined too:
RISC-V gives INT_MAX for NaN where SSE gives INT_MIN and NEON 0. With GCC
on RISC-V that broke GEUVSpanCore in the GEMath unit test. Now the
arithmetic wraps through uint32_t, like the SIMD instructions, and an out
of range or NaN conversion gives INT_MIN, like SSE.
Also skip GERowSum's zero terms before computing their shift, as their
exponent can be INT_MIN. Found with -fsanitize=undefined on the fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coverage and the drawn range took the same three edge bounds, each a
64-bit division. 1% fewer cycles in God of War.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The span functions come in three blend kinds (none, source alpha over the
destination, and the general one), and states with logic ops or signed blend
factors get the pixel at a time function up front instead of checking per
span. 4.5% fewer instructions in God of War.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CheckDepthTestPassed4 only has an SSE path, so on ARM it compared lane by
lane. Adds Vec4S32::LoadU16 for the depth values.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A flat secondary color of zero (no specular) isn't added to each span, and
a flat secondary color or absent fog fills its buffer once per triangle
rather than once per row. 3% fewer instructions in God of War.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
DrawRectangle computed each span's UVs from the planes in 64 bits through the general GEUVSpan. It
now fills the staged rows' buffers (flat colors, z and fog, the coverage and the early depth test)
and draws them with DrawStagedSpans, which takes the 32-bit UV path when the planes allow it. Each
pixel still picks its mip level from its own q.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a plane's values at both ends of the chunk fit in 32 bits, so do all between, and 32-bit
wrapping arithmetic gets them exactly: then in vector lanes, with the clamp.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The coverage stage was a scalar loop per pixel with the snap edge as a branch. A chunk starts a span,
so relative to it the bounds fit in 32 bits and the snap edge's pixel is a fixed offset per lane.
Also read the depth buffer for the early depth test with one load where SSE isn't used.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
