Trace MaskedBitCount32 inputs back to their ballot source and apply wave64 lowering only when that ballot is in uniform control flow. Keep divergent ballot/MBCNT pairs on native wave32 semantics.
* common: Optimize logger by replacing hash map with table lookup and hide implementation details in cpp file
Also bring back assertion noinline functions, this makes call sites of ASSERT/UNREACHABLE much cleaner
* clang format
* review comments
* fix tests
* fix build pt1
* fix build pt2
* fix build pt3
* Add Shutdown method to Socket interface
* Add Shutdown method to PosixSocket class
* Add Shutdown method to UnixSocket class
* Add Shutdown method to P2PSocket class
* Add error handling and logging to sys_shutdown
* Add sys_shutdown function to kernel library
* buffer_cache: Begin memory tracker rework
* buffer_cache: Implement upload batching
* buffer_cache: Dont use copysparsememory for stream copies
Will need a faster way to check if page is mapped, but also using it has heavy perf overhead
* buffer_cache: Adjust region manager so bitset updates are vectorized
Also remove readable/writeable bitsets since they arent actually needed, should allow more data to fit in cpu caches
* memory: Fix build
* buffer_cache: Bring back copy sparse memory
* staging_pool: Allow requesting unsychronized buffers
Prevents large memory spikes from tiler
* page_manager: Add null check
* video_core: Cleanup implementation
* fix build
* buffer_cache: Remove dead function
* liverpool: Fix nullgpu
* host_shaders: embed shaders as spv
* always clang-format and reuse...
* fix commented out #pragma once in generated code
* use glslang to build shaders
* add cstdint to generated headers
* Log errors returned from UFFDIO_* operations
* Maintain userfaultfd registration on all reserved memory at all times
* Set userfaultfd handler thread name
* Do not lock GPU resources while handling writeprotection faults originating from GpuCommandProcessor thread
* Style fixes
* Use gettid to retrieve the GPUComm thread id
* shader_recompiler: implement fragment sample coverage input
* shader_recompiler: move sample coverage handling to IR
Keep the SPIR-V backend limited to raw built-in loads and construct the
guest sample coverage value in the fragment prologue.
Suggested-by: @raphaelthegreat
* shader_recompiler: Emit GetPcLo as constant
Fixes the assertion triggered by this instruction
* shader_recompiler: Convert mbcnt into pseudo-instruction for better pattern matching and improve reliability of detection
* phi_simplification_pass: Copy users before iterating
Running local CSE can mutate instructions in the list
* vk_instance: Enable maintenance5 if available
* video_core: respect SPI_BARYC_CNTL front-face encoding
- Read FRONT_FACE_ALL_BITS from SPI_BARYC_CNTL
- Emit +1.0/-1.0 when sign encoding is selected
- Emit integer 1/0 when all-bits encoding is selected
- Carry the register state through FragmentRuntimeInfo
- Add a boolean IR attribute path for FrontFacing
- Include the mode in shader specialization and cache metadata
- Add regression tests covering both AMD-defined encodings
Fixes according Roamic, Raphael, and IndecisiveTurtle recommendations.
* shader_recompiler: Remove unused IsFrontFace attribute and enhance front face handling in translation, rebase
* shader_recompiler: Fix front-face test compilation
- Use the current hw.fs runtime-info layout for front-face encoding.
- Update the regression helper to the HwStage and SwStage APIs.
- Address test compilation feedback from @raphaelthegreat
---------
Co-authored-by: w1naenator <valdis.bogdans@hotmail.com>
Co-authored-by: TheTurtle <47210458+raphaelthegreat@users.noreply.github.com>
The BufferType::SharedMemory path in Rasterizer::BindBuffers backs a compute
shader's LDS with a region of the stream buffer: it calls StreamBuffer::Map,
zeroes the bytes and binds them as the storage buffer the recompiler lowered
the LDS accesses to. It never called Commit.
Map only chooses the region and waits until it is free; Commit is what
reserves it. Commit flushes the host writes on non-coherent memory, advances
the ring cursor past the mapped bytes and records the completion watch that
makes a later wrap wait for the GPU work that used them. Without it the next
Map on the same stream buffer hands out the same bytes again, so a streamed
upload can overwrite the shader's shared memory while the dispatch that uses
it is in flight, and the shader's LDS stores can land in bytes a later upload
reads as its source.
Every other StreamBuffer::Map in the tree is paired with a Commit; this was
the only call site without one.
Co-authored-by: nikmparis217-ux <295578344+nikmparis217-ux@users.noreply.github.com>
* buffer_cache: Refactor buffer manager with sparse arenas
The buffer cache is responsble for mirroring the GPU accessed address space in device local memory. On main it uses a simple scheme where the address space is divided into 16KB pages.
Buffers are allocated with that granularity and all pages of that buffer point to the same buffer object. When a range overlaps multiple buffers, a join overlap operation is performed
where a new buffer is created and the old buffer contents are copied over to the new one.
This scheme is quite simple and works well but has some issues
* It requires a 2-pass approach to buffer binding, because different bindings in the same draw/dispatch can cause a join overlap between them.
If one binding was a storage buffer being written to, then data would be lost otherwise. This also causes various other issues when trying to batch memory uploads
* Join overlap can cause large VRAM usage spikes and requires heuristics to avoid repeated recreations.
If the cache holds a large 1GB+ buffer, and then a range that partially overlaps it is accessed, the full 1GB range + the non overlapped size is going to get allocated.
The old buffer will eventually be freed but this is still a large VRAM spike that can OOM or force system memory usage on lower VRAM gpus
To resolve these issues a new buffer management scheme is proposed in this pull request, that uses sparse residency buffers.
The address space is divided into 2^32 VA arenas, and each arena holds a certain amount of blocks/tiles which are determined by the sparse bind granularity of the host GPU.
The arena size has been chosen because some lower end apple sillicon GPUs only support creating buffers up to 4GB, even though the GPU hardware support much large address spaces.
Currently the code assumes that arena merges are always successfull, but that case will fail on those GPUs, for that bridge VA buffers need to be created and cached.
However accesses that straddle arenas are expected to be rare enough for this to not be an actual issue in practice.
Any buffer access is checked and overlapping blocks are made resident on demand by allocating memory. Residency requests are batched and submitted before scheduler submits the main command buffer.
If an access overlaps two adjacent arenas, then the arenas are merged into a single larger one and all their backings are brought up.
This new design avoids the old issues. There is no join operation aside from arena joining, which doesnt allocate any memory and is mostly for bookeeping purposes.
The buffer objects are also stable now so the 2-pass approach has been dropped.
For tracking residency a custom sorted vector based interval set has been added which is optimized for fast queries, as checking if a range is resident is in quite a hot path.
* vk_rasterizer: Move vertex and index buffer bind to rasterizer
Feels cleaner in general and wont have to pass barriers
* video_core: Add a new buffer barrier tracker
* buffer_cache: Fix arena migration bug
* video_core: Fully refactor resource management and barrier tracking
Introduces a new runtime class which is responsible for resource operations like buffer copy/fill, image upload/download/copy/resolve etc.
Having a centralized object makes the design significantly easier to maintain and extend. We dont have to pass scheduler in buffer/image objects anymore or blit helper in images.
Also barrier batching is now possible since we have a centralized place to track which buffer ranges were read/written. Image barriers can be batched as well before copies and also with buffer barriers.
* renderer_vulkan: Remove shader hle
New buffer changes make it mostly worthless aside from having vram build slower
* buffer_cache: Don't use VMA for residency allocations
We can avoid some cases of OOM where VMA would try to allocate more behind the scenes. But otherwise we can achieve a bit less memory usage with this
* video_core: Minor barrier fixes
* vk_rasterizer: Fix stupid ahh bug
* vk_runtime: Clear resources array
* buffer_cache: Fix partially null arena merges
* vk_rasterizer: Correct pipeline stage of accessed buffers
* renderer_vulkan: Add dxvk red-black tree for barrier tracking
Sorted vector interval sets are fast for queries but for barriers we have almost 50/50 split between query and insert traffic, so a more balanced structure is needed. the dxvk red-black tree seems pretty optimized for this use case so try it out
* texture_cache: Update size when resolving color depth alias
* renderer_vulkan: Add new staging pool to replace fixed buffers
Instead of allocating a large 512MB host allocation for uploads, now pool consists of 16MB blocks, any request below 16MB will use a ring of those blocks, otherwise it will allocate a dedicate large buffer that will be freed soon
* Revert "renderer_vulkan: Remove shader hle"
Will do it later
This reverts commit 52885faf3e.
* fix LICENSE
Seems I accidentally misunderstood something I saw in my decomp of the submit done ioctl. This one change effectively made submits synchronous, which was good for local testing, but effectively a hack.
This will re-break a bunch of things who's PM4 type 0 issues were fixed by my submit fixes PR.
* shader_recompiler: don't crash on unrecognized sharp source dword
MarkReadConstBufferSharpSources() asserted that every non-immediate
dword of a discovered V#/T#/S# was produced by one of the known
sharp source instructions (GetUserData, ReadConst, ReadConstBuffer),
or by V_READFIRSTLANE_B32 broadcasting one of those into an SGPR
(a common pattern for descriptors computed with per-thread arithmetic
that the compiler has proven is wave-uniform).
Shadwen (CUSA04540) computes its buffer sharp dwords in VGPRs and
broadcasts each one with v_readfirstlane_b32 before using it as a V#.
Neither MarkReadConstBufferSharpSources (resource_discover_pass.cpp)
nor SharpLocationFromSource (resource_patching_pass.cpp) recognized
ReadFirstLane as a passthrough, so the assert fired and crashed the
emulator (issue #5050).
Add a small shared helper, UnwrapReadFirstLane, that walks through any
chain of ReadFirstLane instructions to the real source, and use it in
both places so the pattern is actually recognized. Any dword whose
source still isn't recognized after that now logs a warning and is
left unmarked instead of asserting, falling back to the existing
null buffer/image handling (SharpLocationFromSource/UNKNOWN_LOCATION)
rather than crashing on a future, still-unknown pattern.
* Add IR dump
* shader_recompiler: see through ReadFirstLane at V# dword gathering
Per raphaelthegreat's review on #5059: rather than teaching every
consumer of a sharp dword (MarkReadConstBufferSharpSources,
SharpLocationFromSource) about ReadFirstLane, unwrap it once, right
where V# dwords are gathered in DiscoverBufferSharp. Everything
downstream (the existing 'see through' patterns, discovery, and
runtime patching) then sees the real producer automatically.
Confirmed against a real IR dump for Shadwen's cs shader 0x8673ea57
(dumped pre-res-discover): each V# dword is exactly
ReadFirstLane(ReadConstBuffer(...)) with no extra arithmetic, so a
single, non-recursive unwrap is sufficient here. Reverts the more
general UnwrapReadFirstLane helper and its two call sites, and
restores the original ASSERT in MarkReadConstBufferSharpSources, per
raphaelthegreat's request to replace that code if this simpler,
localized fix works.
---------
Co-authored-by: Gih-pt <gih-pt@users.noreply.github.com>
* Check for adjacent free areas in both directions
We know that the free regions will never contain start_address to end_address, so the original logic is mostly fine. The issue is that checking for both in the same find call will only ever return an area before or after our region.
This separates the adjacency check to look for before and after our unmap region, so situations where an unmap is surrounded by unmaps will properly coalesce into one reserved area.
Technically, by fixing this, host memory calls on POSIX platforms should run slightly better. I don't expect this to be a meaningful change though.
* Obligatory clang commit
* Added the following functions:
sceNpWordFilterCreateTitleCtx
sceNpWordFilterPollAsync
sceNpWordFilterDeleteRequest
sceNpWordFilterAbortRequest
sceNpWordFilterDeleteTitleCtx
sceNpWordFilterCreateAsyncRequest
sceNpWordFilterCensorComment
sceNpWordFilterSanitizeComment
Added context / title context handling, request processing, and helper functions.
Made a few layout changes to the three utility files, moved a few already existing functions (sorry shadow my OCD)
Codex was utilized in research and then utilized lightly during code impl and code review. The added functions were implemented directly based on their RE'd sprx counterparts. All comments were produced by me.
These functions will eventually require shadnet additions to actually process and perform what they are there for.
Testing was performed on CUSA000190 (KZ: SF). Testing for this is easiest to perform by doing any type of multiplayer based name entry, text input, score input, etc on titles that import and actually use these.
If this gets rejected we're gonna have to box.
* fucking clang.
* Properly prevent concurrent submits
In the actual libSceGnmDriver, there is a global mutex they lock in sceGnmDingDong, sceGnmSubmitDone, sceGnmSubmitAndFlipCommandBuffersForWorkload, and sceGnmSubmitCommandBuffersForWorkload (keep in mind, non-workload variants of these functions all redirect to the workload versions in the actual library).
This commit further separates submission logic out from the sceGnm functions, and adds proper locking logic following how the real libSceGnmDriver handles things. Also fixed a harmless inaccuracy in sceGnmSubmitAndFlipCommandBuffersForWorkload. Originally the function would run PatchFlipRequest before WaitGpuIdle, while the actual library does the gpu idle wait before it's equivalent to PatchFlipRequest.
Co-Authored-By: Gih-pt <296242299+gih-pt@users.noreply.github.com>
* Wait for submission lock to release in sceGnmSubmitDone
More accurate to what I've seen from PS4 kernel and fpPS4 code. Doesn't seem to fix anything though.
* Missing error returns in sceGnmSubmitAndFlipCommandBuffersForWorkload
* This very well may be the stupidest bug in shadPS4 history
😆
---------
Co-authored-by: Gih-pt <296242299+gih-pt@users.noreply.github.com>
* Add compare-and-swap operations for DS_CMPST
* Add shared atomic compare and swap functions for U32 and U64
* Add atomic compare and swap functions to SPIRV
* Add SharedAtomicCmpSwap function to IREmitter
* Add SharedAtomicCmpSwap function declaration
* Add SharedAtomicCmpSwap opcodes to microinstruction
* Add SharedAtomicCmpSwap opcodes for 32 and 64 bit
* Add SharedAtomicCmpSwap opcodes to resource pass
* Implement SharedAtomicCmpSwap32 in resource patching
Added support for SharedAtomicCmpSwap32 operation in resource patching pass.
* Add SharedAtomicCmpSwap32 and SharedAtomicCmpSwap64 cases
* Update shared memory barrier check for opcode range
* Add SharedAtomicCmpSwap64 to shared memory checks
* Add SharedAtomicCmpSwap opcodes to shared memory pass
* Add DS_CMPST function declaration to translate.h
* Remove comments from DS_CMPST function
Removed commented documentation for DS_CMPST function.
* Fix formatting in DS_CMPST function
* fix rebase
whos a dumbass, hog is
* shader_recompiler: Add base vertex and base instance attributes
* shader_recompiler: Emit prologue after parsing entire shader
It is needed to know if a fetch shader exists and if it has base vertex/instance add
* liverpool: Consolidate scope marker usage into a single helper
Its a lot less noise
* shader_recompiler: Implement support for base vertex/instance for indirect draws
A normal indexed draw has 3 main parameters other the number of indices; firstIndex, vertexOffset and firstInstance
Of those firstIndex is already provided by PM4 draw packet and simply offsets the index buffer base address.
The other parameters, at least on GCN are implemented in software, the driver can put those values somewhere in user data and fetch shader will add them to the vgpr of the vertex/instance index respectively.
The emulator already supports this, by parsing the fetch program, finding any such additions, which user data register is used and applying the parameters to the draw itself.
Note that in these cases the final vertex/instance index the shader sees is biased by these values, which is consistent with the provided gl_VertexIndex and gl_InstanceID vulkan builtins
Indirect draws work similarly but since the vertexOffset and firstInstance are in GPU modified buffers, the command processor can optionally copy them to a user data of the PM4 packets choosing.
To emulate this, add the packet provided offset into the shader runtime state and optionally set those registers to the vulkan builtins, overwritting the user data load before them.
Additionally since the vertex/instance index builtins have to optionally be unbiased because the hardware only provides unbiased values. This wasnt an issue with normal draws because the act of discovering a base offset
to either builtin implied the index was biased, but now its not guaranteed as such. An indirect draw might not have a fetch shader at all and manually add the base vertex/instance itself, where in that case the base would be added twice
* Fix a crash in the case of the guest trying to open a file with a valid mount point but an invalid malformed unicode string in the relative to mount part of the path string
* error logging
GCN has the concept of hardware and software stages. Software stages are the shader stages exposed in graphics APIs like opengl/vulkan, and each software stage runs on a hardware stage.
Depending on how many and what software stages are used in a graphics pipeline, the hardware stage a software one is executed on can change, so the mapping is not fixed.
The following table from ACO describes the various combinations for HW-SW stages: https://gitlab.freedesktop.org/mesa/mesa/-/blob/c90ae5f7/src/amd/compiler/README.md#gfx6-8
Certain software stages like the vertex and tess eval shader can run a lot of different hardware stages, which creates problems because runtime info is a tagged union on HW stage.
Current main uses struct inheritance to make it work, but that will get clunky when more data is added to other stages. In general there is a lack of separation between what data
is required when a HW stage is used and what for SW stage.
An example is the tess_emulated_primitive variable. This is used for skipping exports in a software vertex shader when tessellation stages have been inserted after it
for Quad/RectList emulation, because it is undefined behavior to not export them from the last geometry stage. Having this data field in the HW vertex info is not semantically
correct because HW vertex is always the last geometry stage that runs. So this only works when there isnt any HW geometry shader active, as the HW vertex shader is not normally processed in that case.
Another example is shader specialization previously checked the HW vertex stage to specialize the fetch shader. This only works on simple VS+PS pipelines when HW VS runs the SW VS.
Fetch shader is part of the SW VS state, its run on first geometry stage not the last, and this issue got exposed because of the new state separation.
This commit refactors runtime info to contain two tagged unions, one from SW stage and the other HW stage. This makes it obvious what is the scope of each state.
When recompiler does a switch on sw stage, it becomes more apparent it should only access sw state members unless it also checks which hw stage it runs on, which will result in less bugs
Valdis Bogdāns