From 1683d3670294f390b333c5b99069beffb2a697fe Mon Sep 17 00:00:00 2001 From: JordanTheToaster Date: Thu, 8 Jan 2026 23:07:05 +0000 Subject: [PATCH] 3rdparty: Update VMA to commit b3cbbb4 --- 3rdparty/vulkan/include/CHANGELOG.md | 15 +- 3rdparty/vulkan/include/LICENSE.txt | 19 + 3rdparty/vulkan/include/README.md | 6 +- 3rdparty/vulkan/include/vk_mem_alloc.h | 2339 ++++++++++------- 3rdparty/vulkan/include/vulkan/vk_mem_alloc.h | 2339 ++++++++++------- 5 files changed, 2720 insertions(+), 1998 deletions(-) create mode 100644 3rdparty/vulkan/include/LICENSE.txt diff --git a/3rdparty/vulkan/include/CHANGELOG.md b/3rdparty/vulkan/include/CHANGELOG.md index 3ba57c0fc0..26f4e9b686 100644 --- a/3rdparty/vulkan/include/CHANGELOG.md +++ b/3rdparty/vulkan/include/CHANGELOG.md @@ -1,3 +1,16 @@ +# 3.4.0 (2026-??-??) + +- Added member `VmaAllocationCreateInfo::minAlignment` (#523). + - Remember to always fully initialize structures with zeros and don't rely on their specific `sizeof` to ensure backward compatibility! + - Function `vmaCreateBufferWithAlignment` is now deprecated. +- Improvements for external memory export & import (#503): + - Added functions `vmaCreateDedicatedBuffer`, `vmaCreateDedicatedImage`, `vmaAllocateDedicatedMemory` offering extra parameter `void* pMemoryAllocateNext`. + - Added function `vmaGetMemoryWin32Handle2` offering extra parameter `VkExternalMemoryHandleTypeFlagBits handleType`. +- Added `VMA_VERSION` macro with library version number (#507). +- Improvements in the algorithm choosing memory type when `VMA_MEMORY_USAGE_AUTO*` is used (#520). +- Fixes for compatibility with C++20 modules on Clang 21 and GCC15 (#513, #514). +- Other fixes and improvements, including compatibility with various platforms and compilers, improvements in documentation, sample application, and tests. + # 3.3.0 (2025-05-12) Additions to the library API: @@ -8,7 +21,7 @@ Other changes: - Added macro `VMA_DEBUG_DONT_EXCEED_HEAP_SIZE_WITH_ALLOCATION_SIZE` with default value 1. - Changed macro `VMA_DEBUG_DONT_EXCEED_MAX_MEMORY_ALLOCATION_COUNT` default value from 0 to 1. -- Added documentation chapter "Frequenty asked questions". +- Added documentation chapter "Frequently asked questions". - Other fixes and improvements, including compatibility with various platforms and compilers. # 3.2.1 (2025-02-05) diff --git a/3rdparty/vulkan/include/LICENSE.txt b/3rdparty/vulkan/include/LICENSE.txt new file mode 100644 index 0000000000..be95175d6a --- /dev/null +++ b/3rdparty/vulkan/include/LICENSE.txt @@ -0,0 +1,19 @@ +Copyright (c) 2017-2026 Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/3rdparty/vulkan/include/README.md b/3rdparty/vulkan/include/README.md index 15d9cae97c..aa0c4c3fa3 100644 --- a/3rdparty/vulkan/include/README.md +++ b/3rdparty/vulkan/include/README.md @@ -8,7 +8,7 @@ Easy to integrate Vulkan memory allocation library. **Changelog:** See [CHANGELOG.md](CHANGELOG.md) -**Product page:** [Vulkan Memory Allocator on GPUOpen](https://gpuopen.com/gaming-product/vulkan-memory-allocator/) +**Product page:** [Vulkan Memory Allocator on GPUOpen](https://gpuopen.com/vulkan-memory-allocator/) [![Average time to resolve an issue](http://isitmaintained.com/badge/resolution/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.svg)](http://isitmaintained.com/project/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator "Average time to resolve an issue") @@ -44,7 +44,7 @@ Additional features: - Custom memory pools: Create a pool with desired parameters (e.g. fixed or limited maximum size) and allocate memory out of it. - Linear allocator: Create a pool with linear algorithm and use it for much faster allocations and deallocations in free-at-once, stack, double stack, or ring buffer fashion. - Support for Vulkan 1.0...1.4. -- Support for extensions (and equivalent functionality included in new Vulkan versions): +- Support for extensions (and equivalent functionality included in new core Vulkan versions): - VK_KHR_dedicated_allocation: Just enable it and it will be used automatically by the library. - VK_KHR_bind_memory2. - VK_KHR_maintenance4. @@ -60,7 +60,7 @@ Additional features: - JSON dump: Obtain a string in JSON format with detailed map of internal state, including list of allocations, their string names, and gaps between them. - Convert this JSON dump into a picture to visualize your memory. See [tools/GpuMemDumpVis](tools/GpuMemDumpVis/README.md). - Debugging incorrect memory usage: Enable initialization of all allocated memory with a bit pattern to detect usage of uninitialized or freed memory. Enable validation of a magic number after every allocation to detect out-of-bounds memory corruption. -- Support for interoperability with OpenGL. +- Support for interoperability with OpenGL, Direct3D, and other graphics APIs through external memory export. - Virtual allocator: Interface for using core allocation algorithm to allocate any custom data, e.g. pieces of one large buffer. # Prerequisites diff --git a/3rdparty/vulkan/include/vk_mem_alloc.h b/3rdparty/vulkan/include/vk_mem_alloc.h index 8df03649b5..24792855e2 100644 --- a/3rdparty/vulkan/include/vk_mem_alloc.h +++ b/3rdparty/vulkan/include/vk_mem_alloc.h @@ -1,5 +1,5 @@ // -// Copyright (c) 2017-2025 Advanced Micro Devices, Inc. All rights reserved. +// Copyright (c) 2017-2026 Advanced Micro Devices, Inc. All rights reserved. // // Permission is hereby granted, free of charge, to any person obtaining a copy // of this software and associated documentation files (the "Software"), to deal @@ -25,11 +25,11 @@ /** \mainpage Vulkan Memory Allocator -Version 3.3.0 +Version 3.4.0-development -Copyright (c) 2017-2025 Advanced Micro Devices, Inc. All rights reserved. \n +Copyright (c) 2017-2026 Advanced Micro Devices, Inc. All rights reserved. \n License: MIT \n -See also: [product page on GPUOpen](https://gpuopen.com/gaming-product/vulkan-memory-allocator/), +See also: [product page on GPUOpen](https://gpuopen.com/vulkan-memory-allocator/), [repository on GitHub](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator) @@ -79,6 +79,8 @@ See also: [product page on GPUOpen](https://gpuopen.com/gaming-product/vulkan-me - [Corruption detection](@ref debugging_memory_usage_corruption_detection) - [Leak detection features](@ref debugging_memory_usage_leak_detection) - \subpage other_api_interop + - [Exporting memory](@ref other_api_interop_exporting_memory) + - [Importing memory](@ref other_api_interop_importing_memory) - \subpage usage_patterns - [GPU-only resource](@ref usage_patterns_gpu_only) - [Staging copy for upload](@ref usage_patterns_staging_copy_upload) @@ -95,7 +97,6 @@ See also: [product page on GPUOpen](https://gpuopen.com/gaming-product/vulkan-me - \subpage enabling_buffer_device_address - \subpage vk_ext_memory_priority - \subpage vk_amd_device_coherent_memory - - \subpage vk_khr_external_memory_win32 - \subpage general_considerations - [Thread safety](@ref general_considerations_thread_safety) - [Versioning and compatibility](@ref general_considerations_versioning_and_compatibility) @@ -132,6 +133,8 @@ extern "C" { #include #endif +#define VMA_VERSION (VK_MAKE_VERSION(3, 4, 0)) + #if !defined(VMA_VULKAN_VERSION) #if defined(VK_VERSION_1_4) #define VMA_VULKAN_VERSION 1004000 @@ -478,7 +481,7 @@ typedef enum VmaAllocatorCreateFlagBits You should set this flag if you found available and enabled this device extension, while creating Vulkan device passed as VmaAllocatorCreateInfo::device. - For more information, see \ref vk_khr_external_memory_win32. + For more information, see \ref other_api_interop. */ VMA_ALLOCATOR_CREATE_KHR_EXTERNAL_MEMORY_WIN32_BIT = 0x00000200, @@ -1335,6 +1338,19 @@ typedef struct VmaAllocationCreateInfo Otherwise, it has the priority of a memory block where it is placed and this variable is ignored. */ float priority; + /** \brief Additional minimum alignment to be used for this allocation. Can be 0. + + Leave 0 (default) not to impose any additional alignment. If not 0, it must be a power of two. + + When creating a buffer or an image, specifying a custom alignment is not needed in most cases, + because Vulkan implementation inspects the `CreateInfo` structure (including intended usage flags) + and returns required alignment through functions like `vkGetBufferMemoryRequirements2`, which VMA automatically + uses and respects. + Extra alignment may be needed in some cases, like when using a buffer for acceleration structure scratch + (`VkPhysicalDeviceAccelerationStructurePropertiesKHR::minAccelerationStructureScratchOffsetAlignment`, see also issue #523) + or when doing interop with OpenGL. + */ + VkDeviceSize minAlignment; } VmaAllocationCreateInfo; /// Describes parameter of created #VmaPool. @@ -1378,8 +1394,14 @@ typedef struct VmaPoolCreateInfo /** \brief Additional minimum alignment to be used for all allocations created from this pool. Can be 0. Leave 0 (default) not to impose any additional alignment. If not 0, it must be a power of two. - It can be useful in cases where alignment returned by Vulkan by functions like `vkGetBufferMemoryRequirements` is not enough, - e.g. when doing interop with OpenGL. + + When creating a buffer or an image, specifying a custom alignment is not needed in most cases, + because Vulkan implementation inspects the `CreateInfo` structure (including intended usage flags) + and returns required alignment through functions like `vkGetBufferMemoryRequirements2`, which VMA automatically + uses and respects. + Extra alignment may be needed in some cases, like when using a buffer for acceleration structure scratch + (`VkPhysicalDeviceAccelerationStructurePropertiesKHR::minAccelerationStructureScratchOffsetAlignment`, see also issue #523) + or when doing interop with OpenGL. */ VkDeviceSize minAllocationAlignment; /** \brief Additional `pNext` chain to be attached to `VkMemoryAllocateInfo` used for every allocation made by this pool. Optional. @@ -1820,20 +1842,20 @@ VMA_CALL_PRE void VMA_CALL_POST vmaGetHeapBudgets( */ /** -\brief Helps to find memoryTypeIndex, given memoryTypeBits and VmaAllocationCreateInfo. +\brief Helps to find `memoryTypeIndex`, given `memoryTypeBits` and #VmaAllocationCreateInfo. This algorithm tries to find a memory type that: -- Is allowed by memoryTypeBits. -- Contains all the flags from pAllocationCreateInfo->requiredFlags. +- Is allowed by `memoryTypeBits`. +- Contains all the flags from `pAllocationCreateInfo->requiredFlags`. - Matches intended usage. -- Has as many flags from pAllocationCreateInfo->preferredFlags as possible. +- Has as many flags from `pAllocationCreateInfo->preferredFlags` as possible. -\return Returns VK_ERROR_FEATURE_NOT_PRESENT if not found. Receiving such result +\return Returns `VK_ERROR_FEATURE_NOT_PRESENT` if not found. Receiving such result from this function or any other allocating function probably means that your device doesn't support any memory type with requested features for the specific type of resource you want to use it for. Please check parameters of your -resource, like image layout (OPTIMAL versus LINEAR) or mip level count. +resource, like image layout (`OPTIMAL` versus `LINEAR`) or mip level count. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndex( VmaAllocator VMA_NOT_NULL allocator, @@ -1842,10 +1864,10 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndex( uint32_t* VMA_NOT_NULL pMemoryTypeIndex); /** -\brief Helps to find memoryTypeIndex, given VkBufferCreateInfo and VmaAllocationCreateInfo. +\brief Helps to find `memoryTypeIndex`, given `VkBufferCreateInfo` and #VmaAllocationCreateInfo. It can be useful e.g. to determine value to be used as VmaPoolCreateInfo::memoryTypeIndex. -It internally creates a temporary, dummy buffer that never has memory bound. +It may need to internally create a temporary, dummy buffer that never has memory bound. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( VmaAllocator VMA_NOT_NULL allocator, @@ -1854,10 +1876,10 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( uint32_t* VMA_NOT_NULL pMemoryTypeIndex); /** -\brief Helps to find memoryTypeIndex, given VkImageCreateInfo and VmaAllocationCreateInfo. +\brief Helps to find `memoryTypeIndex`, given `VkImageCreateInfo` and #VmaAllocationCreateInfo. It can be useful e.g. to determine value to be used as VmaPoolCreateInfo::memoryTypeIndex. -It internally creates a temporary, dummy image that never has memory bound. +It may need to internally create a temporary, dummy image that never has memory bound. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo( VmaAllocator VMA_NOT_NULL allocator, @@ -1962,16 +1984,25 @@ VMA_CALL_PRE void VMA_CALL_POST vmaSetPoolName( /** \brief General purpose memory allocation. -\param allocator -\param pVkMemoryRequirements -\param pCreateInfo +\param allocator The main allocator object. +\param pVkMemoryRequirements Requirements for the allocated memory. +\param pCreateInfo Allocation creation parameters. \param[out] pAllocation Handle to allocated memory. -\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo(). +\param[out] pAllocationInfo Optional, can be null. Information about allocated memory. It can be also fetched later using vmaGetAllocationInfo(). -You should free the memory using vmaFreeMemory() or vmaFreeMemoryPages(). +The function creates a #VmaAllocation object without creating a buffer or an image together with it. -It is recommended to use vmaAllocateMemoryForBuffer(), vmaAllocateMemoryForImage(), -vmaCreateBuffer(), vmaCreateImage() instead whenever possible. +- It is recommended to use vmaAllocateMemoryForBuffer(), vmaAllocateMemoryForImage(), + vmaCreateBuffer(), vmaCreateImage() instead whenever possible. +- You can also create a buffer or an image later in an existing allocation using + vmaCreateAliasingBuffer2(), vmaCreateAliasingImage2(). +- You can also create a buffer or an image on your own and bind it to an existing allocation + using vmaBindBufferMemory2(), vmaBindImageMemory2(). + +You must free the returned allocation object using vmaFreeMemory() or vmaFreeMemoryPages(). + +There is also extended version of this function: vmaAllocateDedicatedMemory() +that offers additional parameter `pMemoryAllocateNext`. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory( VmaAllocator VMA_NOT_NULL allocator, @@ -1980,6 +2011,22 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory( VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); +/** \brief General purpose allocation of a dedicated memory. + +This function is similar vmaAllocateMemory(), but +it always allocates dedicated memory - flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is implied. +It offers additional parameter `pMemoryAllocateNext`, +which can be used to attach `pNext` chain to the `VkMemoryAllocateInfo` structure. +It can be useful for importing external memory. For more information, see \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateDedicatedMemory( + VmaAllocator VMA_NOT_NULL allocator, + const VkMemoryRequirements* VMA_NOT_NULL pVkMemoryRequirements, + const VmaAllocationCreateInfo* VMA_NOT_NULL pCreateInfo, + void* VMA_NULLABLE VMA_EXTENDS_VK_STRUCT(VkMemoryAllocateInfo) pMemoryAllocateNext, + VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, + VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); + /** \brief General purpose memory allocation for multiple allocation objects at once. \param allocator Allocator object. @@ -2142,13 +2189,16 @@ VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocationMemoryProperties( /** \brief Given an allocation, returns Win32 handle that may be imported by other processes or APIs. -\param hTargetProcess Must be a valid handle to target process or null. If it's null, the function returns +\param allocator The main allocator object. +\param allocation Allocation. +\param hTargetProcess A valid handle to target process or null. If it's null, the function returns handle for the current process. \param[out] pHandle Output parameter that returns the handle. The function fills `pHandle` with handle that can be used in target process. The handle is fetched using function `vkGetMemoryWin32HandleKHR`. -When no longer needed, you must close it using: + +Each call to this function creates a new handle that must be closed using: \code CloseHandle(handle); @@ -2161,14 +2211,76 @@ Note the handle is returned for the entire `VkDeviceMemory` block that the alloc If the allocation is sub-allocated from a larger block, you may need to consider the offset of the allocation (VmaAllocationInfo::offset). +This function always uses `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT`. +An extended version of this function is available as vmaGetMemoryWin32Handle2() +that allows using other handle type. + +This function is available compile-time only when VK_KHR_external_memory_win32 extension is available. +It can be manually disabled by predefining `VMA_EXTERNAL_MEMORY_WIN32=0` macro. + If the function fails with `VK_ERROR_FEATURE_NOT_PRESENT` error code, please double-check -that VmaVulkanFunctions::vkGetMemoryWin32HandleKHR function pointer is set, e.g. either by using `VMA_DYNAMIC_VULKAN_FUNCTIONS` +that VmaVulkanFunctions::vkGetMemoryWin32HandleKHR function pointer is set, e.g. +either by using macro `VMA_DYNAMIC_VULKAN_FUNCTIONS` or by manually passing it through VmaAllocatorCreateInfo::pVulkanFunctions. -For more information, see chapter \ref vk_khr_external_memory_win32. +For more information, see chapter \ref other_api_interop. */ -VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle(VmaAllocator VMA_NOT_NULL allocator, - VmaAllocation VMA_NOT_NULL allocation, HANDLE hTargetProcess, HANDLE* VMA_NOT_NULL pHandle); +VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle( + VmaAllocator VMA_NOT_NULL allocator, + VmaAllocation VMA_NOT_NULL allocation, + HANDLE hTargetProcess, + HANDLE* VMA_NOT_NULL pHandle); + +/** +\brief Given an allocation, returns Win32 handle that may be imported by other processes or APIs. + +\param allocator The main allocator object. +\param allocation Allocation. +\param handleType Type of handle to be exported. It should be one of: + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_HEAP_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KHR` +\param hTargetProcess A valid handle to target process or null. If it's null, the function returns + handle for the current process. +\param[out] pHandle Output parameter that returns the handle. + +The function fills `pHandle` with handle that can be used in target process. +The handle is fetched using function `vkGetMemoryWin32HandleKHR`. + +If `handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR`, +or other NT handle types, +each call to this function creates a new handle that must be closed using: + +\code +CloseHandle(handle); +\endcode + +You can close it any time, before or after destroying the allocation object. +It is reference-counted internally by Windows. + +Note the handle is returned for the entire `VkDeviceMemory` block that the allocation belongs to. +If the allocation is sub-allocated from a larger block, you may need to consider the offset of the allocation +(VmaAllocationInfo::offset). + +This function is available compile-time only when VK_KHR_external_memory_win32 extension is available. +It can be manually disabled by predefining `VMA_EXTERNAL_MEMORY_WIN32=0` macro. + +If the function fails with `VK_ERROR_FEATURE_NOT_PRESENT` error code, please double-check +that VmaVulkanFunctions::vkGetMemoryWin32HandleKHR function pointer is set, e.g. +either by using macro `VMA_DYNAMIC_VULKAN_FUNCTIONS` +or by manually passing it through VmaAllocatorCreateInfo::pVulkanFunctions. + +For more information, see chapter \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle2( + VmaAllocator VMA_NOT_NULL allocator, + VmaAllocation VMA_NOT_NULL allocation, + VkExternalMemoryHandleTypeFlagBits handleType, + HANDLE hTargetProcess, + HANDLE* VMA_NOT_NULL pHandle); #endif // VMA_EXTERNAL_MEMORY_WIN32 /** \brief Maps memory represented by given allocation and returns pointer to it. @@ -2541,12 +2653,13 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindImageMemory2( /** \brief Creates a new `VkBuffer`, allocates and binds memory for it. -\param allocator -\param pBufferCreateInfo -\param pAllocationCreateInfo +\param allocator The main allocator object. +\param pBufferCreateInfo Buffer creation parameters. +\param pAllocationCreateInfo Allocation creation parameters. \param[out] pBuffer Buffer that was created. \param[out] pAllocation Allocation that was created. -\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo(). +\param[out] pAllocationInfo Optional, can be null. Information about allocated memory. + It can be also fetched later using vmaGetAllocationInfo(). This function automatically: @@ -2555,14 +2668,14 @@ This function automatically: -# Binds the buffer with the memory. If any of these operations fail, buffer and allocation are not created, -returned value is negative error code, `*pBuffer` and `*pAllocation` are null. +returned value is negative error code, `*pBuffer` and `*pAllocation` are returned as null. If the function succeeded, you must destroy both buffer and allocation when you no longer need them using either convenience function vmaDestroyBuffer() or separately, using `vkDestroyBuffer()` and vmaFreeMemory(). -If #VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT flag was used, -VK_KHR_dedicated_allocation extension is used internally to query driver whether +If VK_KHR_dedicated_allocation extenion or Vulkan version >= 1.1 is used, +the function queries the driver whether it requires or prefers the new buffer to have dedicated allocation. If yes, and if dedicated allocation is possible (#VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT is not used), it creates dedicated @@ -2572,6 +2685,9 @@ allocation for this buffer, just like when using \note This function creates a new `VkBuffer`. Sub-allocation of parts of one large buffer, although recommended as a good practice, is out of scope of this library and could be implemented by the user as a higher-level logic on top of VMA. + +There is also an extended versions of this function available with additional parameter `pMemoryAllocateNext` - +see vmaCreateDedicatedBuffer(). */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer( VmaAllocator VMA_NOT_NULL allocator, @@ -2586,6 +2702,10 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer( Similar to vmaCreateBuffer() but provides additional parameter `minAlignment` which allows to specify custom, minimum alignment to be used when placing the buffer inside a larger memory block, which may be needed e.g. for interop with OpenGL. + +\deprecated +This function in obsolete since new VmaAllocationCreateInfo::minAlignment member allows specifying custom +alignment while using any allocation function, like the standard vmaCreateBuffer(). */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( VmaAllocator VMA_NOT_NULL allocator, @@ -2596,6 +2716,23 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); +/** \brief Creates a dedicated buffer while offering extra parameter `pMemoryAllocateNext`. + +This function is similar vmaCreateBuffer(), but +it always allocates dedicated memory for the buffer - flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is implied. +It offers additional parameter `pMemoryAllocateNext`, +which can be used to attach `pNext` chain to the `VkMemoryAllocateInfo` structure. +It can be useful for importing external memory. For more information, see \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedBuffer( + VmaAllocator VMA_NOT_NULL allocator, + const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo, + const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo, + void* VMA_NULLABLE VMA_EXTENDS_VK_STRUCT(VkMemoryAllocateInfo) pMemoryAllocateNext, + VkBuffer VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pBuffer, + VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, + VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); + /** \brief Creates a new `VkBuffer`, binds already created memory for it. \param allocator @@ -2668,7 +2805,11 @@ VMA_CALL_PRE void VMA_CALL_POST vmaDestroyBuffer( VkBuffer VMA_NULLABLE_NON_DISPATCHABLE buffer, VmaAllocation VMA_NULLABLE allocation); -/// Function similar to vmaCreateBuffer(). +/** \brief Function similar to vmaCreateBuffer() but for images. + +There is also an extended version of this function available: vmaCreateDedicatedImage() +which offers additional parameter `pMemoryAllocateNext`. +*/ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage( VmaAllocator VMA_NOT_NULL allocator, const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo, @@ -2677,6 +2818,23 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage( VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); +/** \brief Function similar to vmaCreateDedicatedBuffer() but for images. + +This function is similar vmaCreateImage(), but +it always allocates dedicated memory for the image - flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is implied. +It offers additional parameter `pMemoryAllocateNext`, +which can be used to attach `pNext` chain to the `VkMemoryAllocateInfo` structure. +It can be useful for importing external memory. For more information, see \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedImage( + VmaAllocator VMA_NOT_NULL allocator, + const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo, + const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo, + void* VMA_NULLABLE VMA_EXTENDS_VK_STRUCT(VkMemoryAllocateInfo) pMemoryAllocateNext, + VkImage VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pImage, + VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, + VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); + /// Function similar to vmaCreateAliasingBuffer() but for images. VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingImage( VmaAllocator VMA_NOT_NULL allocator, @@ -3017,7 +3175,9 @@ remove them if not needed. #if defined(__ANDROID_API__) && (__ANDROID_API__ < 16) #include -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace +{ +void* vma_aligned_alloc(size_t alignment, size_t size) { // alignment must be >= sizeof(void*) if(alignment < sizeof(void*)) @@ -3027,6 +3187,7 @@ static void* vma_aligned_alloc(size_t alignment, size_t size) return memalign(alignment, size); } +} // namespace #elif defined(__APPLE__) || defined(__ANDROID__) || (defined(__linux__) && defined(__GLIBCXX__) && !defined(_GLIBCXX_HAVE_ALIGNED_ALLOC)) #include @@ -3034,7 +3195,9 @@ static void* vma_aligned_alloc(size_t alignment, size_t size) #include #endif -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace +{ +void* vma_aligned_alloc(size_t alignment, size_t size) { // Unfortunately, aligned_alloc causes VMA to crash due to it returning null pointers. (At least under 11.4) // Therefore, for now disable this specific exception until a proper solution is found. @@ -3062,35 +3225,46 @@ static void* vma_aligned_alloc(size_t alignment, size_t size) return pointer; return VMA_NULL; } +} // namespace #elif defined(_WIN32) -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace { +void* vma_aligned_alloc(size_t alignment, size_t size) { return _aligned_malloc(size, alignment); } +} // namespace #elif __cplusplus >= 201703L || _MSVC_LANG >= 201703L // C++17 -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace { +void* vma_aligned_alloc(size_t alignment, size_t size) { return aligned_alloc(alignment, size); } +} // namespace #else -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace +{ +void* vma_aligned_alloc(size_t alignment, size_t size) { VMA_ASSERT(0 && "Could not implement aligned_alloc automatically. Please enable C++17 or later in your compiler or provide custom implementation of macro VMA_SYSTEM_ALIGNED_MALLOC (and VMA_SYSTEM_ALIGNED_FREE if needed) using the API of your system."); return VMA_NULL; } +} // namespace #endif +namespace +{ #if defined(_WIN32) -static void vma_aligned_free(void* ptr) +void vma_aligned_free(void* ptr) { _aligned_free(ptr); } #else -static void vma_aligned_free(void* VMA_NULLABLE ptr) +void vma_aligned_free(void* VMA_NULLABLE ptr) { free(ptr); } #endif +} // namespace #ifndef VMA_ALIGN_OF #define VMA_ALIGN_OF(type) (alignof(type)) @@ -3171,18 +3345,20 @@ static void vma_aligned_free(void* VMA_NULLABLE ptr) // Define this macro to 1 to enable functions: vmaBuildStatsString, vmaFreeStatsString. #if VMA_STATS_STRING_ENABLED - static inline void VmaUint32ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint32_t num) +namespace { + inline void VmaUint32ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint32_t num) { snprintf(outStr, strLen, "%" PRIu32, num); } - static inline void VmaUint64ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint64_t num) + inline void VmaUint64ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint64_t num) { snprintf(outStr, strLen, "%" PRIu64, num); } - static inline void VmaPtrToStr(char* VMA_NOT_NULL outStr, size_t strLen, const void* ptr) + inline void VmaPtrToStr(char* VMA_NOT_NULL outStr, size_t strLen, const void* ptr) { snprintf(outStr, strLen, "%p", ptr); } +} // namespace #endif #ifndef VMA_MUTEX @@ -3377,31 +3553,33 @@ END OF CONFIGURATION */ #endif // _VMA_CONFIGURATION - -static const uint8_t VMA_ALLOCATION_FILL_PATTERN_CREATED = 0xDC; -static const uint8_t VMA_ALLOCATION_FILL_PATTERN_DESTROYED = 0xEF; +namespace +{ +constexpr uint8_t VMA_ALLOCATION_FILL_PATTERN_CREATED = 0xDC; +constexpr uint8_t VMA_ALLOCATION_FILL_PATTERN_DESTROYED = 0xEF; // Decimal 2139416166, float NaN, little-endian binary 66 E6 84 7F. -static const uint32_t VMA_CORRUPTION_DETECTION_MAGIC_VALUE = 0x7F84E666; +constexpr uint32_t VMA_CORRUPTION_DETECTION_MAGIC_VALUE = 0x7F84E666; // Copy of some Vulkan definitions so we don't need to check their existence just to handle few constants. -static const uint32_t VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY = 0x00000040; -static const uint32_t VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY = 0x00000080; -static const uint32_t VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY = 0x00020000; -static const uint32_t VK_IMAGE_CREATE_DISJOINT_BIT_COPY = 0x00000200; -static const int32_t VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT_COPY = 1000158000; -static const uint32_t VMA_ALLOCATION_INTERNAL_STRATEGY_MIN_OFFSET = 0x10000000U; -static const uint32_t VMA_ALLOCATION_TRY_COUNT = 32; -static const uint32_t VMA_VENDOR_ID_AMD = 4098; +constexpr uint32_t VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY = 0x00000040; +constexpr uint32_t VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY = 0x00000080; +constexpr uint32_t VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY = 0x00020000; +constexpr uint32_t VK_IMAGE_CREATE_DISJOINT_BIT_COPY = 0x00000200; +constexpr int32_t VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT_COPY = 1000158000; +constexpr uint32_t VMA_ALLOCATION_INTERNAL_STRATEGY_MIN_OFFSET = 0x10000000U; +constexpr uint32_t VMA_ALLOCATION_TRY_COUNT = 32; +constexpr uint32_t VMA_VENDOR_ID_AMD = 4098; // This one is tricky. Vulkan specification defines this code as available since // Vulkan 1.0, but doesn't actually define it in Vulkan SDK earlier than 1.2.131. // See pull request #207. #define VK_ERROR_UNKNOWN_COPY ((VkResult)-13) +} // namespace #if VMA_STATS_STRING_ENABLED // Correspond to values of enum VmaSuballocationType. -static const char* const VMA_SUBALLOCATION_TYPE_NAMES[] = +const char* const VMA_SUBALLOCATION_TYPE_NAMES[] = { "FREE", "UNKNOWN", @@ -3412,7 +3590,7 @@ static const char* const VMA_SUBALLOCATION_TYPE_NAMES[] = }; #endif -static const VkAllocationCallbacks VmaEmptyAllocationCallbacks = +const VkAllocationCallbacks VmaEmptyAllocationCallbacks = { VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL }; @@ -3451,6 +3629,8 @@ enum class VmaAllocationRequestType // Opaque handle used by allocation algorithms to identify single allocation in any conforming way. VK_DEFINE_NON_DISPATCHABLE_HANDLE(VmaAllocHandle); +struct VmaBufferImageUsage; + struct VmaMutexLock; struct VmaMutexLockRead; struct VmaMutexLockWrite; @@ -3515,419 +3695,12 @@ class VmaAllocationObjectAllocator; #endif // _VMA_FORWARD_DECLARATIONS +#ifndef _VMA_BUFFER_IMAGE_USAGE -#ifndef _VMA_FUNCTIONS - -/* -Returns number of bits set to 1 in (v). - -On specific platforms and compilers you can use intrinsics like: - -Visual Studio: - return __popcnt(v); -GCC, Clang: - return static_cast(__builtin_popcount(v)); - -Define macro VMA_COUNT_BITS_SET to provide your optimized implementation. -But you need to check in runtime whether user's CPU supports these, as some old processors don't. -*/ -static inline uint32_t VmaCountBitsSet(uint32_t v) -{ -#if VMA_CPP20 - return std::popcount(v); -#else - uint32_t c = v - ((v >> 1) & 0x55555555); - c = ((c >> 2) & 0x33333333) + (c & 0x33333333); - c = ((c >> 4) + c) & 0x0F0F0F0F; - c = ((c >> 8) + c) & 0x00FF00FF; - c = ((c >> 16) + c) & 0x0000FFFF; - return c; -#endif -} - -static inline uint8_t VmaBitScanLSB(uint64_t mask) -{ -#if defined(_MSC_VER) && defined(_WIN64) - unsigned long pos; - if (_BitScanForward64(&pos, mask)) - return static_cast(pos); - return UINT8_MAX; -#elif VMA_CPP20 - if(mask != 0) - return static_cast(std::countr_zero(mask)); - return UINT8_MAX; -#elif defined __GNUC__ || defined __clang__ - return static_cast(__builtin_ffsll(mask)) - 1U; -#else - uint8_t pos = 0; - uint64_t bit = 1; - do - { - if (mask & bit) - return pos; - bit <<= 1; - } while (pos++ < 63); - return UINT8_MAX; -#endif -} - -static inline uint8_t VmaBitScanLSB(uint32_t mask) -{ -#ifdef _MSC_VER - unsigned long pos; - if (_BitScanForward(&pos, mask)) - return static_cast(pos); - return UINT8_MAX; -#elif VMA_CPP20 - if(mask != 0) - return static_cast(std::countr_zero(mask)); - return UINT8_MAX; -#elif defined __GNUC__ || defined __clang__ - return static_cast(__builtin_ffs(mask)) - 1U; -#else - uint8_t pos = 0; - uint32_t bit = 1; - do - { - if (mask & bit) - return pos; - bit <<= 1; - } while (pos++ < 31); - return UINT8_MAX; -#endif -} - -static inline uint8_t VmaBitScanMSB(uint64_t mask) -{ -#if defined(_MSC_VER) && defined(_WIN64) - unsigned long pos; - if (_BitScanReverse64(&pos, mask)) - return static_cast(pos); -#elif VMA_CPP20 - if(mask != 0) - return 63 - static_cast(std::countl_zero(mask)); -#elif defined __GNUC__ || defined __clang__ - if (mask != 0) - return 63 - static_cast(__builtin_clzll(mask)); -#else - uint8_t pos = 63; - uint64_t bit = 1ULL << 63; - do - { - if (mask & bit) - return pos; - bit >>= 1; - } while (pos-- > 0); -#endif - return UINT8_MAX; -} - -static inline uint8_t VmaBitScanMSB(uint32_t mask) -{ -#ifdef _MSC_VER - unsigned long pos; - if (_BitScanReverse(&pos, mask)) - return static_cast(pos); -#elif VMA_CPP20 - if(mask != 0) - return 31 - static_cast(std::countl_zero(mask)); -#elif defined __GNUC__ || defined __clang__ - if (mask != 0) - return 31 - static_cast(__builtin_clz(mask)); -#else - uint8_t pos = 31; - uint32_t bit = 1UL << 31; - do - { - if (mask & bit) - return pos; - bit >>= 1; - } while (pos-- > 0); -#endif - return UINT8_MAX; -} - -/* -Returns true if given number is a power of two. -T must be unsigned integer number or signed integer but always nonnegative. -For 0 returns true. -*/ -template -inline bool VmaIsPow2(T x) -{ - return (x & (x - 1)) == 0; -} - -// Aligns given value up to nearest multiply of align value. For example: VmaAlignUp(11, 8) = 16. -// Use types like uint32_t, uint64_t as T. -template -static inline T VmaAlignUp(T val, T alignment) -{ - VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); - return (val + alignment - 1) & ~(alignment - 1); -} - -// Aligns given value down to nearest multiply of align value. For example: VmaAlignDown(11, 8) = 8. -// Use types like uint32_t, uint64_t as T. -template -static inline T VmaAlignDown(T val, T alignment) -{ - VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); - return val & ~(alignment - 1); -} - -// Division with mathematical rounding to nearest number. -template -static inline T VmaRoundDiv(T x, T y) -{ - return (x + (y / (T)2)) / y; -} - -// Divide by 'y' and round up to nearest integer. -template -static inline T VmaDivideRoundingUp(T x, T y) -{ - return (x + y - (T)1) / y; -} - -// Returns smallest power of 2 greater or equal to v. -static inline uint32_t VmaNextPow2(uint32_t v) -{ - v--; - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v++; - return v; -} - -static inline uint64_t VmaNextPow2(uint64_t v) -{ - v--; - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v |= v >> 32; - v++; - return v; -} - -// Returns largest power of 2 less or equal to v. -static inline uint32_t VmaPrevPow2(uint32_t v) -{ - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v = v ^ (v >> 1); - return v; -} - -static inline uint64_t VmaPrevPow2(uint64_t v) -{ - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v |= v >> 32; - v = v ^ (v >> 1); - return v; -} - -static inline bool VmaStrIsEmpty(const char* pStr) -{ - return pStr == VMA_NULL || *pStr == '\0'; -} - -/* -Returns true if two memory blocks occupy overlapping pages. -ResourceA must be in less memory offset than ResourceB. - -Algorithm is based on "Vulkan 1.0.39 - A Specification (with all registered Vulkan extensions)" -chapter 11.6 "Resource Memory Association", paragraph "Buffer-Image Granularity". -*/ -static inline bool VmaBlocksOnSamePage( - VkDeviceSize resourceAOffset, - VkDeviceSize resourceASize, - VkDeviceSize resourceBOffset, - VkDeviceSize pageSize) -{ - VMA_ASSERT(resourceAOffset + resourceASize <= resourceBOffset && resourceASize > 0 && pageSize > 0); - VkDeviceSize resourceAEnd = resourceAOffset + resourceASize - 1; - VkDeviceSize resourceAEndPage = resourceAEnd & ~(pageSize - 1); - VkDeviceSize resourceBStart = resourceBOffset; - VkDeviceSize resourceBStartPage = resourceBStart & ~(pageSize - 1); - return resourceAEndPage == resourceBStartPage; -} - -/* -Returns true if given suballocation types could conflict and must respect -VkPhysicalDeviceLimits::bufferImageGranularity. They conflict if one is buffer -or linear image and another one is optimal image. If type is unknown, behave -conservatively. -*/ -static inline bool VmaIsBufferImageGranularityConflict( - VmaSuballocationType suballocType1, - VmaSuballocationType suballocType2) -{ - if (suballocType1 > suballocType2) - { - std::swap(suballocType1, suballocType2); - } - - switch (suballocType1) - { - case VMA_SUBALLOCATION_TYPE_FREE: - return false; - case VMA_SUBALLOCATION_TYPE_UNKNOWN: - return true; - case VMA_SUBALLOCATION_TYPE_BUFFER: - return - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; - case VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN: - return - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR || - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; - case VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR: - return - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; - case VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL: - return false; - default: - VMA_ASSERT(0); - return true; - } -} - -static void VmaWriteMagicValue(void* pData, VkDeviceSize offset) -{ -#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION - uint32_t* pDst = (uint32_t*)((char*)pData + offset); - const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); - for (size_t i = 0; i < numberCount; ++i, ++pDst) - { - *pDst = VMA_CORRUPTION_DETECTION_MAGIC_VALUE; - } -#else - // no-op -#endif -} - -static bool VmaValidateMagicValue(const void* pData, VkDeviceSize offset) -{ -#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION - const uint32_t* pSrc = (const uint32_t*)((const char*)pData + offset); - const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); - for (size_t i = 0; i < numberCount; ++i, ++pSrc) - { - if (*pSrc != VMA_CORRUPTION_DETECTION_MAGIC_VALUE) - { - return false; - } - } -#endif - return true; -} - -/* -Fills structure with parameters of an example buffer to be used for transfers -during GPU memory defragmentation. -*/ -static void VmaFillGpuDefragmentationBufferCreateInfo(VkBufferCreateInfo& outBufCreateInfo) -{ - memset(&outBufCreateInfo, 0, sizeof(outBufCreateInfo)); - outBufCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO; - outBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; - outBufCreateInfo.size = (VkDeviceSize)VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE; // Example size. -} - - -/* -Performs binary search and returns iterator to first element that is greater or -equal to (key), according to comparison (cmp). - -Cmp should return true if first argument is less than second argument. - -Returned value is the found element, if present in the collection or place where -new element with value (key) should be inserted. -*/ -template -static IterT VmaBinaryFindFirstNotLess(IterT beg, IterT end, const KeyT& key, const CmpLess& cmp) -{ - size_t down = 0; - size_t up = size_t(end - beg); - while (down < up) - { - const size_t mid = down + (up - down) / 2; // Overflow-safe midpoint calculation - if (cmp(*(beg + mid), key)) - { - down = mid + 1; - } - else - { - up = mid; - } - } - return beg + down; -} - -template -IterT VmaBinaryFindSorted(const IterT& beg, const IterT& end, const KeyT& value, const CmpLess& cmp) -{ - IterT it = VmaBinaryFindFirstNotLess( - beg, end, value, cmp); - if (it == end || - (!cmp(*it, value) && !cmp(value, *it))) - { - return it; - } - return end; -} - -/* -Returns true if all pointers in the array are not-null and unique. -Warning! O(n^2) complexity. Use only inside VMA_HEAVY_ASSERT. -T must be pointer type, e.g. VmaAllocation, VmaPool. -*/ -template -static bool VmaValidatePointerArray(uint32_t count, const T* arr) -{ - for (uint32_t i = 0; i < count; ++i) - { - const T iPtr = arr[i]; - if (iPtr == VMA_NULL) - { - return false; - } - for (uint32_t j = i + 1; j < count; ++j) - { - if (iPtr == arr[j]) - { - return false; - } - } - } - return true; -} - -template -static inline void VmaPnextChainPushFront(MainT* mainStruct, NewT* newStruct) -{ - newStruct->pNext = mainStruct->pNext; - mainStruct->pNext = newStruct; -} // Finds structure with s->sType == sType in mainStruct->pNext chain. // Returns pointer to it. If not found, returns null. template -static inline const FindT* VmaPnextChainFind(const MainT* mainStruct, VkStructureType sType) +inline const FindT* VmaPnextChainFind(const MainT* mainStruct, VkStructureType sType) { for(const VkBaseInStructure* s = (const VkBaseInStructure*)mainStruct->pNext; s != VMA_NULL; s = s->pNext) @@ -4000,9 +3773,423 @@ VmaBufferImageUsage::VmaBufferImageUsage(const VkImageCreateInfo &createInfo) // VkImageUsageFlags2CreateInfoKHR, like the one for buffers... } +#endif // _VMA_BUFFER_IMAGE_USAGE + +#ifndef _VMA_FUNCTIONS + +namespace +{ + +/* +Returns number of bits set to 1 in (v). + +On specific platforms and compilers you can use intrinsics like: + +Visual Studio: + return __popcnt(v); +GCC, Clang: + return static_cast(__builtin_popcount(v)); + +Define macro VMA_COUNT_BITS_SET to provide your optimized implementation. +But you need to check in runtime whether user's CPU supports these, as some old processors don't. +*/ +inline uint32_t VmaCountBitsSet(uint32_t v) +{ +#if VMA_CPP20 + return std::popcount(v); +#else + uint32_t c = v - ((v >> 1) & 0x55555555); + c = ((c >> 2) & 0x33333333) + (c & 0x33333333); + c = ((c >> 4) + c) & 0x0F0F0F0F; + c = ((c >> 8) + c) & 0x00FF00FF; + c = ((c >> 16) + c) & 0x0000FFFF; + return c; +#endif +} + +inline uint8_t VmaBitScanLSB(uint64_t mask) +{ +#if defined(_MSC_VER) && defined(_WIN64) + unsigned long pos; + if (_BitScanForward64(&pos, mask)) + return static_cast(pos); + return UINT8_MAX; +#elif VMA_CPP20 + if(mask != 0) + return static_cast(std::countr_zero(mask)); + return UINT8_MAX; +#elif defined __GNUC__ || defined __clang__ + return static_cast(__builtin_ffsll(mask)) - 1U; +#else + uint8_t pos = 0; + uint64_t bit = 1; + do + { + if (mask & bit) + return pos; + bit <<= 1; + } while (pos++ < 63); + return UINT8_MAX; +#endif +} + +inline uint8_t VmaBitScanLSB(uint32_t mask) +{ +#ifdef _MSC_VER + unsigned long pos; + if (_BitScanForward(&pos, mask)) + return static_cast(pos); + return UINT8_MAX; +#elif VMA_CPP20 + if(mask != 0) + return static_cast(std::countr_zero(mask)); + return UINT8_MAX; +#elif defined __GNUC__ || defined __clang__ + return static_cast(__builtin_ffs(mask)) - 1U; +#else + uint8_t pos = 0; + uint32_t bit = 1; + do + { + if (mask & bit) + return pos; + bit <<= 1; + } while (pos++ < 31); + return UINT8_MAX; +#endif +} + +inline uint8_t VmaBitScanMSB(uint64_t mask) +{ +#if defined(_MSC_VER) && defined(_WIN64) + unsigned long pos; + if (_BitScanReverse64(&pos, mask)) + return static_cast(pos); +#elif VMA_CPP20 + if(mask != 0) + return 63 - static_cast(std::countl_zero(mask)); +#elif defined __GNUC__ || defined __clang__ + if (mask != 0) + return 63 - static_cast(__builtin_clzll(mask)); +#else + uint8_t pos = 63; + uint64_t bit = 1ULL << 63; + do + { + if (mask & bit) + return pos; + bit >>= 1; + } while (pos-- > 0); +#endif + return UINT8_MAX; +} + +inline uint8_t VmaBitScanMSB(uint32_t mask) +{ +#ifdef _MSC_VER + unsigned long pos; + if (_BitScanReverse(&pos, mask)) + return static_cast(pos); +#elif VMA_CPP20 + if(mask != 0) + return 31 - static_cast(std::countl_zero(mask)); +#elif defined __GNUC__ || defined __clang__ + if (mask != 0) + return 31 - static_cast(__builtin_clz(mask)); +#else + uint8_t pos = 31; + uint32_t bit = 1UL << 31; + do + { + if (mask & bit) + return pos; + bit >>= 1; + } while (pos-- > 0); +#endif + return UINT8_MAX; +} + +/* +Returns true if given number is a power of two. +T must be unsigned integer number or signed integer but always nonnegative. +For 0 returns true. +*/ +template +inline bool VmaIsPow2(T x) +{ + return (x & (x - 1)) == 0; +} + +// Aligns given value up to nearest multiply of align value. For example: VmaAlignUp(11, 8) = 16. +// Use types like uint32_t, uint64_t as T. +template +inline T VmaAlignUp(T val, T alignment) +{ + VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); + return (val + alignment - 1) & ~(alignment - 1); +} + +// Aligns given value down to nearest multiply of align value. For example: VmaAlignDown(11, 8) = 8. +// Use types like uint32_t, uint64_t as T. +template +inline T VmaAlignDown(T val, T alignment) +{ + VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); + return val & ~(alignment - 1); +} + +// Division with mathematical rounding to nearest number. +template +inline T VmaRoundDiv(T x, T y) +{ + return (x + (y / (T)2)) / y; +} + +// Divide by 'y' and round up to nearest integer. +template +inline T VmaDivideRoundingUp(T x, T y) +{ + return (x + y - (T)1) / y; +} + +// Returns smallest power of 2 greater or equal to v. +inline uint32_t VmaNextPow2(uint32_t v) +{ + v--; + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v++; + return v; +} + +inline uint64_t VmaNextPow2(uint64_t v) +{ + v--; + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v |= v >> 32; + v++; + return v; +} + +// Returns largest power of 2 less or equal to v. +inline uint32_t VmaPrevPow2(uint32_t v) +{ + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v = v ^ (v >> 1); + return v; +} + +inline uint64_t VmaPrevPow2(uint64_t v) +{ + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v |= v >> 32; + v = v ^ (v >> 1); + return v; +} + +inline bool VmaStrIsEmpty(const char* pStr) +{ + return pStr == VMA_NULL || *pStr == '\0'; +} + +/* +Returns true if two memory blocks occupy overlapping pages. +ResourceA must be in less memory offset than ResourceB. + +Algorithm is based on "Vulkan 1.0.39 - A Specification (with all registered Vulkan extensions)" +chapter 11.6 "Resource Memory Association", paragraph "Buffer-Image Granularity". +*/ +inline bool VmaBlocksOnSamePage( + VkDeviceSize resourceAOffset, + VkDeviceSize resourceASize, + VkDeviceSize resourceBOffset, + VkDeviceSize pageSize) +{ + VMA_ASSERT(resourceAOffset + resourceASize <= resourceBOffset && resourceASize > 0 && pageSize > 0); + VkDeviceSize resourceAEnd = resourceAOffset + resourceASize - 1; + VkDeviceSize resourceAEndPage = resourceAEnd & ~(pageSize - 1); + VkDeviceSize resourceBStart = resourceBOffset; + VkDeviceSize resourceBStartPage = resourceBStart & ~(pageSize - 1); + return resourceAEndPage == resourceBStartPage; +} + +/* +Returns true if given suballocation types could conflict and must respect +VkPhysicalDeviceLimits::bufferImageGranularity. They conflict if one is buffer +or linear image and another one is optimal image. If type is unknown, behave +conservatively. +*/ +inline bool VmaIsBufferImageGranularityConflict( + VmaSuballocationType suballocType1, + VmaSuballocationType suballocType2) +{ + if (suballocType1 > suballocType2) + { + std::swap(suballocType1, suballocType2); + } + + switch (suballocType1) + { + case VMA_SUBALLOCATION_TYPE_FREE: + return false; + case VMA_SUBALLOCATION_TYPE_UNKNOWN: + return true; + case VMA_SUBALLOCATION_TYPE_BUFFER: + return + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; + case VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN: + return + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR || + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; + case VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR: + return + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; + case VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL: + return false; + default: + VMA_ASSERT(0); + return true; + } +} + +void VmaWriteMagicValue(void* pData, VkDeviceSize offset) +{ +#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION + uint32_t* pDst = (uint32_t*)((char*)pData + offset); + const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); + for (size_t i = 0; i < numberCount; ++i, ++pDst) + { + *pDst = VMA_CORRUPTION_DETECTION_MAGIC_VALUE; + } +#else + // no-op +#endif +} + +bool VmaValidateMagicValue(const void* pData, VkDeviceSize offset) +{ +#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION + const uint32_t* pSrc = (const uint32_t*)((const char*)pData + offset); + const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); + for (size_t i = 0; i < numberCount; ++i, ++pSrc) + { + if (*pSrc != VMA_CORRUPTION_DETECTION_MAGIC_VALUE) + { + return false; + } + } +#endif + return true; +} + +/* +Fills structure with parameters of an example buffer to be used for transfers +during GPU memory defragmentation. +*/ +void VmaFillGpuDefragmentationBufferCreateInfo(VkBufferCreateInfo& outBufCreateInfo) +{ + memset(&outBufCreateInfo, 0, sizeof(outBufCreateInfo)); + outBufCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO; + outBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; + outBufCreateInfo.size = (VkDeviceSize)VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE; // Example size. +} + + +/* +Performs binary search and returns iterator to first element that is greater or +equal to (key), according to comparison (cmp). + +Cmp should return true if first argument is less than second argument. + +Returned value is the found element, if present in the collection or place where +new element with value (key) should be inserted. +*/ +template +IterT VmaBinaryFindFirstNotLess(IterT beg, IterT end, const KeyT& key, const CmpLess& cmp) +{ + size_t down = 0; + size_t up = size_t(end - beg); + while (down < up) + { + const size_t mid = down + (up - down) / 2; // Overflow-safe midpoint calculation + if (cmp(*(beg + mid), key)) + { + down = mid + 1; + } + else + { + up = mid; + } + } + return beg + down; +} + +template +IterT VmaBinaryFindSorted(const IterT& beg, const IterT& end, const KeyT& value, const CmpLess& cmp) +{ + IterT it = VmaBinaryFindFirstNotLess( + beg, end, value, cmp); + if (it == end || + (!cmp(*it, value) && !cmp(value, *it))) + { + return it; + } + return end; +} + +/* +Returns true if all pointers in the array are not-null and unique. +Warning! O(n^2) complexity. Use only inside VMA_HEAVY_ASSERT. +T must be pointer type, e.g. VmaAllocation, VmaPool. +*/ +template +bool VmaValidatePointerArray(uint32_t count, const T* arr) +{ + for (uint32_t i = 0; i < count; ++i) + { + const T iPtr = arr[i]; + if (iPtr == VMA_NULL) + { + return false; + } + for (uint32_t j = i + 1; j < count; ++j) + { + if (iPtr == arr[j]) + { + return false; + } + } + } + return true; +} + +template +inline void VmaPnextChainPushFront(MainT* mainStruct, NewT* newStruct) +{ + newStruct->pNext = mainStruct->pNext; + mainStruct->pNext = newStruct; +} + // This is the main algorithm that guides the selection of a memory type best for an allocation - // converts usage to required/preferred/not preferred flags. -static bool FindMemoryPreferences( +bool FindMemoryPreferences( bool isIntegratedGPU, const VmaAllocationCreateInfo& allocCreateInfo, VmaBufferImageUsage bufImgUsage, @@ -4078,8 +4265,11 @@ static bool FindMemoryPreferences( } else { - // Always CPU memory. - outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + if(hostAccessAllowTransferInstead) + outPreferredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + else + // Always CPU memory. + outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; } } // CPU sequential write - may be CPU or host-visible GPU memory, uncached and write-combined. @@ -4094,7 +4284,12 @@ static bool FindMemoryPreferences( } else { - outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + if(hostAccessAllowTransferInstead) + outPreferredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + else + // Always CPU memory. + outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + // Direct GPU access, CPU sequential write (e.g. a dynamic uniform buffer updated every frame) if(deviceAccess) { @@ -4154,7 +4349,7 @@ static bool FindMemoryPreferences( //////////////////////////////////////////////////////////////////////////////// // Memory allocation -static void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t size, size_t alignment) +inline void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t size, size_t alignment) { void* result = VMA_NULL; if ((pAllocationCallbacks != VMA_NULL) && @@ -4174,7 +4369,7 @@ static void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t return result; } -static void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr) +inline void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr) { if ((pAllocationCallbacks != VMA_NULL) && (pAllocationCallbacks->pfnFree != VMA_NULL)) @@ -4188,13 +4383,13 @@ static void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr } template -static T* VmaAllocate(const VkAllocationCallbacks* pAllocationCallbacks) +T* VmaAllocate(const VkAllocationCallbacks* pAllocationCallbacks) { return (T*)VmaMalloc(pAllocationCallbacks, sizeof(T), VMA_ALIGN_OF(T)); } template -static T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, size_t count) +T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, size_t count) { return (T*)VmaMalloc(pAllocationCallbacks, sizeof(T) * count, VMA_ALIGN_OF(T)); } @@ -4204,14 +4399,14 @@ static T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, si #define vma_new_array(allocator, type, count) new(VmaAllocateArray((allocator), (count)))(type) template -static void vma_delete(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr) +void vma_delete(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr) { ptr->~T(); VmaFree(pAllocationCallbacks, ptr); } template -static void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr, size_t count) +void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr, size_t count) { if (ptr != VMA_NULL) { @@ -4223,7 +4418,7 @@ static void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, } } -static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr) +char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr) { if (srcStr != VMA_NULL) { @@ -4236,7 +4431,7 @@ static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char } #if VMA_STATS_STRING_ENABLED -static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr, size_t strLen) +char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr, size_t strLen) { if (srcStr != VMA_NULL) { @@ -4249,7 +4444,7 @@ static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char } #endif // VMA_STATS_STRING_ENABLED -static void VmaFreeString(const VkAllocationCallbacks* allocs, char* str) +void VmaFreeString(const VkAllocationCallbacks* allocs, char* str) { if (str != VMA_NULL) { @@ -4287,11 +4482,17 @@ bool VmaVectorRemoveSorted(VectorT& vector, const typename VectorT::value_type& } return false; } + +} // namespace + #endif // _VMA_FUNCTIONS #ifndef _VMA_STATISTICS_FUNCTIONS -static void VmaClearStatistics(VmaStatistics& outStats) +namespace +{ + +void VmaClearStatistics(VmaStatistics& outStats) { outStats.blockCount = 0; outStats.allocationCount = 0; @@ -4299,7 +4500,7 @@ static void VmaClearStatistics(VmaStatistics& outStats) outStats.allocationBytes = 0; } -static void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src) +void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src) { inoutStats.blockCount += src.blockCount; inoutStats.allocationCount += src.allocationCount; @@ -4307,7 +4508,7 @@ static void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src inoutStats.allocationBytes += src.allocationBytes; } -static void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats) +void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats) { VmaClearStatistics(outStats.statistics); outStats.unusedRangeCount = 0; @@ -4317,7 +4518,7 @@ static void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats) outStats.unusedRangeSizeMax = 0; } -static void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats, VkDeviceSize size) +void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats, VkDeviceSize size) { inoutStats.statistics.allocationCount++; inoutStats.statistics.allocationBytes += size; @@ -4325,14 +4526,14 @@ static void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats inoutStats.allocationSizeMax = VMA_MAX(inoutStats.allocationSizeMax, size); } -static void VmaAddDetailedStatisticsUnusedRange(VmaDetailedStatistics& inoutStats, VkDeviceSize size) +void VmaAddDetailedStatisticsUnusedRange(VmaDetailedStatistics& inoutStats, VkDeviceSize size) { inoutStats.unusedRangeCount++; inoutStats.unusedRangeSizeMin = VMA_MIN(inoutStats.unusedRangeSizeMin, size); inoutStats.unusedRangeSizeMax = VMA_MAX(inoutStats.unusedRangeSizeMax, size); } -static void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const VmaDetailedStatistics& src) +void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const VmaDetailedStatistics& src) { VmaAddStatistics(inoutStats.statistics, src.statistics); inoutStats.unusedRangeCount += src.unusedRangeCount; @@ -4342,6 +4543,8 @@ static void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const Vm inoutStats.unusedRangeSizeMax = VMA_MAX(inoutStats.unusedRangeSizeMax, src.unusedRangeSizeMax); } +} // namespace + #endif // _VMA_STATISTICS_FUNCTIONS #ifndef _VMA_MUTEX_LOCK @@ -4441,8 +4644,8 @@ struct VmaStlAllocator VmaStlAllocator(const VmaStlAllocator&) = default; VmaStlAllocator& operator=(const VmaStlAllocator&) = delete; - T* allocate(size_t n) { return VmaAllocateArray(m_pCallbacks, n); } - void deallocate(T* p, size_t n) { VmaFree(m_pCallbacks, p); } + T* allocate(size_t n); + void deallocate(T* p, size_t n); template bool operator==(const VmaStlAllocator& rhs) const @@ -4455,6 +4658,12 @@ struct VmaStlAllocator return m_pCallbacks != rhs.m_pCallbacks; } }; + +template +T* VmaStlAllocator::allocate(size_t n) { return VmaAllocateArray(m_pCallbacks, n); } + +template +void VmaStlAllocator::deallocate(T* p, size_t n) { VmaFree(m_pCallbacks, p); } #endif // _VMA_STL_ALLOCATOR #ifndef _VMA_VECTOR @@ -4476,7 +4685,7 @@ public: VmaVector(size_t count, const T& value, const AllocatorT& allocator) : VmaVector(count, allocator) {} VmaVector(const VmaVector& src); VmaVector& operator=(const VmaVector& rhs); - ~VmaVector() { VmaFree(m_Allocator.m_pCallbacks, m_pArray); } + ~VmaVector(); bool empty() const { return m_Count == 0; } size_t size() const { return m_Count; } @@ -4517,6 +4726,9 @@ private: }; #ifndef _VMA_VECTOR_FUNCTIONS +template +VmaVector::~VmaVector() { VmaFree(m_Allocator.m_pCallbacks, m_pArray); } + template VmaVector::VmaVector(const AllocatorT& allocator) : m_Allocator(allocator), @@ -4600,7 +4812,8 @@ void VmaVector::resize(size_t newCount) if (newCapacity != m_Capacity) { - T* const newArray = newCapacity ? VmaAllocateArray(m_Allocator.m_pCallbacks, newCapacity) : VMA_NULL; + VMA_HEAVY_ASSERT(newCapacity > 0); + T* const newArray = VmaAllocateArray(m_Allocator.m_pCallbacks, newCapacity); const size_t elementsToCopy = VMA_MIN(m_Count, newCount); if (elementsToCopy != 0) { @@ -4657,17 +4870,23 @@ void VmaVector::remove(size_t index) } #endif // _VMA_VECTOR_FUNCTIONS +namespace +{ + template -static void VmaVectorInsert(VmaVector& vec, size_t index, const T& item) +void VmaVectorInsert(VmaVector& vec, size_t index, const T& item) { vec.insert(index, item); } template -static void VmaVectorRemove(VmaVector& vec, size_t index) +void VmaVectorRemove(VmaVector& vec, size_t index) { vec.remove(index); } + +} // namespace + #endif // _VMA_VECTOR #ifndef _VMA_SMALL_VECTOR @@ -6065,7 +6284,10 @@ void VmaJsonWriter::WriteIndent(bool oneLess) } #endif // _VMA_JSON_WRITER_FUNCTIONS -static void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedStatistics& stat) +namespace +{ + +void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedStatistics& stat) { json.BeginObject(); @@ -6096,6 +6318,9 @@ static void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedSta } json.EndObject(); } + +} // namespace + #endif // _VMA_JSON_WRITER #ifndef _VMA_MAPPING_HYSTERESIS @@ -6176,7 +6401,7 @@ public: } private: - static const int32_t COUNTER_MIN_EXTRA_MAPPING = 7; + static constexpr int32_t COUNTER_MIN_EXTRA_MAPPING = 7; uint32_t m_MinorCounter = 0; uint32_t m_MajorCounter = 0; @@ -6203,40 +6428,45 @@ class VmaWin32Handle { public: VmaWin32Handle() noexcept : m_hHandle(VMA_NULL) { } - explicit VmaWin32Handle(HANDLE hHandle) noexcept : m_hHandle(hHandle) { } - ~VmaWin32Handle() noexcept { if (m_hHandle != VMA_NULL) { ::CloseHandle(m_hHandle); } } + explicit VmaWin32Handle(HANDLE hHandle) noexcept + : m_hHandle(hHandle) + , m_IsNTHandle(IsNTHandle(hHandle)) + { + } + ~VmaWin32Handle() noexcept { if (m_hHandle != VMA_NULL && m_IsNTHandle) { ::CloseHandle(m_hHandle); } } VMA_CLASS_NO_COPY_NO_MOVE(VmaWin32Handle) public: // Strengthened - VkResult GetHandle(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, HANDLE hTargetProcess, bool useMutex, HANDLE* pHandle) noexcept + VkResult GetHandle(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, bool useMutex, HANDLE* pHandle) noexcept { *pHandle = VMA_NULL; // Try to get handle first. - if (m_hHandle != VMA_NULL) - { - *pHandle = Duplicate(hTargetProcess); - return VK_SUCCESS; - } - VkResult res = VK_SUCCESS; - // If failed, try to create it. + if (m_hHandle == VMA_NULL) { VmaMutexLockWrite lock(m_Mutex, useMutex); if (m_hHandle == VMA_NULL) { - res = Create(device, memory, pvkGetMemoryWin32HandleKHR, &m_hHandle); + res = Create(device, memory, pvkGetMemoryWin32HandleKHR, handleType, &m_hHandle); + if (res != VK_SUCCESS) { + m_hHandle = VMA_NULL; + return res; + } + m_IsNTHandle = IsNTHandle(m_hHandle); } } - - *pHandle = Duplicate(hTargetProcess); + if (res == VK_SUCCESS) { + // KMT handle is returned as is. + *pHandle = m_IsNTHandle ? Duplicate(hTargetProcess) : m_hHandle; + } return res; } operator bool() const noexcept { return m_hHandle != VMA_NULL; } private: // Not atomic - static VkResult Create(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, HANDLE* pHandle) noexcept + static VkResult Create(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE* pHandle) noexcept { VkResult res = VK_ERROR_FEATURE_NOT_PRESENT; if (pvkGetMemoryWin32HandleKHR != VMA_NULL) @@ -6244,7 +6474,7 @@ private: VkMemoryGetWin32HandleInfoKHR handleInfo{ }; handleInfo.sType = VK_STRUCTURE_TYPE_MEMORY_GET_WIN32_HANDLE_INFO_KHR; handleInfo.memory = memory; - handleInfo.handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR; + handleInfo.handleType = handleType; res = pvkGetMemoryWin32HandleKHR(device, &handleInfo, pHandle); } return res; @@ -6262,9 +6492,15 @@ private: } return hDupHandle; } + static bool IsNTHandle(HANDLE hHandle) noexcept + { + DWORD flags = 0; + return (hHandle != VMA_NULL) ? (::GetHandleInformation(hHandle, &flags) != 0) : false; + } private: HANDLE m_hHandle; VMA_RW_MUTEX m_Mutex; // Protects access m_Handle + bool m_IsNTHandle = false; // True if m_Handle is NT handle, false if it's a KMT handle. }; #else class VmaWin32Handle @@ -6272,6 +6508,7 @@ class VmaWin32Handle // ABI compatibility void* placeholder = VMA_NULL; VMA_RW_MUTEX placeholder2; + bool placeholder3 = false; }; #endif // VMA_EXTERNAL_MEMORY_WIN32 @@ -6346,6 +6583,7 @@ public: VkResult CreateWin32Handle( const VmaAllocator hAllocator, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, + VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* pHandle)noexcept; #endif // VMA_EXTERNAL_MEMORY_WIN32 @@ -6460,7 +6698,7 @@ public: #endif #if VMA_EXTERNAL_MEMORY_WIN32 - VkResult GetWin32Handle(VmaAllocator hAllocator, HANDLE hTargetProcess, HANDLE* hHandle) noexcept; + VkResult GetWin32Handle(VmaAllocator hAllocator, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* hHandle) noexcept; #endif // VMA_EXTERNAL_MEMORY_WIN32 private: @@ -6966,7 +7204,7 @@ public: bool FinishValidation(ValidationContext& ctx) const; private: - static const uint16_t MAX_LOW_BUFFER_IMAGE_GRANULARITY = 256; + static constexpr uint16_t MAX_LOW_BUFFER_IMAGE_GRANULARITY = 256; struct RegionInfo { @@ -8907,11 +9145,11 @@ private: // According to original paper it should be preferable 4 or 5: // M. Masmano, I. Ripoll, A. Crespo, and J. Real "TLSF: a New Dynamic Memory Allocator for Real-Time Systems" // http://www.gii.upv.es/tlsf/files/ecrts04_tlsf.pdf - static const uint8_t SECOND_LEVEL_INDEX = 5; - static const uint16_t SMALL_BUFFER_SIZE = 256; - static const uint32_t INITIAL_BLOCK_ALLOC_COUNT = 16; - static const uint8_t MEMORY_CLASS_SHIFT = 7; - static const uint8_t MAX_MEMORY_CLASSES = 65 - MEMORY_CLASS_SHIFT; + static constexpr uint8_t SECOND_LEVEL_INDEX = 5; + static constexpr uint16_t SMALL_BUFFER_SIZE = 256; + static constexpr uint32_t INITIAL_BLOCK_ALLOC_COUNT = 16; + static constexpr uint8_t MEMORY_CLASS_SHIFT = 7; + static constexpr uint8_t MAX_MEMORY_CLASSES = 65 - MEMORY_CLASS_SHIFT; class Block { @@ -9927,7 +10165,7 @@ public: private: // Max number of allocations to ignore due to size constraints before ending single pass - static const uint8_t MAX_ALLOCS_TO_IGNORE = 16; + static constexpr uint8_t MAX_ALLOCS_TO_IGNORE = 16; enum class CounterStatus { Pass, Ignore, End }; struct FragmentedBlock @@ -10385,14 +10623,32 @@ public: VmaBufferImageUsage bufImgUsage, uint32_t* pMemoryTypeIndex) const; + // Common code for public functions vmaCreateBuffer, vmaCreateBufferWithAlignment, etc. + VkResult CreateBuffer( + const VkBufferCreateInfo* pBufferCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, // pNext chain for VkMemoryAllocateInfo. + VkBuffer* pBuffer, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo); + // Common code for public functions vmaCreateImage, vmaCreateDedicatedImage. + VkResult CreateImage( + const VkImageCreateInfo* pImageCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, // pNext chain for VkMemoryAllocateInfo. + VkImage* pImage, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo); + // Main allocation function. VkResult AllocateMemory( - const VkMemoryRequirements& vkMemReq, + VkMemoryRequirements vkMemReq, bool requiresDedicatedAllocation, bool prefersDedicatedAllocation, VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + void* pMemoryAllocateNext, // Optional pNext chain for VkMemoryAllocateInfo. const VmaAllocationCreateInfo& createInfo, VmaSuballocationType suballocType, size_t allocationCount, @@ -10538,6 +10794,7 @@ private: VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + void* pMemoryAllocateNext, // Optional pNext chain for VkMemoryAllocateInfo. const VmaAllocationCreateInfo& createInfo, uint32_t memTypeIndex, VmaSuballocationType suballocType, @@ -10577,7 +10834,7 @@ private: VmaBufferImageUsage dedicatedBufferImageUsage, size_t allocationCount, VmaAllocation* pAllocations, - const void* pNextChain = VMA_NULL); + const void* pNextChain); void FreeDedicatedMemory(VmaAllocation allocation); @@ -10609,30 +10866,32 @@ private: #ifndef _VMA_MEMORY_FUNCTIONS -static void* VmaMalloc(VmaAllocator hAllocator, size_t size, size_t alignment) +namespace +{ +inline void* VmaMalloc(VmaAllocator hAllocator, size_t size, size_t alignment) { return VmaMalloc(&hAllocator->m_AllocationCallbacks, size, alignment); } -static void VmaFree(VmaAllocator hAllocator, void* ptr) +inline void VmaFree(VmaAllocator hAllocator, void* ptr) { VmaFree(&hAllocator->m_AllocationCallbacks, ptr); } template -static T* VmaAllocate(VmaAllocator hAllocator) +T* VmaAllocate(VmaAllocator hAllocator) { return (T*)VmaMalloc(hAllocator, sizeof(T), VMA_ALIGN_OF(T)); } template -static T* VmaAllocateArray(VmaAllocator hAllocator, size_t count) +T* VmaAllocateArray(VmaAllocator hAllocator, size_t count) { return (T*)VmaMalloc(hAllocator, sizeof(T) * count, VMA_ALIGN_OF(T)); } template -static void vma_delete(VmaAllocator hAllocator, T* ptr) +void vma_delete(VmaAllocator hAllocator, T* ptr) { if(ptr != VMA_NULL) { @@ -10642,7 +10901,7 @@ static void vma_delete(VmaAllocator hAllocator, T* ptr) } template -static void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count) +void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count) { if(ptr != VMA_NULL) { @@ -10651,6 +10910,7 @@ static void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count) VmaFree(hAllocator, ptr); } } +} // namespace #endif // _VMA_MEMORY_FUNCTIONS #ifndef _VMA_DEVICE_MEMORY_BLOCK_FUNCTIONS @@ -10904,10 +11164,10 @@ VkResult VmaDeviceMemoryBlock::BindImageMemory( } #if VMA_EXTERNAL_MEMORY_WIN32 -VkResult VmaDeviceMemoryBlock::CreateWin32Handle(const VmaAllocator hAllocator, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, HANDLE hTargetProcess, HANDLE* pHandle) noexcept +VkResult VmaDeviceMemoryBlock::CreateWin32Handle(const VmaAllocator hAllocator, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* pHandle) noexcept { VMA_ASSERT(pHandle); - return m_Handle.GetHandle(hAllocator->m_hDevice, m_hMemory, pvkGetMemoryWin32HandleKHR, hTargetProcess, hAllocator->m_UseMutex, pHandle); + return m_Handle.GetHandle(hAllocator->m_hDevice, m_hMemory, pvkGetMemoryWin32HandleKHR, handleType, hTargetProcess, hAllocator->m_UseMutex, pHandle); } #endif // VMA_EXTERNAL_MEMORY_WIN32 #endif // _VMA_DEVICE_MEMORY_BLOCK_FUNCTIONS @@ -11225,16 +11485,16 @@ void VmaAllocation_T::PrintParameters(class VmaJsonWriter& json) const } } #if VMA_EXTERNAL_MEMORY_WIN32 -VkResult VmaAllocation_T::GetWin32Handle(VmaAllocator hAllocator, HANDLE hTargetProcess, HANDLE* pHandle) noexcept +VkResult VmaAllocation_T::GetWin32Handle(VmaAllocator hAllocator, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* pHandle) noexcept { auto pvkGetMemoryWin32HandleKHR = hAllocator->GetVulkanFunctions().vkGetMemoryWin32HandleKHR; switch (m_Type) { case ALLOCATION_TYPE_BLOCK: - return m_BlockAllocation.m_Block->CreateWin32Handle(hAllocator, pvkGetMemoryWin32HandleKHR, hTargetProcess, pHandle); + return m_BlockAllocation.m_Block->CreateWin32Handle(hAllocator, pvkGetMemoryWin32HandleKHR, handleType, hTargetProcess, pHandle); case ALLOCATION_TYPE_DEDICATED: EnsureExtraData(hAllocator); - return m_DedicatedAllocation.m_ExtraData->m_Handle.GetHandle(hAllocator->m_hDevice, m_DedicatedAllocation.m_hMemory, pvkGetMemoryWin32HandleKHR, hTargetProcess, hAllocator->m_UseMutex, pHandle); + return m_DedicatedAllocation.m_ExtraData->m_Handle.GetHandle(hAllocator->m_hDevice, m_DedicatedAllocation.m_hMemory, pvkGetMemoryWin32HandleKHR, handleType, hTargetProcess, hAllocator->m_UseMutex, pHandle); default: VMA_ASSERT(0); return VK_ERROR_FEATURE_NOT_PRESENT; @@ -11348,11 +11608,13 @@ bool VmaBlockVector::IsEmpty() bool VmaBlockVector::IsCorruptionDetectionEnabled() const { - const uint32_t requiredMemFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; - return (VMA_DEBUG_DETECT_CORRUPTION != 0) && - (VMA_DEBUG_MARGIN > 0) && - (m_Algorithm == 0 || m_Algorithm == VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT) && +#if (VMA_DEBUG_DETECT_CORRUPTION == 0) || (VMA_DEBUG_MARGIN == 0) + return false; +#else + constexpr uint32_t requiredMemFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; + return (m_Algorithm == 0 || m_Algorithm == VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT) && (m_hAllocator->m_MemProps.memoryTypes[m_MemoryTypeIndex].propertyFlags & requiredMemFlags) == requiredMemFlags; +#endif } VkResult VmaBlockVector::Allocate( @@ -11802,10 +12064,11 @@ VkResult VmaBlockVector::CommitAllocationRequest( else (*pAllocation)->SetUserData(m_hAllocator, pUserData); m_hAllocator->m_Budget.AddAllocation(m_hAllocator->MemoryTypeIndexToHeapIndex(m_MemoryTypeIndex), allocRequest.size); - if (VMA_DEBUG_INITIALIZE_ALLOCATIONS) - { - m_hAllocator->FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); - } + +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + m_hAllocator->FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); +#endif + if (IsCorruptionDetectionEnabled()) { VkResult res = pBlock->WriteMagicValueAfterAllocation(m_hAllocator, (*pAllocation)->GetOffset(), allocRequest.size); @@ -13496,6 +13759,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + void* pMemoryAllocateNext, const VmaAllocationCreateInfo& createInfo, uint32_t memTypeIndex, VmaSuballocationType suballocType, @@ -13516,6 +13780,14 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( if(res != VK_SUCCESS) return res; + const void* allocateNextPtr = blockVector.GetAllocationNextPtr(); + if(pMemoryAllocateNext != VMA_NULL) + { + VMA_ASSERT(allocateNextPtr == VMA_NULL && + "You shouldn't create a dedicated allocation with a custom pMemoryAllocateNext if the pNext chain is already provided for this pool."); + allocateNextPtr = pMemoryAllocateNext; + } + if((finalCreateInfo.flags & VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT) != 0) { return AllocateDedicatedMemory( @@ -13536,7 +13808,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( dedicatedBufferImageUsage, allocationCount, pAllocations, - blockVector.GetAllocationNextPtr()); + allocateNextPtr); } const bool canAllocateDedicated = @@ -13579,7 +13851,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( dedicatedBufferImageUsage, allocationCount, pAllocations, - blockVector.GetAllocationNextPtr()); + allocateNextPtr); if(res == VK_SUCCESS) { // Succeeded: AllocateDedicatedMemory function already filled pMemory, nothing more to do here. @@ -13620,7 +13892,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( dedicatedBufferImageUsage, allocationCount, pAllocations, - blockVector.GetAllocationNextPtr()); + allocateNextPtr); if(res == VK_SUCCESS) { // Succeeded: AllocateDedicatedMemory function already filled pMemory, nothing more to do here. @@ -13628,6 +13900,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( return VK_SUCCESS; } } + // Everything failed: Return error code. VMA_DEBUG_LOG(" vkAllocateMemory FAILED"); return res; @@ -13825,10 +14098,10 @@ VkResult VmaAllocator_T::AllocateDedicatedMemoryPage( else (*pAllocation)->SetUserData(this, pUserData); m_Budget.AddAllocation(MemoryTypeIndexToHeapIndex(memTypeIndex), size); - if(VMA_DEBUG_INITIALIZE_ALLOCATIONS) - { - FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); - } + +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); +#endif return VK_SUCCESS; } @@ -14027,11 +14300,12 @@ VkResult VmaAllocator_T::CalcAllocationParams( return VK_ERROR_FEATURE_NOT_PRESENT; } - if(VMA_DEBUG_ALWAYS_DEDICATED_MEMORY && - (inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) != 0) +#if VMA_DEBUG_ALWAYS_DEDICATED_MEMORY + if((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) != 0) { inoutCreateInfo.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; } +#endif // Non-auto USAGE values imply HOST_ACCESS flags. // And so does VMA_MEMORY_USAGE_UNKNOWN because it is used with custom pools. @@ -14050,13 +14324,176 @@ VkResult VmaAllocator_T::CalcAllocationParams( return VK_SUCCESS; } +VkResult VmaAllocator_T::CreateBuffer( + const VkBufferCreateInfo* pBufferCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkBuffer* pBuffer, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + *pBuffer = VK_NULL_HANDLE; + *pAllocation = VK_NULL_HANDLE; + + if (pBufferCreateInfo->size == 0) + { + return VK_ERROR_INITIALIZATION_FAILED; + } + if ((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 && + !m_UseKhrBufferDeviceAddress) + { + VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used."); + return VK_ERROR_INITIALIZATION_FAILED; + } + + // 1. Create VkBuffer. + VkResult res = (*m_VulkanFunctions.vkCreateBuffer)(m_hDevice, pBufferCreateInfo, + GetAllocationCallbacks(), pBuffer); + if (res >= 0) + { + // 2. vkGetBufferMemoryRequirements. + VkMemoryRequirements vkMemReq = {}; + bool requiresDedicatedAllocation = false; + bool prefersDedicatedAllocation = false; + GetBufferMemoryRequirements(*pBuffer, vkMemReq, + requiresDedicatedAllocation, prefersDedicatedAllocation); + + // 3. Allocate memory using allocator. + res = AllocateMemory( + vkMemReq, + requiresDedicatedAllocation, + prefersDedicatedAllocation, + *pBuffer, // dedicatedBuffer + VK_NULL_HANDLE, // dedicatedImage + VmaBufferImageUsage(*pBufferCreateInfo, m_UseKhrMaintenance5), // dedicatedBufferImageUsage + pMemoryAllocateNext, + *pAllocationCreateInfo, + VMA_SUBALLOCATION_TYPE_BUFFER, + 1, // allocationCount + pAllocation); + if (res >= 0) + { + // 3. Bind buffer with memory. + if ((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) + { + res = BindBufferMemory(*pAllocation, 0, *pBuffer, VMA_NULL); + } + if (res >= 0) + { + // All steps succeeded. +#if VMA_STATS_STRING_ENABLED + (*pAllocation)->InitBufferUsage(*pBufferCreateInfo, m_UseKhrMaintenance5); +#endif + if (pAllocationInfo != VMA_NULL) + { + GetAllocationInfo(*pAllocation, pAllocationInfo); + } + + return VK_SUCCESS; + } + FreeMemory(1, pAllocation); + *pAllocation = VK_NULL_HANDLE; + (*m_VulkanFunctions.vkDestroyBuffer)(m_hDevice, *pBuffer, GetAllocationCallbacks()); + *pBuffer = VK_NULL_HANDLE; + return res; + } + (*m_VulkanFunctions.vkDestroyBuffer)(m_hDevice, *pBuffer, GetAllocationCallbacks()); + *pBuffer = VK_NULL_HANDLE; + return res; + } + return res; +} + +VkResult VmaAllocator_T::CreateImage( + const VkImageCreateInfo* pImageCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkImage* pImage, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + *pImage = VK_NULL_HANDLE; + *pAllocation = VK_NULL_HANDLE; + + if (pImageCreateInfo->extent.width == 0 || + pImageCreateInfo->extent.height == 0 || + pImageCreateInfo->extent.depth == 0 || + pImageCreateInfo->mipLevels == 0 || + pImageCreateInfo->arrayLayers == 0) + { + return VK_ERROR_INITIALIZATION_FAILED; + } + + // 1. Create VkImage. + VkResult res = (*m_VulkanFunctions.vkCreateImage)(m_hDevice, pImageCreateInfo, + GetAllocationCallbacks(), pImage); + if (res == VK_SUCCESS) + { + VmaSuballocationType suballocType = pImageCreateInfo->tiling == VK_IMAGE_TILING_OPTIMAL ? + VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL : + VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR; + + // 2. Allocate memory using allocator. + VkMemoryRequirements vkMemReq = {}; + bool requiresDedicatedAllocation = false; + bool prefersDedicatedAllocation = false; + GetImageMemoryRequirements(*pImage, vkMemReq, + requiresDedicatedAllocation, prefersDedicatedAllocation); + + res = AllocateMemory( + vkMemReq, + requiresDedicatedAllocation, + prefersDedicatedAllocation, + VK_NULL_HANDLE, // dedicatedBuffer + *pImage, // dedicatedImage + VmaBufferImageUsage(*pImageCreateInfo), // dedicatedBufferImageUsage + pMemoryAllocateNext, + *pAllocationCreateInfo, + suballocType, + 1, // allocationCount + pAllocation); + if (res == VK_SUCCESS) + { + // 3. Bind image with memory. + if ((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) + { + res = BindImageMemory(*pAllocation, 0, *pImage, VMA_NULL); + } + if (res == VK_SUCCESS) + { + // All steps succeeded. +#if VMA_STATS_STRING_ENABLED + (*pAllocation)->InitImageUsage(*pImageCreateInfo); +#endif + if (pAllocationInfo != VMA_NULL) + { + GetAllocationInfo(*pAllocation, pAllocationInfo); + } + + return VK_SUCCESS; + } + FreeMemory(1, pAllocation); + *pAllocation = VK_NULL_HANDLE; + (*m_VulkanFunctions.vkDestroyImage)(m_hDevice, *pImage, GetAllocationCallbacks()); + *pImage = VK_NULL_HANDLE; + return res; + } + (*m_VulkanFunctions.vkDestroyImage)(m_hDevice, *pImage, GetAllocationCallbacks()); + *pImage = VK_NULL_HANDLE; + return res; + } + return res; +} + VkResult VmaAllocator_T::AllocateMemory( - const VkMemoryRequirements& vkMemReq, + VkMemoryRequirements vkMemReq, bool requiresDedicatedAllocation, bool prefersDedicatedAllocation, VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + // pNext chain for VkMemoryAllocateInfo. When used, must specify requiresDedicatedAllocation = true. + void* pMemoryAllocateNext, const VmaAllocationCreateInfo& createInfo, VmaSuballocationType suballocType, size_t allocationCount, @@ -14064,8 +14501,15 @@ VkResult VmaAllocator_T::AllocateMemory( { memset(pAllocations, 0, sizeof(VmaAllocation) * allocationCount); + vkMemReq.alignment = VMA_MAX(vkMemReq.alignment, createInfo.minAlignment); VMA_ASSERT(VmaIsPow2(vkMemReq.alignment)); + // If using custom pNext chain for VkMemoryAllocateInfo, must require dedicated allocations. + if(pMemoryAllocateNext != VMA_NULL) + { + requiresDedicatedAllocation = true; + } + if(vkMemReq.size == 0) { return VK_ERROR_INITIALIZATION_FAILED; @@ -14087,6 +14531,7 @@ VkResult VmaAllocator_T::AllocateMemory( dedicatedBuffer, dedicatedImage, dedicatedBufferImageUsage, + pMemoryAllocateNext, createInfoFinal, blockVector.GetMemoryTypeIndex(), suballocType, @@ -14116,6 +14561,7 @@ VkResult VmaAllocator_T::AllocateMemory( dedicatedBuffer, dedicatedImage, dedicatedBufferImageUsage, + pMemoryAllocateNext, createInfoFinal, memTypeIndex, suballocType, @@ -14150,10 +14596,9 @@ void VmaAllocator_T::FreeMemory( if(allocation != VK_NULL_HANDLE) { - if(VMA_DEBUG_INITIALIZE_ALLOCATIONS) - { - FillAllocation(allocation, VMA_ALLOCATION_FILL_PATTERN_DESTROYED); - } +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + FillAllocation(allocation, VMA_ALLOCATION_FILL_PATTERN_DESTROYED); +#endif switch(allocation->GetType()) { @@ -14997,8 +15442,8 @@ void VmaAllocator_T::UpdateVulkanBudget() void VmaAllocator_T::FillAllocation(VmaAllocation hAllocation, uint8_t pattern) { - if(VMA_DEBUG_INITIALIZE_ALLOCATIONS && - hAllocation->IsMappingAllowed() && +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + if(hAllocation->IsMappingAllowed() && (m_MemProps.memoryTypes[hAllocation->GetMemoryTypeIndex()].propertyFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) != 0) { void* pData = VMA_NULL; @@ -15014,6 +15459,7 @@ void VmaAllocator_T::FillAllocation(VmaAllocation hAllocation, uint8_t pattern) VMA_ASSERT(0 && "VMA_DEBUG_INITIALIZE_ALLOCATIONS is enabled, but couldn't map memory to fill allocation."); } } +#endif // #if VMA_DEBUG_INITIALIZE_ALLOCATIONS } uint32_t VmaAllocator_T::GetGpuDefragmentationMemoryTypeBits() @@ -15543,7 +15989,8 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( VkResult res = VK_SUCCESS; #if VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 - if(funcs->vkGetDeviceBufferMemoryRequirements) + if (funcs->vkGetDeviceBufferMemoryRequirements && + (allocator->m_UseKhrMaintenance4 || allocator->m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))) { // Can query straight from VkBufferCreateInfo :) VkDeviceBufferMemoryRequirementsKHR devBufMemReq = {VK_STRUCTURE_TYPE_DEVICE_BUFFER_MEMORY_REQUIREMENTS_KHR}; @@ -15552,29 +15999,27 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( VkMemoryRequirements2 memReq = {VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2}; (*funcs->vkGetDeviceBufferMemoryRequirements)(hDev, &devBufMemReq, &memReq); - res = allocator->FindMemoryTypeIndex( + return allocator->FindMemoryTypeIndex( memReq.memoryRequirements.memoryTypeBits, pAllocationCreateInfo, VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), pMemoryTypeIndex); } - else #endif // VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 + + // Must create a dummy buffer to query :( + VkBuffer hBuffer = VK_NULL_HANDLE; + res = funcs->vkCreateBuffer( + hDev, pBufferCreateInfo, allocator->GetAllocationCallbacks(), &hBuffer); + if(res == VK_SUCCESS) { - // Must create a dummy buffer to query :( - VkBuffer hBuffer = VK_NULL_HANDLE; - res = funcs->vkCreateBuffer( - hDev, pBufferCreateInfo, allocator->GetAllocationCallbacks(), &hBuffer); - if(res == VK_SUCCESS) - { - VkMemoryRequirements memReq = {}; - funcs->vkGetBufferMemoryRequirements(hDev, hBuffer, &memReq); + VkMemoryRequirements memReq = {}; + funcs->vkGetBufferMemoryRequirements(hDev, hBuffer, &memReq); - res = allocator->FindMemoryTypeIndex( - memReq.memoryTypeBits, pAllocationCreateInfo, - VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), pMemoryTypeIndex); + res = allocator->FindMemoryTypeIndex( + memReq.memoryTypeBits, pAllocationCreateInfo, + VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), pMemoryTypeIndex); - funcs->vkDestroyBuffer( - hDev, hBuffer, allocator->GetAllocationCallbacks()); - } + funcs->vkDestroyBuffer( + hDev, hBuffer, allocator->GetAllocationCallbacks()); } return res; } @@ -15595,7 +16040,8 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo( VkResult res = VK_SUCCESS; #if VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 - if(funcs->vkGetDeviceImageMemoryRequirements) + if(funcs->vkGetDeviceImageMemoryRequirements && + (allocator->m_UseKhrMaintenance4 || allocator->m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))) { // Can query straight from VkImageCreateInfo :) VkDeviceImageMemoryRequirementsKHR devImgMemReq = {VK_STRUCTURE_TYPE_DEVICE_IMAGE_MEMORY_REQUIREMENTS_KHR}; @@ -15606,29 +16052,27 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo( VkMemoryRequirements2 memReq = {VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2}; (*funcs->vkGetDeviceImageMemoryRequirements)(hDev, &devImgMemReq, &memReq); - res = allocator->FindMemoryTypeIndex( + return allocator->FindMemoryTypeIndex( memReq.memoryRequirements.memoryTypeBits, pAllocationCreateInfo, VmaBufferImageUsage(*pImageCreateInfo), pMemoryTypeIndex); } - else #endif // VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 + + // Must create a dummy image to query :( + VkImage hImage = VK_NULL_HANDLE; + res = funcs->vkCreateImage( + hDev, pImageCreateInfo, allocator->GetAllocationCallbacks(), &hImage); + if(res == VK_SUCCESS) { - // Must create a dummy image to query :( - VkImage hImage = VK_NULL_HANDLE; - res = funcs->vkCreateImage( - hDev, pImageCreateInfo, allocator->GetAllocationCallbacks(), &hImage); - if(res == VK_SUCCESS) - { - VkMemoryRequirements memReq = {}; - funcs->vkGetImageMemoryRequirements(hDev, hImage, &memReq); + VkMemoryRequirements memReq = {}; + funcs->vkGetImageMemoryRequirements(hDev, hImage, &memReq); - res = allocator->FindMemoryTypeIndex( - memReq.memoryTypeBits, pAllocationCreateInfo, - VmaBufferImageUsage(*pImageCreateInfo), pMemoryTypeIndex); + res = allocator->FindMemoryTypeIndex( + memReq.memoryTypeBits, pAllocationCreateInfo, + VmaBufferImageUsage(*pImageCreateInfo), pMemoryTypeIndex); - funcs->vkDestroyImage( - hDev, hImage, allocator->GetAllocationCallbacks()); - } + funcs->vkDestroyImage( + hDev, hImage, allocator->GetAllocationCallbacks()); } return res; } @@ -15748,6 +16192,42 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory( VK_NULL_HANDLE, // dedicatedBuffer VK_NULL_HANDLE, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext + *pCreateInfo, + VMA_SUBALLOCATION_TYPE_UNKNOWN, + 1, // allocationCount + pAllocation); + + if(pAllocationInfo != VMA_NULL && result == VK_SUCCESS) + { + allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); + } + + return result; +} + +VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateDedicatedMemory( + VmaAllocator allocator, + const VkMemoryRequirements* pVkMemoryRequirements, + const VmaAllocationCreateInfo* pCreateInfo, + void* pMemoryAllocateNext, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + VMA_ASSERT(allocator && pVkMemoryRequirements && pCreateInfo && pAllocation); + + VMA_DEBUG_LOG("vmaAllocateDedicatedMemory"); + + VMA_DEBUG_GLOBAL_MUTEX_LOCK + + VkResult result = allocator->AllocateMemory( + *pVkMemoryRequirements, + true, // requiresDedicatedAllocation + false, // prefersDedicatedAllocation + VK_NULL_HANDLE, // dedicatedBuffer + VK_NULL_HANDLE, // dedicatedImage + VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + pMemoryAllocateNext, *pCreateInfo, VMA_SUBALLOCATION_TYPE_UNKNOWN, 1, // allocationCount @@ -15787,6 +16267,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryPages( VK_NULL_HANDLE, // dedicatedBuffer VK_NULL_HANDLE, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext *pCreateInfo, VMA_SUBALLOCATION_TYPE_UNKNOWN, allocationCount, @@ -15830,6 +16311,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForBuffer( buffer, // dedicatedBuffer VK_NULL_HANDLE, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext *pCreateInfo, VMA_SUBALLOCATION_TYPE_BUFFER, 1, // allocationCount @@ -15869,6 +16351,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForImage( VK_NULL_HANDLE, // dedicatedBuffer image, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext *pCreateInfo, VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN, 1, // allocationCount @@ -16264,86 +16747,13 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer( VmaAllocationInfo* pAllocationInfo) { VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && pBuffer && pAllocation); - - if(pBufferCreateInfo->size == 0) - { - return VK_ERROR_INITIALIZATION_FAILED; - } - if((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 && - !allocator->m_UseKhrBufferDeviceAddress) - { - VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used."); - return VK_ERROR_INITIALIZATION_FAILED; - } - VMA_DEBUG_LOG("vmaCreateBuffer"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - VMA_DEBUG_GLOBAL_MUTEX_LOCK + return allocator->CreateBuffer(pBufferCreateInfo, pAllocationCreateInfo, + VMA_NULL, // pMemoryAllocateNext + pBuffer, pAllocation, pAllocationInfo); - *pBuffer = VK_NULL_HANDLE; - *pAllocation = VK_NULL_HANDLE; - - // 1. Create VkBuffer. - VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)( - allocator->m_hDevice, - pBufferCreateInfo, - allocator->GetAllocationCallbacks(), - pBuffer); - if(res >= 0) - { - // 2. vkGetBufferMemoryRequirements. - VkMemoryRequirements vkMemReq = {}; - bool requiresDedicatedAllocation = false; - bool prefersDedicatedAllocation = false; - allocator->GetBufferMemoryRequirements(*pBuffer, vkMemReq, - requiresDedicatedAllocation, prefersDedicatedAllocation); - - // 3. Allocate memory using allocator. - res = allocator->AllocateMemory( - vkMemReq, - requiresDedicatedAllocation, - prefersDedicatedAllocation, - *pBuffer, // dedicatedBuffer - VK_NULL_HANDLE, // dedicatedImage - VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), // dedicatedBufferImageUsage - *pAllocationCreateInfo, - VMA_SUBALLOCATION_TYPE_BUFFER, - 1, // allocationCount - pAllocation); - - if(res >= 0) - { - // 3. Bind buffer with memory. - if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) - { - res = allocator->BindBufferMemory(*pAllocation, 0, *pBuffer, VMA_NULL); - } - if(res >= 0) - { - // All steps succeeded. - #if VMA_STATS_STRING_ENABLED - (*pAllocation)->InitBufferUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5); - #endif - if(pAllocationInfo != VMA_NULL) - { - allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); - } - - return VK_SUCCESS; - } - allocator->FreeMemory( - 1, // allocationCount - pAllocation); - *pAllocation = VK_NULL_HANDLE; - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - return res; } VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( @@ -16356,89 +16766,36 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( VmaAllocationInfo* pAllocationInfo) { VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && VmaIsPow2(minAlignment) && pBuffer && pAllocation); - - if(pBufferCreateInfo->size == 0) - { - return VK_ERROR_INITIALIZATION_FAILED; - } - if((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 && - !allocator->m_UseKhrBufferDeviceAddress) - { - VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used."); - return VK_ERROR_INITIALIZATION_FAILED; - } - VMA_DEBUG_LOG("vmaCreateBufferWithAlignment"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - VMA_DEBUG_GLOBAL_MUTEX_LOCK + VmaAllocationCreateInfo allocCreateInfoCopy = *pAllocationCreateInfo; + allocCreateInfoCopy.minAlignment = VMA_MAX(allocCreateInfoCopy.minAlignment, minAlignment); - *pBuffer = VK_NULL_HANDLE; - *pAllocation = VK_NULL_HANDLE; + return allocator->CreateBuffer(pBufferCreateInfo, &allocCreateInfoCopy, + VMA_NULL, // pMemoryAllocateNext + pBuffer, pAllocation, pAllocationInfo); +} - // 1. Create VkBuffer. - VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)( - allocator->m_hDevice, - pBufferCreateInfo, - allocator->GetAllocationCallbacks(), - pBuffer); - if(res >= 0) - { - // 2. vkGetBufferMemoryRequirements. - VkMemoryRequirements vkMemReq = {}; - bool requiresDedicatedAllocation = false; - bool prefersDedicatedAllocation = false; - allocator->GetBufferMemoryRequirements(*pBuffer, vkMemReq, - requiresDedicatedAllocation, prefersDedicatedAllocation); +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedBuffer( + VmaAllocator allocator, + const VkBufferCreateInfo* pBufferCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkBuffer* pBuffer, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && pBuffer && pAllocation); + VMA_DEBUG_LOG("vmaCreateDedicatedBuffer"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - // 2a. Include minAlignment - vkMemReq.alignment = VMA_MAX(vkMemReq.alignment, minAlignment); + VmaAllocationCreateInfo allocCreateInfoCopy = *pAllocationCreateInfo; + allocCreateInfoCopy.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; - // 3. Allocate memory using allocator. - res = allocator->AllocateMemory( - vkMemReq, - requiresDedicatedAllocation, - prefersDedicatedAllocation, - *pBuffer, // dedicatedBuffer - VK_NULL_HANDLE, // dedicatedImage - VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), // dedicatedBufferImageUsage - *pAllocationCreateInfo, - VMA_SUBALLOCATION_TYPE_BUFFER, - 1, // allocationCount - pAllocation); - - if(res >= 0) - { - // 3. Bind buffer with memory. - if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) - { - res = allocator->BindBufferMemory(*pAllocation, 0, *pBuffer, VMA_NULL); - } - if(res >= 0) - { - // All steps succeeded. - #if VMA_STATS_STRING_ENABLED - (*pAllocation)->InitBufferUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5); - #endif - if(pAllocationInfo != VMA_NULL) - { - allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); - } - - return VK_SUCCESS; - } - allocator->FreeMemory( - 1, // allocationCount - pAllocation); - *pAllocation = VK_NULL_HANDLE; - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - return res; + return allocator->CreateBuffer(pBufferCreateInfo, &allocCreateInfoCopy, + pMemoryAllocateNext, // pMemoryAllocateNext + pBuffer, pAllocation, pAllocationInfo); } VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingBuffer( @@ -16534,90 +16891,37 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage( VmaAllocationInfo* pAllocationInfo) { VMA_ASSERT(allocator && pImageCreateInfo && pAllocationCreateInfo && pImage && pAllocation); - VMA_ASSERT((pImageCreateInfo->flags & VK_IMAGE_CREATE_DISJOINT_BIT_COPY) == 0 && "vmaCreateImage() doesn't support disjoint multi-planar images. Please allocate memory for the planes using vmaAllocateMemory() and bind them using vmaBindImageMemory2()."); - - if(pImageCreateInfo->extent.width == 0 || - pImageCreateInfo->extent.height == 0 || - pImageCreateInfo->extent.depth == 0 || - pImageCreateInfo->mipLevels == 0 || - pImageCreateInfo->arrayLayers == 0) - { - return VK_ERROR_INITIALIZATION_FAILED; - } - VMA_DEBUG_LOG("vmaCreateImage"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - VMA_DEBUG_GLOBAL_MUTEX_LOCK + return allocator->CreateImage(pImageCreateInfo, pAllocationCreateInfo, + VMA_NULL, // pMemoryAllocateNext + pImage, pAllocation, pAllocationInfo); +} - *pImage = VK_NULL_HANDLE; - *pAllocation = VK_NULL_HANDLE; +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedImage( + VmaAllocator allocator, + const VkImageCreateInfo* pImageCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkImage* pImage, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + VMA_ASSERT(allocator && pImageCreateInfo && pAllocationCreateInfo && pImage && pAllocation); + VMA_ASSERT((pImageCreateInfo->flags & VK_IMAGE_CREATE_DISJOINT_BIT_COPY) == 0 && + "vmaCreateDedicatedImage() doesn't support disjoint multi-planar images. Please allocate memory for the planes using vmaAllocateMemory() and bind them using vmaBindImageMemory2()."); + VMA_DEBUG_LOG("vmaCreateDedicatedImage"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - // 1. Create VkImage. - VkResult res = (*allocator->GetVulkanFunctions().vkCreateImage)( - allocator->m_hDevice, - pImageCreateInfo, - allocator->GetAllocationCallbacks(), - pImage); - if(res == VK_SUCCESS) - { - VmaSuballocationType suballocType = pImageCreateInfo->tiling == VK_IMAGE_TILING_OPTIMAL ? - VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL : - VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR; + VmaAllocationCreateInfo allocCreateInfoCopy = *pAllocationCreateInfo; + allocCreateInfoCopy.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; - // 2. Allocate memory using allocator. - VkMemoryRequirements vkMemReq = {}; - bool requiresDedicatedAllocation = false; - bool prefersDedicatedAllocation = false; - allocator->GetImageMemoryRequirements(*pImage, vkMemReq, - requiresDedicatedAllocation, prefersDedicatedAllocation); - - res = allocator->AllocateMemory( - vkMemReq, - requiresDedicatedAllocation, - prefersDedicatedAllocation, - VK_NULL_HANDLE, // dedicatedBuffer - *pImage, // dedicatedImage - VmaBufferImageUsage(*pImageCreateInfo), // dedicatedBufferImageUsage - *pAllocationCreateInfo, - suballocType, - 1, // allocationCount - pAllocation); - - if(res == VK_SUCCESS) - { - // 3. Bind image with memory. - if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) - { - res = allocator->BindImageMemory(*pAllocation, 0, *pImage, VMA_NULL); - } - if(res == VK_SUCCESS) - { - // All steps succeeded. - #if VMA_STATS_STRING_ENABLED - (*pAllocation)->InitImageUsage(*pImageCreateInfo); - #endif - if(pAllocationInfo != VMA_NULL) - { - allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); - } - - return VK_SUCCESS; - } - allocator->FreeMemory( - 1, // allocationCount - pAllocation); - *pAllocation = VK_NULL_HANDLE; - (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks()); - *pImage = VK_NULL_HANDLE; - return res; - } - (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks()); - *pImage = VK_NULL_HANDLE; - return res; - } - return res; + return allocator->CreateImage(pImageCreateInfo, &allocCreateInfoCopy, + pMemoryAllocateNext, // pMemoryAllocateNext + pImage, pAllocation, pAllocationInfo); } VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingImage( @@ -16837,7 +17141,20 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle(VmaAllocator VMA_NOT { VMA_ASSERT(allocator && allocation && pHandle); VMA_DEBUG_GLOBAL_MUTEX_LOCK; - return allocation->GetWin32Handle(allocator, hTargetProcess, pHandle); + return allocation->GetWin32Handle(allocator, VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT, hTargetProcess, pHandle); +} +VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle2(VmaAllocator VMA_NOT_NULL allocator, + VmaAllocation VMA_NOT_NULL allocation, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* VMA_NOT_NULL pHandle) +{ + VMA_ASSERT(allocator && allocation && pHandle); + VMA_ASSERT(handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_HEAP_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KHR); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; + return allocation->GetWin32Handle(allocator, handleType, hTargetProcess, pHandle); } #endif // VMA_EXTERNAL_MEMORY_WIN32 #endif // VMA_STATS_STRING_ENABLED @@ -16845,7 +17162,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle(VmaAllocator VMA_NOT #endif // VMA_IMPLEMENTATION /** -\page faq Frequenty asked questions +\page faq Frequently asked questions What is VMA? @@ -17006,7 +17323,6 @@ Among many extensions available for Vulkan, only a few interact with memory mana VMA can automatically take advantage of them. Some of them are: VK_EXT_memory_budget, VK_EXT_memory_priority, VK_KHR_external_memory_win32, and VK_KHR_maintenance* extensions that are later promoted to the new versions of the core Vulkan API. - To use them, it is your responsibility to validate if they are available on the current system and if so, enable them while creating the Vulkan device object. You also need to pass appropriate #VmaAllocatorCreateFlagBits to inform VMA that they are enabled. @@ -17057,8 +17373,8 @@ You can access it in multiple ways: Is it a mature project? -Yes! The library is in development since June 2017, has over 1000 commits, over 400 issue tickets -and pull requests (most of them resolved), and over 70 contributors. +Yes! The library is in development since June 2017, has over 1000 commits, over 500 issue tickets +and pull requests (most of them resolved), and over 80 contributors. It is distributed together with Vulkan SDK. It is used by many software projects, including some large and popular ones like Qt or Blender, as well as some AAA games. @@ -18008,7 +18324,7 @@ Many of the common concerns can be addressed in a different way than using custo - If you want to select specific memory type for your allocation, you can set VmaAllocationCreateInfo::memoryTypeBits to `(1U << myMemoryTypeIndex)` instead. - If you need to create a buffer with certain minimum alignment, you can still do it - using default pools with dedicated function vmaCreateBufferWithAlignment(). + using default pools by specifying VmaAllocationCreateInfo::minAlignment. \section linear_algorithm Linear allocation algorithm @@ -18636,31 +18952,163 @@ individual names to allocations using vmaSetAllocationName(), can greatly aid in \page other_api_interop Interop with other graphics APIs -VMA provides some features that help with interoperability with other graphics APIs, e.g. OpenGL. +VMA provides some features that help with interoperability with other graphics APIs, e.g. OpenGL, Direct3D 11, Direct3D 12. -\section opengl_interop_exporting_memory Exporting memory +\section other_api_interop_exporting_memory Exporting memory -If you want to attach `VkExportMemoryAllocateInfoKHR` or other structure to `pNext` chain of memory allocations made by the library: +On Windows, the VK_KHR_external_memory_win32 device extension allows exporting a Win32 `HANDLE` +of a `VkDeviceMemory` block, to be able to reference the memory on other Vulkan logical devices or instances, +in multiple processes, and/or in multiple APIs. +VMA offers support for it. -You can create \ref custom_memory_pools for such allocations. -Define and fill in your `VkExportMemoryAllocateInfoKHR` structure and attach it to VmaPoolCreateInfo::pMemoryAllocateNext -while creating the custom pool. -Please note that the structure must remain alive and unchanged for the whole lifetime of the #VmaPool, -not only while creating it, as no copy of the structure is made, -but its original pointer is used for each allocation instead. +\subsection other_api_interop_exporting_initialization Initialization + +1) Make sure the extension is defined in the code by including following header before including VMA: + +\code +#include +\endcode + +2) Check if "VK_KHR_external_memory_win32" is available among device extensions. +Enable it when creating the `VkDevice` object. + +3) Enable the usage of this extension in VMA by setting flag #VMA_ALLOCATOR_CREATE_KHR_EXTERNAL_MEMORY_WIN32_BIT +when calling vmaCreateAllocator(). + +4) Make sure that VMA has access to the `vkGetMemoryWin32HandleKHR` function by either enabling `VMA_DYNAMIC_VULKAN_FUNCTIONS` macro +or setting VmaVulkanFunctions::vkGetMemoryWin32HandleKHR explicitly. +For more information, see \ref quick_start_initialization_importing_vulkan_functions. + +\subsection other_api_interop_exporting_preparations Preparations + +You can find example usage among tests, in file "Tests.cpp", function `TestWin32Handles()`. + +To use the extenion, buffers need to be created with `VkExternalMemoryBufferCreateInfoKHR` attached to their `pNext` chain, +and memory allocations need to be made with `VkExportMemoryAllocateInfoKHR` attached to their `pNext` chain. +To make use of them, you need to use \ref custom_memory_pools. Example: + +\code +constexpr VkExternalMemoryHandleTypeFlagsKHR handleType = + VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR; + +// Define an example buffer and allocation parameters. +VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { + VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, + nullptr, + handleType +}; +VkBufferCreateInfo exampleBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; +exampleBufCreateInfo.size = 0x10000; // Doesn't matter here. +exampleBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; +exampleBufCreateInfo.pNext = &externalMemBufCreateInfo; + +VmaAllocationCreateInfo exampleAllocCreateInfo = {}; +exampleAllocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO; + +// Find memory type index to use for the custom pool. +uint32_t memTypeIndex; +VkResult res = vmaFindMemoryTypeIndexForBufferInfo(g_Allocator, + &exampleBufCreateInfo, &exampleAllocCreateInfo, &memTypeIndex); +// Check res... + +// Create a custom pool. +constexpr static VkExportMemoryAllocateInfoKHR exportMemAllocInfo = { + VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR, + nullptr, + handleType +}; +VmaPoolCreateInfo poolCreateInfo = {}; +poolCreateInfo.memoryTypeIndex = memTypeIndex; +poolCreateInfo.pMemoryAllocateNext = (void*)&exportMemAllocInfo; + +VmaPool pool; +res = vmaCreatePool(g_Allocator, &poolCreateInfo, &pool); +// Check res... + +// YOUR OTHER CODE COMES HERE.... + +// At the end, don't forget to destroy it! +vmaDestroyPool(g_Allocator, pool); +\endcode + +Note that the structure passed as VmaPoolCreateInfo::pMemoryAllocateNext must remain alive and unchanged +for the whole lifetime of the custom pool, because it will be used when the pool allocates a new device memory block. +No copy is made internally. This is why variable `exportMemAllocInfo` is defined as static. If you want to export all memory allocated by VMA from certain memory types, -also dedicated allocations or other allocations made from default pools, +including dedicated allocations and allocations made from default pools, an alternative solution is to fill in VmaAllocatorCreateInfo::pTypeExternalMemoryHandleTypes. It should point to an array with `VkExternalMemoryHandleTypeFlagsKHR` to be automatically passed by the library through `VkExportMemoryAllocateInfoKHR` on each allocation made from a specific memory type. -Please note that new versions of the library also support dedicated allocations created in custom pools. - You should not mix these two methods in a way that allows to apply both to the same memory type. Otherwise, `VkExportMemoryAllocateInfoKHR` structure would be attached twice to the `pNext` chain of `VkMemoryAllocateInfo`. +\subsection other_api_interop_exporting_memory_allocation Memory allocation -\section opengl_interop_custom_alignment Custom alignment +Finally, you can create a buffer with an allocation out of the custom pool. +The buffer should use same flags as the sample buffer used to find the memory type. +It should also specify `VkExternalMemoryBufferCreateInfoKHR` in its `pNext` chain. + +\code +VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { + VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, + nullptr, + handleType +}; +VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; +bufCreateInfo.size = // Your desired buffer size. +bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; +bufCreateInfo.pNext = &externalMemBufCreateInfo; + +VmaAllocationCreateInfo allocCreateInfo = {}; +allocCreateInfo.pool = pool; // It is enough to set this one member. + +VkBuffer buf; +VmaAllocation alloc; +res = vmaCreateBuffer(g_Allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, nullptr); +// Check res... + +// YOUR OTHER CODE COMES HERE.... + +// At the end, don't forget to destroy it! +vmaDestroyBuffer(g_Allocator, buf, alloc); +\endcode + +If you need each allocation to have its own device memory block and start at offset 0, you can still do +by using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT flag. It works also with custom pools. + +Alternatively, you can use convenient functions vmaCreateDedicatedBuffer(), vmaCreateDedicatedImage() that +always allocate dedicated memory for the buffer/image created, and also allow specifying custom `pNext` chain +for the `VkMemoryAllocateInfo` structure. + +\subsection other_api_interop_exporting_exporting_win32_handle Exporting Win32 handle + +After the allocation is created, you can acquire a Win32 `HANDLE` to the `VkDeviceMemory` block it belongs to. +VMA function vmaGetMemoryWin32Handle2() is a replacement of the Vulkan function `vkGetMemoryWin32HandleKHR`. + +\code +HANDLE handle; +res = vmaGetMemoryWin32Handle2(g_Allocator, alloc, handleType, nullptr, &handle); +// Check res... + +// YOUR OTHER CODE COMES HERE.... + +// At the end, you must close the handle. +CloseHandle(handle); +\endcode + +Documentation of the VK_KHR_external_memory_win32 extension states that: + +> If handleType is defined as an NT handle, vkGetMemoryWin32HandleKHR must be called no more than once for each valid unique combination of memory and handleType. + +This is ensured automatically inside VMA. +If `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT` is used as the handle type, +or other NT handle types, +the library fetches the handle on first use, remembers it internally, and closes it when the memory block or dedicated allocation is destroyed. +Every time you call vmaGetMemoryWin32Handle2(), VMA calls `DuplicateHandle` and returns a new handle that you need to close. +For further information, please check the documentation of this function. + +\subsection other_api_interop_exporting_custom_alignment Custom alignment Buffers or images exported to a different API like OpenGL may require a different alignment, higher than the one used by the library automatically, queried from functions like `vkGetBufferMemoryRequirements`. @@ -18672,20 +19120,59 @@ to be made out of this pool. The alignment actually used will be the maximum of this member and the alignment returned for the specific buffer or image from a function like `vkGetBufferMemoryRequirements`, which is called by VMA automatically. -If you want to create a buffer with a specific minimum alignment out of default pools, -use special function vmaCreateBufferWithAlignment(), which takes additional parameter `minAlignment`. +If you want to create a buffer/image/allocate memory with a specific minimum alignment out of default pools, +you can use VmaAllocationCreateInfo::minAlignment. Note the problem of alignment affects only resources placed inside bigger `VkDeviceMemory` blocks and not dedicated allocations, as these, by definition, always have alignment = 0 because the resource is bound to the beginning of its dedicated block. You can ensure that an allocation is created as dedicated by using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT. Contrary to Direct3D 12, Vulkan doesn't have a concept of alignment of the entire memory block passed on its allocation. -\section opengl_interop_extended_allocation_information Extended allocation information +\subsection other_api_interop_exporting_extended_allocation_information Extended allocation information If you want to rely on VMA to allocate your buffers and images inside larger memory blocks, but you need to know the size of the entire block and whether the allocation was made with its own dedicated memory, use function vmaGetAllocationInfo2() to retrieve -extended allocation information in structure #VmaAllocationInfo2. +extended allocation information in structure #VmaAllocationInfo2, which provides extra members: +`blockSize` and `dedicatedMemory`. + +\section other_api_interop_importing_memory Importing memory + +Importing external memory requires attaching an extra structure like `VkImportMemoryWin32HandleInfoKHR` +to the `pNext` chain of `VkMemoryAllocateInfo` structure. +VMA offers support for it by providing functions that allocate memory, create a buffer or an image +always with a dedicated `VkDeviceMemory` block and accept custom `pNext` pointer: +vmaAllocateDedicatedMemory(), vmaCreateDedicatedBuffer(), vmaCreateDedicatedImage(). +Example: + +\code +constexpr VkExternalMemoryHandleTypeFlagBits handleType = + VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT; + +VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { + VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR }; +externalMemBufCreateInfo.handleTypes = handleType; + +VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; +bufCreateInfo.pNext = &externalMemBufCreateInfo; // !!! +bufCreateInfo.size = ... +bufCreateInfo.usage = ... + +VkImportMemoryWin32HandleInfoKHR importInfo = { + VK_STRUCTURE_TYPE_IMPORT_MEMORY_WIN32_HANDLE_INFO_KHR }; +importInfo.handleType = handleType; +importInfo.handle = myExternalHandleToImport; + +VmaAllocationCreateInfo allocCreateInfo = {}; +allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO; + +VkBuffer buf = VK_NULL_HANDLE; +VmaAllocation alloc = VK_NULL_HANDLE; +VkResult res = vmaCreateDedicatedBuffer(allocator, &bufCreateInfo, &allocCreateInfo, + &importInfo, // pMemoryAllocateNext !!! + &buf, &alloc, nullptr); +// Check res... +\endcode @@ -18865,11 +19352,9 @@ vmaGetAllocationMemoryProperties(allocator, alloc, &memPropFlags); if(memPropFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) { - // Allocation ended up in a mappable memory and is already mapped - write to it directly. - - // [Executed in runtime]: - memcpy(allocInfo.pMappedData, myData, myDataSize); - result = vmaFlushAllocation(allocator, alloc, 0, VK_WHOLE_SIZE); + // The Allocation ended up in a mappable memory. + // Calling vmaCopyMemoryToAllocation() does vmaMapMemory(), memcpy(), vmaUnmapMemory(), and vmaFlushAllocation(). + result = vmaCopyMemoryToAllocation(allocator, myData, alloc, 0, myDataSize); // Check result... VkBufferMemoryBarrier bufMemBarrier = { VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER }; @@ -18881,6 +19366,7 @@ if(memPropFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) bufMemBarrier.offset = 0; bufMemBarrier.size = VK_WHOLE_SIZE; + // It's important to insert a buffer memory barrier here to ensure writing to the buffer has finished. vkCmdPipelineBarrier(cmdBuf, VK_PIPELINE_STAGE_HOST_BIT, VK_PIPELINE_STAGE_VERTEX_SHADER_BIT, 0, 0, nullptr, 1, &bufMemBarrier, 0, nullptr); } @@ -18903,9 +19389,8 @@ else &stagingBuf, &stagingAlloc, &stagingAllocInfo); // Check result... - // [Executed in runtime]: - memcpy(stagingAllocInfo.pMappedData, myData, myDataSize); - result = vmaFlushAllocation(allocator, stagingAlloc, 0, VK_WHOLE_SIZE); + // Calling vmaCopyMemoryToAllocation() does vmaMapMemory(), memcpy(), vmaUnmapMemory(), and vmaFlushAllocation(). + result = vmaCopyMemoryToAllocation(allocator, myData, stagingAlloc, 0, myDataSize); // Check result... VkBufferMemoryBarrier bufMemBarrier = { VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER }; @@ -18917,6 +19402,7 @@ else bufMemBarrier.offset = 0; bufMemBarrier.size = VK_WHOLE_SIZE; + // Insert a buffer memory barrier to make sure writing to the staging buffer has finished. vkCmdPipelineBarrier(cmdBuf, VK_PIPELINE_STAGE_HOST_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 1, &bufMemBarrier, 0, nullptr); @@ -18937,6 +19423,7 @@ else bufMemBarrier2.offset = 0; bufMemBarrier2.size = VK_WHOLE_SIZE; + // Make sure copying from staging buffer to the actual buffer has finished by inserting a buffer memory barrier. vkCmdPipelineBarrier(cmdBuf, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_VERTEX_SHADER_BIT, 0, 0, nullptr, 1, &bufMemBarrier2, 0, nullptr); } @@ -19241,145 +19728,6 @@ Example use of this extension can be found in the code of the sample and test su accompanying this library. -\page vk_khr_external_memory_win32 VK_KHR_external_memory_win32 - -On Windows, the VK_KHR_external_memory_win32 device extension allows exporting a Win32 `HANDLE` -of a `VkDeviceMemory` block, to be able to reference the memory on other Vulkan logical devices or instances, -in multiple processes, and/or in multiple APIs. -VMA offers support for it. - -\section vk_khr_external_memory_win32_initialization Initialization - -1) Make sure the extension is defined in the code by including following header before including VMA: - -\code -#include -\endcode - -2) Check if "VK_KHR_external_memory_win32" is available among device extensions. -Enable it when creating the `VkDevice` object. - -3) Enable the usage of this extension in VMA by setting flag #VMA_ALLOCATOR_CREATE_KHR_EXTERNAL_MEMORY_WIN32_BIT -when calling vmaCreateAllocator(). - -4) Make sure that VMA has access to the `vkGetMemoryWin32HandleKHR` function by either enabling `VMA_DYNAMIC_VULKAN_FUNCTIONS` macro -or setting VmaVulkanFunctions::vkGetMemoryWin32HandleKHR explicitly. -For more information, see \ref quick_start_initialization_importing_vulkan_functions. - -\section vk_khr_external_memory_win32_preparations Preparations - -You can find example usage among tests, in file "Tests.cpp", function `TestWin32Handles()`. - -To use the extenion, buffers need to be created with `VkExternalMemoryBufferCreateInfoKHR` attached to their `pNext` chain, -and memory allocations need to be made with `VkExportMemoryAllocateInfoKHR` attached to their `pNext` chain. -To make use of them, you need to use \ref custom_memory_pools. Example: - -\code -// Define an example buffer and allocation parameters. -VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { - VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, - nullptr, - VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT -}; -VkBufferCreateInfo exampleBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; -exampleBufCreateInfo.size = 0x10000; // Doesn't matter here. -exampleBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; -exampleBufCreateInfo.pNext = &externalMemBufCreateInfo; - -VmaAllocationCreateInfo exampleAllocCreateInfo = {}; -exampleAllocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO; - -// Find memory type index to use for the custom pool. -uint32_t memTypeIndex; -VkResult res = vmaFindMemoryTypeIndexForBufferInfo(g_Allocator, - &exampleBufCreateInfo, &exampleAllocCreateInfo, &memTypeIndex); -// Check res... - -// Create a custom pool. -constexpr static VkExportMemoryAllocateInfoKHR exportMemAllocInfo = { - VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR, - nullptr, - VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT -}; -VmaPoolCreateInfo poolCreateInfo = {}; -poolCreateInfo.memoryTypeIndex = memTypeIndex; -poolCreateInfo.pMemoryAllocateNext = (void*)&exportMemAllocInfo; - -VmaPool pool; -res = vmaCreatePool(g_Allocator, &poolCreateInfo, &pool); -// Check res... - -// YOUR OTHER CODE COMES HERE.... - -// At the end, don't forget to destroy it! -vmaDestroyPool(g_Allocator, pool); -\endcode - -Note that the structure passed as VmaPoolCreateInfo::pMemoryAllocateNext must remain alive and unchanged -for the whole lifetime of the custom pool, because it will be used when the pool allocates a new device memory block. -No copy is made internally. This is why variable `exportMemAllocInfo` is defined as `static`. - -\section vk_khr_external_memory_win32_memory_allocation Memory allocation - -Finally, you can create a buffer with an allocation out of the custom pool. -The buffer should use same flags as the sample buffer used to find the memory type. -It should also specify `VkExternalMemoryBufferCreateInfoKHR` in its `pNext` chain. - -\code -VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { - VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, - nullptr, - VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT -}; -VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; -bufCreateInfo.size = // Your desired buffer size. -bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; -bufCreateInfo.pNext = &externalMemBufCreateInfo; - -VmaAllocationCreateInfo allocCreateInfo = {}; -allocCreateInfo.pool = pool; // It is enough to set this one member. - -VkBuffer buf; -VmaAllocation alloc; -res = vmaCreateBuffer(g_Allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, nullptr); -// Check res... - -// YOUR OTHER CODE COMES HERE.... - -// At the end, don't forget to destroy it! -vmaDestroyBuffer(g_Allocator, buf, alloc); -\endcode - -If you need each allocation to have its own device memory block and start at offset 0, you can still do -by using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT flag. It works also with custom pools. - -\section vk_khr_external_memory_win32_exporting_win32_handle Exporting Win32 handle - -After the allocation is created, you can acquire a Win32 `HANDLE` to the `VkDeviceMemory` block it belongs to. -VMA function vmaGetMemoryWin32Handle() is a replacement of the Vulkan function `vkGetMemoryWin32HandleKHR`. - -\code -HANDLE handle; -res = vmaGetMemoryWin32Handle(g_Allocator, alloc, nullptr, &handle); -// Check res... - -// YOUR OTHER CODE COMES HERE.... - -// At the end, you must close the handle. -CloseHandle(handle); -\endcode - -Documentation of the VK_KHR_external_memory_win32 extension states that: - -> If handleType is defined as an NT handle, vkGetMemoryWin32HandleKHR must be called no more than once for each valid unique combination of memory and handleType. - -This is ensured automatically inside VMA. -The library fetches the handle on first use, remembers it internally, and closes it when the memory block or dedicated allocation is destroyed. -Every time you call vmaGetMemoryWin32Handle(), VMA calls `DuplicateHandle` and returns a new handle that you need to close. - -For further information, please check documentation of the vmaGetMemoryWin32Handle() function. - - \page enabling_buffer_device_address Enabling buffer device address Device extension VK_KHR_buffer_device_address @@ -19475,9 +19823,6 @@ When using this library, you can meet following types of warnings issued by Vulkan validation layer. They don't necessarily indicate a bug, so you may need to just ignore them. -- *vkBindBufferMemory(): Binding memory to buffer 0xeb8e4 but vkGetBufferMemoryRequirements() has not been called on that buffer.* - - It happens when VK_KHR_dedicated_allocation extension is enabled. - `vkGetBufferMemoryRequirements2KHR` function is used instead, while validation layer seems to be unaware of it. - *Mapping an image with layout VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL can result in undefined behavior if this memory is used by the device. Only GENERAL or PREINITIALIZED should be used.* - It happens when you map a buffer or image, because the library maps entire `VkDeviceMemory` block, where different types of images and buffers may end diff --git a/3rdparty/vulkan/include/vulkan/vk_mem_alloc.h b/3rdparty/vulkan/include/vulkan/vk_mem_alloc.h index 8df03649b5..24792855e2 100644 --- a/3rdparty/vulkan/include/vulkan/vk_mem_alloc.h +++ b/3rdparty/vulkan/include/vulkan/vk_mem_alloc.h @@ -1,5 +1,5 @@ // -// Copyright (c) 2017-2025 Advanced Micro Devices, Inc. All rights reserved. +// Copyright (c) 2017-2026 Advanced Micro Devices, Inc. All rights reserved. // // Permission is hereby granted, free of charge, to any person obtaining a copy // of this software and associated documentation files (the "Software"), to deal @@ -25,11 +25,11 @@ /** \mainpage Vulkan Memory Allocator -Version 3.3.0 +Version 3.4.0-development -Copyright (c) 2017-2025 Advanced Micro Devices, Inc. All rights reserved. \n +Copyright (c) 2017-2026 Advanced Micro Devices, Inc. All rights reserved. \n License: MIT \n -See also: [product page on GPUOpen](https://gpuopen.com/gaming-product/vulkan-memory-allocator/), +See also: [product page on GPUOpen](https://gpuopen.com/vulkan-memory-allocator/), [repository on GitHub](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator) @@ -79,6 +79,8 @@ See also: [product page on GPUOpen](https://gpuopen.com/gaming-product/vulkan-me - [Corruption detection](@ref debugging_memory_usage_corruption_detection) - [Leak detection features](@ref debugging_memory_usage_leak_detection) - \subpage other_api_interop + - [Exporting memory](@ref other_api_interop_exporting_memory) + - [Importing memory](@ref other_api_interop_importing_memory) - \subpage usage_patterns - [GPU-only resource](@ref usage_patterns_gpu_only) - [Staging copy for upload](@ref usage_patterns_staging_copy_upload) @@ -95,7 +97,6 @@ See also: [product page on GPUOpen](https://gpuopen.com/gaming-product/vulkan-me - \subpage enabling_buffer_device_address - \subpage vk_ext_memory_priority - \subpage vk_amd_device_coherent_memory - - \subpage vk_khr_external_memory_win32 - \subpage general_considerations - [Thread safety](@ref general_considerations_thread_safety) - [Versioning and compatibility](@ref general_considerations_versioning_and_compatibility) @@ -132,6 +133,8 @@ extern "C" { #include #endif +#define VMA_VERSION (VK_MAKE_VERSION(3, 4, 0)) + #if !defined(VMA_VULKAN_VERSION) #if defined(VK_VERSION_1_4) #define VMA_VULKAN_VERSION 1004000 @@ -478,7 +481,7 @@ typedef enum VmaAllocatorCreateFlagBits You should set this flag if you found available and enabled this device extension, while creating Vulkan device passed as VmaAllocatorCreateInfo::device. - For more information, see \ref vk_khr_external_memory_win32. + For more information, see \ref other_api_interop. */ VMA_ALLOCATOR_CREATE_KHR_EXTERNAL_MEMORY_WIN32_BIT = 0x00000200, @@ -1335,6 +1338,19 @@ typedef struct VmaAllocationCreateInfo Otherwise, it has the priority of a memory block where it is placed and this variable is ignored. */ float priority; + /** \brief Additional minimum alignment to be used for this allocation. Can be 0. + + Leave 0 (default) not to impose any additional alignment. If not 0, it must be a power of two. + + When creating a buffer or an image, specifying a custom alignment is not needed in most cases, + because Vulkan implementation inspects the `CreateInfo` structure (including intended usage flags) + and returns required alignment through functions like `vkGetBufferMemoryRequirements2`, which VMA automatically + uses and respects. + Extra alignment may be needed in some cases, like when using a buffer for acceleration structure scratch + (`VkPhysicalDeviceAccelerationStructurePropertiesKHR::minAccelerationStructureScratchOffsetAlignment`, see also issue #523) + or when doing interop with OpenGL. + */ + VkDeviceSize minAlignment; } VmaAllocationCreateInfo; /// Describes parameter of created #VmaPool. @@ -1378,8 +1394,14 @@ typedef struct VmaPoolCreateInfo /** \brief Additional minimum alignment to be used for all allocations created from this pool. Can be 0. Leave 0 (default) not to impose any additional alignment. If not 0, it must be a power of two. - It can be useful in cases where alignment returned by Vulkan by functions like `vkGetBufferMemoryRequirements` is not enough, - e.g. when doing interop with OpenGL. + + When creating a buffer or an image, specifying a custom alignment is not needed in most cases, + because Vulkan implementation inspects the `CreateInfo` structure (including intended usage flags) + and returns required alignment through functions like `vkGetBufferMemoryRequirements2`, which VMA automatically + uses and respects. + Extra alignment may be needed in some cases, like when using a buffer for acceleration structure scratch + (`VkPhysicalDeviceAccelerationStructurePropertiesKHR::minAccelerationStructureScratchOffsetAlignment`, see also issue #523) + or when doing interop with OpenGL. */ VkDeviceSize minAllocationAlignment; /** \brief Additional `pNext` chain to be attached to `VkMemoryAllocateInfo` used for every allocation made by this pool. Optional. @@ -1820,20 +1842,20 @@ VMA_CALL_PRE void VMA_CALL_POST vmaGetHeapBudgets( */ /** -\brief Helps to find memoryTypeIndex, given memoryTypeBits and VmaAllocationCreateInfo. +\brief Helps to find `memoryTypeIndex`, given `memoryTypeBits` and #VmaAllocationCreateInfo. This algorithm tries to find a memory type that: -- Is allowed by memoryTypeBits. -- Contains all the flags from pAllocationCreateInfo->requiredFlags. +- Is allowed by `memoryTypeBits`. +- Contains all the flags from `pAllocationCreateInfo->requiredFlags`. - Matches intended usage. -- Has as many flags from pAllocationCreateInfo->preferredFlags as possible. +- Has as many flags from `pAllocationCreateInfo->preferredFlags` as possible. -\return Returns VK_ERROR_FEATURE_NOT_PRESENT if not found. Receiving such result +\return Returns `VK_ERROR_FEATURE_NOT_PRESENT` if not found. Receiving such result from this function or any other allocating function probably means that your device doesn't support any memory type with requested features for the specific type of resource you want to use it for. Please check parameters of your -resource, like image layout (OPTIMAL versus LINEAR) or mip level count. +resource, like image layout (`OPTIMAL` versus `LINEAR`) or mip level count. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndex( VmaAllocator VMA_NOT_NULL allocator, @@ -1842,10 +1864,10 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndex( uint32_t* VMA_NOT_NULL pMemoryTypeIndex); /** -\brief Helps to find memoryTypeIndex, given VkBufferCreateInfo and VmaAllocationCreateInfo. +\brief Helps to find `memoryTypeIndex`, given `VkBufferCreateInfo` and #VmaAllocationCreateInfo. It can be useful e.g. to determine value to be used as VmaPoolCreateInfo::memoryTypeIndex. -It internally creates a temporary, dummy buffer that never has memory bound. +It may need to internally create a temporary, dummy buffer that never has memory bound. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( VmaAllocator VMA_NOT_NULL allocator, @@ -1854,10 +1876,10 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( uint32_t* VMA_NOT_NULL pMemoryTypeIndex); /** -\brief Helps to find memoryTypeIndex, given VkImageCreateInfo and VmaAllocationCreateInfo. +\brief Helps to find `memoryTypeIndex`, given `VkImageCreateInfo` and #VmaAllocationCreateInfo. It can be useful e.g. to determine value to be used as VmaPoolCreateInfo::memoryTypeIndex. -It internally creates a temporary, dummy image that never has memory bound. +It may need to internally create a temporary, dummy image that never has memory bound. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo( VmaAllocator VMA_NOT_NULL allocator, @@ -1962,16 +1984,25 @@ VMA_CALL_PRE void VMA_CALL_POST vmaSetPoolName( /** \brief General purpose memory allocation. -\param allocator -\param pVkMemoryRequirements -\param pCreateInfo +\param allocator The main allocator object. +\param pVkMemoryRequirements Requirements for the allocated memory. +\param pCreateInfo Allocation creation parameters. \param[out] pAllocation Handle to allocated memory. -\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo(). +\param[out] pAllocationInfo Optional, can be null. Information about allocated memory. It can be also fetched later using vmaGetAllocationInfo(). -You should free the memory using vmaFreeMemory() or vmaFreeMemoryPages(). +The function creates a #VmaAllocation object without creating a buffer or an image together with it. -It is recommended to use vmaAllocateMemoryForBuffer(), vmaAllocateMemoryForImage(), -vmaCreateBuffer(), vmaCreateImage() instead whenever possible. +- It is recommended to use vmaAllocateMemoryForBuffer(), vmaAllocateMemoryForImage(), + vmaCreateBuffer(), vmaCreateImage() instead whenever possible. +- You can also create a buffer or an image later in an existing allocation using + vmaCreateAliasingBuffer2(), vmaCreateAliasingImage2(). +- You can also create a buffer or an image on your own and bind it to an existing allocation + using vmaBindBufferMemory2(), vmaBindImageMemory2(). + +You must free the returned allocation object using vmaFreeMemory() or vmaFreeMemoryPages(). + +There is also extended version of this function: vmaAllocateDedicatedMemory() +that offers additional parameter `pMemoryAllocateNext`. */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory( VmaAllocator VMA_NOT_NULL allocator, @@ -1980,6 +2011,22 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory( VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); +/** \brief General purpose allocation of a dedicated memory. + +This function is similar vmaAllocateMemory(), but +it always allocates dedicated memory - flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is implied. +It offers additional parameter `pMemoryAllocateNext`, +which can be used to attach `pNext` chain to the `VkMemoryAllocateInfo` structure. +It can be useful for importing external memory. For more information, see \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateDedicatedMemory( + VmaAllocator VMA_NOT_NULL allocator, + const VkMemoryRequirements* VMA_NOT_NULL pVkMemoryRequirements, + const VmaAllocationCreateInfo* VMA_NOT_NULL pCreateInfo, + void* VMA_NULLABLE VMA_EXTENDS_VK_STRUCT(VkMemoryAllocateInfo) pMemoryAllocateNext, + VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, + VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); + /** \brief General purpose memory allocation for multiple allocation objects at once. \param allocator Allocator object. @@ -2142,13 +2189,16 @@ VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocationMemoryProperties( /** \brief Given an allocation, returns Win32 handle that may be imported by other processes or APIs. -\param hTargetProcess Must be a valid handle to target process or null. If it's null, the function returns +\param allocator The main allocator object. +\param allocation Allocation. +\param hTargetProcess A valid handle to target process or null. If it's null, the function returns handle for the current process. \param[out] pHandle Output parameter that returns the handle. The function fills `pHandle` with handle that can be used in target process. The handle is fetched using function `vkGetMemoryWin32HandleKHR`. -When no longer needed, you must close it using: + +Each call to this function creates a new handle that must be closed using: \code CloseHandle(handle); @@ -2161,14 +2211,76 @@ Note the handle is returned for the entire `VkDeviceMemory` block that the alloc If the allocation is sub-allocated from a larger block, you may need to consider the offset of the allocation (VmaAllocationInfo::offset). +This function always uses `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT`. +An extended version of this function is available as vmaGetMemoryWin32Handle2() +that allows using other handle type. + +This function is available compile-time only when VK_KHR_external_memory_win32 extension is available. +It can be manually disabled by predefining `VMA_EXTERNAL_MEMORY_WIN32=0` macro. + If the function fails with `VK_ERROR_FEATURE_NOT_PRESENT` error code, please double-check -that VmaVulkanFunctions::vkGetMemoryWin32HandleKHR function pointer is set, e.g. either by using `VMA_DYNAMIC_VULKAN_FUNCTIONS` +that VmaVulkanFunctions::vkGetMemoryWin32HandleKHR function pointer is set, e.g. +either by using macro `VMA_DYNAMIC_VULKAN_FUNCTIONS` or by manually passing it through VmaAllocatorCreateInfo::pVulkanFunctions. -For more information, see chapter \ref vk_khr_external_memory_win32. +For more information, see chapter \ref other_api_interop. */ -VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle(VmaAllocator VMA_NOT_NULL allocator, - VmaAllocation VMA_NOT_NULL allocation, HANDLE hTargetProcess, HANDLE* VMA_NOT_NULL pHandle); +VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle( + VmaAllocator VMA_NOT_NULL allocator, + VmaAllocation VMA_NOT_NULL allocation, + HANDLE hTargetProcess, + HANDLE* VMA_NOT_NULL pHandle); + +/** +\brief Given an allocation, returns Win32 handle that may be imported by other processes or APIs. + +\param allocator The main allocator object. +\param allocation Allocation. +\param handleType Type of handle to be exported. It should be one of: + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_HEAP_BIT_KHR` + - `VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KHR` +\param hTargetProcess A valid handle to target process or null. If it's null, the function returns + handle for the current process. +\param[out] pHandle Output parameter that returns the handle. + +The function fills `pHandle` with handle that can be used in target process. +The handle is fetched using function `vkGetMemoryWin32HandleKHR`. + +If `handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR`, +or other NT handle types, +each call to this function creates a new handle that must be closed using: + +\code +CloseHandle(handle); +\endcode + +You can close it any time, before or after destroying the allocation object. +It is reference-counted internally by Windows. + +Note the handle is returned for the entire `VkDeviceMemory` block that the allocation belongs to. +If the allocation is sub-allocated from a larger block, you may need to consider the offset of the allocation +(VmaAllocationInfo::offset). + +This function is available compile-time only when VK_KHR_external_memory_win32 extension is available. +It can be manually disabled by predefining `VMA_EXTERNAL_MEMORY_WIN32=0` macro. + +If the function fails with `VK_ERROR_FEATURE_NOT_PRESENT` error code, please double-check +that VmaVulkanFunctions::vkGetMemoryWin32HandleKHR function pointer is set, e.g. +either by using macro `VMA_DYNAMIC_VULKAN_FUNCTIONS` +or by manually passing it through VmaAllocatorCreateInfo::pVulkanFunctions. + +For more information, see chapter \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle2( + VmaAllocator VMA_NOT_NULL allocator, + VmaAllocation VMA_NOT_NULL allocation, + VkExternalMemoryHandleTypeFlagBits handleType, + HANDLE hTargetProcess, + HANDLE* VMA_NOT_NULL pHandle); #endif // VMA_EXTERNAL_MEMORY_WIN32 /** \brief Maps memory represented by given allocation and returns pointer to it. @@ -2541,12 +2653,13 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindImageMemory2( /** \brief Creates a new `VkBuffer`, allocates and binds memory for it. -\param allocator -\param pBufferCreateInfo -\param pAllocationCreateInfo +\param allocator The main allocator object. +\param pBufferCreateInfo Buffer creation parameters. +\param pAllocationCreateInfo Allocation creation parameters. \param[out] pBuffer Buffer that was created. \param[out] pAllocation Allocation that was created. -\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo(). +\param[out] pAllocationInfo Optional, can be null. Information about allocated memory. + It can be also fetched later using vmaGetAllocationInfo(). This function automatically: @@ -2555,14 +2668,14 @@ This function automatically: -# Binds the buffer with the memory. If any of these operations fail, buffer and allocation are not created, -returned value is negative error code, `*pBuffer` and `*pAllocation` are null. +returned value is negative error code, `*pBuffer` and `*pAllocation` are returned as null. If the function succeeded, you must destroy both buffer and allocation when you no longer need them using either convenience function vmaDestroyBuffer() or separately, using `vkDestroyBuffer()` and vmaFreeMemory(). -If #VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT flag was used, -VK_KHR_dedicated_allocation extension is used internally to query driver whether +If VK_KHR_dedicated_allocation extenion or Vulkan version >= 1.1 is used, +the function queries the driver whether it requires or prefers the new buffer to have dedicated allocation. If yes, and if dedicated allocation is possible (#VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT is not used), it creates dedicated @@ -2572,6 +2685,9 @@ allocation for this buffer, just like when using \note This function creates a new `VkBuffer`. Sub-allocation of parts of one large buffer, although recommended as a good practice, is out of scope of this library and could be implemented by the user as a higher-level logic on top of VMA. + +There is also an extended versions of this function available with additional parameter `pMemoryAllocateNext` - +see vmaCreateDedicatedBuffer(). */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer( VmaAllocator VMA_NOT_NULL allocator, @@ -2586,6 +2702,10 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer( Similar to vmaCreateBuffer() but provides additional parameter `minAlignment` which allows to specify custom, minimum alignment to be used when placing the buffer inside a larger memory block, which may be needed e.g. for interop with OpenGL. + +\deprecated +This function in obsolete since new VmaAllocationCreateInfo::minAlignment member allows specifying custom +alignment while using any allocation function, like the standard vmaCreateBuffer(). */ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( VmaAllocator VMA_NOT_NULL allocator, @@ -2596,6 +2716,23 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); +/** \brief Creates a dedicated buffer while offering extra parameter `pMemoryAllocateNext`. + +This function is similar vmaCreateBuffer(), but +it always allocates dedicated memory for the buffer - flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is implied. +It offers additional parameter `pMemoryAllocateNext`, +which can be used to attach `pNext` chain to the `VkMemoryAllocateInfo` structure. +It can be useful for importing external memory. For more information, see \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedBuffer( + VmaAllocator VMA_NOT_NULL allocator, + const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo, + const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo, + void* VMA_NULLABLE VMA_EXTENDS_VK_STRUCT(VkMemoryAllocateInfo) pMemoryAllocateNext, + VkBuffer VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pBuffer, + VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, + VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); + /** \brief Creates a new `VkBuffer`, binds already created memory for it. \param allocator @@ -2668,7 +2805,11 @@ VMA_CALL_PRE void VMA_CALL_POST vmaDestroyBuffer( VkBuffer VMA_NULLABLE_NON_DISPATCHABLE buffer, VmaAllocation VMA_NULLABLE allocation); -/// Function similar to vmaCreateBuffer(). +/** \brief Function similar to vmaCreateBuffer() but for images. + +There is also an extended version of this function available: vmaCreateDedicatedImage() +which offers additional parameter `pMemoryAllocateNext`. +*/ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage( VmaAllocator VMA_NOT_NULL allocator, const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo, @@ -2677,6 +2818,23 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage( VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); +/** \brief Function similar to vmaCreateDedicatedBuffer() but for images. + +This function is similar vmaCreateImage(), but +it always allocates dedicated memory for the image - flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is implied. +It offers additional parameter `pMemoryAllocateNext`, +which can be used to attach `pNext` chain to the `VkMemoryAllocateInfo` structure. +It can be useful for importing external memory. For more information, see \ref other_api_interop. +*/ +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedImage( + VmaAllocator VMA_NOT_NULL allocator, + const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo, + const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo, + void* VMA_NULLABLE VMA_EXTENDS_VK_STRUCT(VkMemoryAllocateInfo) pMemoryAllocateNext, + VkImage VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pImage, + VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation, + VmaAllocationInfo* VMA_NULLABLE pAllocationInfo); + /// Function similar to vmaCreateAliasingBuffer() but for images. VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingImage( VmaAllocator VMA_NOT_NULL allocator, @@ -3017,7 +3175,9 @@ remove them if not needed. #if defined(__ANDROID_API__) && (__ANDROID_API__ < 16) #include -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace +{ +void* vma_aligned_alloc(size_t alignment, size_t size) { // alignment must be >= sizeof(void*) if(alignment < sizeof(void*)) @@ -3027,6 +3187,7 @@ static void* vma_aligned_alloc(size_t alignment, size_t size) return memalign(alignment, size); } +} // namespace #elif defined(__APPLE__) || defined(__ANDROID__) || (defined(__linux__) && defined(__GLIBCXX__) && !defined(_GLIBCXX_HAVE_ALIGNED_ALLOC)) #include @@ -3034,7 +3195,9 @@ static void* vma_aligned_alloc(size_t alignment, size_t size) #include #endif -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace +{ +void* vma_aligned_alloc(size_t alignment, size_t size) { // Unfortunately, aligned_alloc causes VMA to crash due to it returning null pointers. (At least under 11.4) // Therefore, for now disable this specific exception until a proper solution is found. @@ -3062,35 +3225,46 @@ static void* vma_aligned_alloc(size_t alignment, size_t size) return pointer; return VMA_NULL; } +} // namespace #elif defined(_WIN32) -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace { +void* vma_aligned_alloc(size_t alignment, size_t size) { return _aligned_malloc(size, alignment); } +} // namespace #elif __cplusplus >= 201703L || _MSVC_LANG >= 201703L // C++17 -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace { +void* vma_aligned_alloc(size_t alignment, size_t size) { return aligned_alloc(alignment, size); } +} // namespace #else -static void* vma_aligned_alloc(size_t alignment, size_t size) +namespace +{ +void* vma_aligned_alloc(size_t alignment, size_t size) { VMA_ASSERT(0 && "Could not implement aligned_alloc automatically. Please enable C++17 or later in your compiler or provide custom implementation of macro VMA_SYSTEM_ALIGNED_MALLOC (and VMA_SYSTEM_ALIGNED_FREE if needed) using the API of your system."); return VMA_NULL; } +} // namespace #endif +namespace +{ #if defined(_WIN32) -static void vma_aligned_free(void* ptr) +void vma_aligned_free(void* ptr) { _aligned_free(ptr); } #else -static void vma_aligned_free(void* VMA_NULLABLE ptr) +void vma_aligned_free(void* VMA_NULLABLE ptr) { free(ptr); } #endif +} // namespace #ifndef VMA_ALIGN_OF #define VMA_ALIGN_OF(type) (alignof(type)) @@ -3171,18 +3345,20 @@ static void vma_aligned_free(void* VMA_NULLABLE ptr) // Define this macro to 1 to enable functions: vmaBuildStatsString, vmaFreeStatsString. #if VMA_STATS_STRING_ENABLED - static inline void VmaUint32ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint32_t num) +namespace { + inline void VmaUint32ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint32_t num) { snprintf(outStr, strLen, "%" PRIu32, num); } - static inline void VmaUint64ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint64_t num) + inline void VmaUint64ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint64_t num) { snprintf(outStr, strLen, "%" PRIu64, num); } - static inline void VmaPtrToStr(char* VMA_NOT_NULL outStr, size_t strLen, const void* ptr) + inline void VmaPtrToStr(char* VMA_NOT_NULL outStr, size_t strLen, const void* ptr) { snprintf(outStr, strLen, "%p", ptr); } +} // namespace #endif #ifndef VMA_MUTEX @@ -3377,31 +3553,33 @@ END OF CONFIGURATION */ #endif // _VMA_CONFIGURATION - -static const uint8_t VMA_ALLOCATION_FILL_PATTERN_CREATED = 0xDC; -static const uint8_t VMA_ALLOCATION_FILL_PATTERN_DESTROYED = 0xEF; +namespace +{ +constexpr uint8_t VMA_ALLOCATION_FILL_PATTERN_CREATED = 0xDC; +constexpr uint8_t VMA_ALLOCATION_FILL_PATTERN_DESTROYED = 0xEF; // Decimal 2139416166, float NaN, little-endian binary 66 E6 84 7F. -static const uint32_t VMA_CORRUPTION_DETECTION_MAGIC_VALUE = 0x7F84E666; +constexpr uint32_t VMA_CORRUPTION_DETECTION_MAGIC_VALUE = 0x7F84E666; // Copy of some Vulkan definitions so we don't need to check their existence just to handle few constants. -static const uint32_t VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY = 0x00000040; -static const uint32_t VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY = 0x00000080; -static const uint32_t VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY = 0x00020000; -static const uint32_t VK_IMAGE_CREATE_DISJOINT_BIT_COPY = 0x00000200; -static const int32_t VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT_COPY = 1000158000; -static const uint32_t VMA_ALLOCATION_INTERNAL_STRATEGY_MIN_OFFSET = 0x10000000U; -static const uint32_t VMA_ALLOCATION_TRY_COUNT = 32; -static const uint32_t VMA_VENDOR_ID_AMD = 4098; +constexpr uint32_t VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY = 0x00000040; +constexpr uint32_t VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY = 0x00000080; +constexpr uint32_t VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY = 0x00020000; +constexpr uint32_t VK_IMAGE_CREATE_DISJOINT_BIT_COPY = 0x00000200; +constexpr int32_t VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT_COPY = 1000158000; +constexpr uint32_t VMA_ALLOCATION_INTERNAL_STRATEGY_MIN_OFFSET = 0x10000000U; +constexpr uint32_t VMA_ALLOCATION_TRY_COUNT = 32; +constexpr uint32_t VMA_VENDOR_ID_AMD = 4098; // This one is tricky. Vulkan specification defines this code as available since // Vulkan 1.0, but doesn't actually define it in Vulkan SDK earlier than 1.2.131. // See pull request #207. #define VK_ERROR_UNKNOWN_COPY ((VkResult)-13) +} // namespace #if VMA_STATS_STRING_ENABLED // Correspond to values of enum VmaSuballocationType. -static const char* const VMA_SUBALLOCATION_TYPE_NAMES[] = +const char* const VMA_SUBALLOCATION_TYPE_NAMES[] = { "FREE", "UNKNOWN", @@ -3412,7 +3590,7 @@ static const char* const VMA_SUBALLOCATION_TYPE_NAMES[] = }; #endif -static const VkAllocationCallbacks VmaEmptyAllocationCallbacks = +const VkAllocationCallbacks VmaEmptyAllocationCallbacks = { VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL }; @@ -3451,6 +3629,8 @@ enum class VmaAllocationRequestType // Opaque handle used by allocation algorithms to identify single allocation in any conforming way. VK_DEFINE_NON_DISPATCHABLE_HANDLE(VmaAllocHandle); +struct VmaBufferImageUsage; + struct VmaMutexLock; struct VmaMutexLockRead; struct VmaMutexLockWrite; @@ -3515,419 +3695,12 @@ class VmaAllocationObjectAllocator; #endif // _VMA_FORWARD_DECLARATIONS +#ifndef _VMA_BUFFER_IMAGE_USAGE -#ifndef _VMA_FUNCTIONS - -/* -Returns number of bits set to 1 in (v). - -On specific platforms and compilers you can use intrinsics like: - -Visual Studio: - return __popcnt(v); -GCC, Clang: - return static_cast(__builtin_popcount(v)); - -Define macro VMA_COUNT_BITS_SET to provide your optimized implementation. -But you need to check in runtime whether user's CPU supports these, as some old processors don't. -*/ -static inline uint32_t VmaCountBitsSet(uint32_t v) -{ -#if VMA_CPP20 - return std::popcount(v); -#else - uint32_t c = v - ((v >> 1) & 0x55555555); - c = ((c >> 2) & 0x33333333) + (c & 0x33333333); - c = ((c >> 4) + c) & 0x0F0F0F0F; - c = ((c >> 8) + c) & 0x00FF00FF; - c = ((c >> 16) + c) & 0x0000FFFF; - return c; -#endif -} - -static inline uint8_t VmaBitScanLSB(uint64_t mask) -{ -#if defined(_MSC_VER) && defined(_WIN64) - unsigned long pos; - if (_BitScanForward64(&pos, mask)) - return static_cast(pos); - return UINT8_MAX; -#elif VMA_CPP20 - if(mask != 0) - return static_cast(std::countr_zero(mask)); - return UINT8_MAX; -#elif defined __GNUC__ || defined __clang__ - return static_cast(__builtin_ffsll(mask)) - 1U; -#else - uint8_t pos = 0; - uint64_t bit = 1; - do - { - if (mask & bit) - return pos; - bit <<= 1; - } while (pos++ < 63); - return UINT8_MAX; -#endif -} - -static inline uint8_t VmaBitScanLSB(uint32_t mask) -{ -#ifdef _MSC_VER - unsigned long pos; - if (_BitScanForward(&pos, mask)) - return static_cast(pos); - return UINT8_MAX; -#elif VMA_CPP20 - if(mask != 0) - return static_cast(std::countr_zero(mask)); - return UINT8_MAX; -#elif defined __GNUC__ || defined __clang__ - return static_cast(__builtin_ffs(mask)) - 1U; -#else - uint8_t pos = 0; - uint32_t bit = 1; - do - { - if (mask & bit) - return pos; - bit <<= 1; - } while (pos++ < 31); - return UINT8_MAX; -#endif -} - -static inline uint8_t VmaBitScanMSB(uint64_t mask) -{ -#if defined(_MSC_VER) && defined(_WIN64) - unsigned long pos; - if (_BitScanReverse64(&pos, mask)) - return static_cast(pos); -#elif VMA_CPP20 - if(mask != 0) - return 63 - static_cast(std::countl_zero(mask)); -#elif defined __GNUC__ || defined __clang__ - if (mask != 0) - return 63 - static_cast(__builtin_clzll(mask)); -#else - uint8_t pos = 63; - uint64_t bit = 1ULL << 63; - do - { - if (mask & bit) - return pos; - bit >>= 1; - } while (pos-- > 0); -#endif - return UINT8_MAX; -} - -static inline uint8_t VmaBitScanMSB(uint32_t mask) -{ -#ifdef _MSC_VER - unsigned long pos; - if (_BitScanReverse(&pos, mask)) - return static_cast(pos); -#elif VMA_CPP20 - if(mask != 0) - return 31 - static_cast(std::countl_zero(mask)); -#elif defined __GNUC__ || defined __clang__ - if (mask != 0) - return 31 - static_cast(__builtin_clz(mask)); -#else - uint8_t pos = 31; - uint32_t bit = 1UL << 31; - do - { - if (mask & bit) - return pos; - bit >>= 1; - } while (pos-- > 0); -#endif - return UINT8_MAX; -} - -/* -Returns true if given number is a power of two. -T must be unsigned integer number or signed integer but always nonnegative. -For 0 returns true. -*/ -template -inline bool VmaIsPow2(T x) -{ - return (x & (x - 1)) == 0; -} - -// Aligns given value up to nearest multiply of align value. For example: VmaAlignUp(11, 8) = 16. -// Use types like uint32_t, uint64_t as T. -template -static inline T VmaAlignUp(T val, T alignment) -{ - VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); - return (val + alignment - 1) & ~(alignment - 1); -} - -// Aligns given value down to nearest multiply of align value. For example: VmaAlignDown(11, 8) = 8. -// Use types like uint32_t, uint64_t as T. -template -static inline T VmaAlignDown(T val, T alignment) -{ - VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); - return val & ~(alignment - 1); -} - -// Division with mathematical rounding to nearest number. -template -static inline T VmaRoundDiv(T x, T y) -{ - return (x + (y / (T)2)) / y; -} - -// Divide by 'y' and round up to nearest integer. -template -static inline T VmaDivideRoundingUp(T x, T y) -{ - return (x + y - (T)1) / y; -} - -// Returns smallest power of 2 greater or equal to v. -static inline uint32_t VmaNextPow2(uint32_t v) -{ - v--; - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v++; - return v; -} - -static inline uint64_t VmaNextPow2(uint64_t v) -{ - v--; - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v |= v >> 32; - v++; - return v; -} - -// Returns largest power of 2 less or equal to v. -static inline uint32_t VmaPrevPow2(uint32_t v) -{ - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v = v ^ (v >> 1); - return v; -} - -static inline uint64_t VmaPrevPow2(uint64_t v) -{ - v |= v >> 1; - v |= v >> 2; - v |= v >> 4; - v |= v >> 8; - v |= v >> 16; - v |= v >> 32; - v = v ^ (v >> 1); - return v; -} - -static inline bool VmaStrIsEmpty(const char* pStr) -{ - return pStr == VMA_NULL || *pStr == '\0'; -} - -/* -Returns true if two memory blocks occupy overlapping pages. -ResourceA must be in less memory offset than ResourceB. - -Algorithm is based on "Vulkan 1.0.39 - A Specification (with all registered Vulkan extensions)" -chapter 11.6 "Resource Memory Association", paragraph "Buffer-Image Granularity". -*/ -static inline bool VmaBlocksOnSamePage( - VkDeviceSize resourceAOffset, - VkDeviceSize resourceASize, - VkDeviceSize resourceBOffset, - VkDeviceSize pageSize) -{ - VMA_ASSERT(resourceAOffset + resourceASize <= resourceBOffset && resourceASize > 0 && pageSize > 0); - VkDeviceSize resourceAEnd = resourceAOffset + resourceASize - 1; - VkDeviceSize resourceAEndPage = resourceAEnd & ~(pageSize - 1); - VkDeviceSize resourceBStart = resourceBOffset; - VkDeviceSize resourceBStartPage = resourceBStart & ~(pageSize - 1); - return resourceAEndPage == resourceBStartPage; -} - -/* -Returns true if given suballocation types could conflict and must respect -VkPhysicalDeviceLimits::bufferImageGranularity. They conflict if one is buffer -or linear image and another one is optimal image. If type is unknown, behave -conservatively. -*/ -static inline bool VmaIsBufferImageGranularityConflict( - VmaSuballocationType suballocType1, - VmaSuballocationType suballocType2) -{ - if (suballocType1 > suballocType2) - { - std::swap(suballocType1, suballocType2); - } - - switch (suballocType1) - { - case VMA_SUBALLOCATION_TYPE_FREE: - return false; - case VMA_SUBALLOCATION_TYPE_UNKNOWN: - return true; - case VMA_SUBALLOCATION_TYPE_BUFFER: - return - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; - case VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN: - return - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR || - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; - case VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR: - return - suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; - case VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL: - return false; - default: - VMA_ASSERT(0); - return true; - } -} - -static void VmaWriteMagicValue(void* pData, VkDeviceSize offset) -{ -#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION - uint32_t* pDst = (uint32_t*)((char*)pData + offset); - const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); - for (size_t i = 0; i < numberCount; ++i, ++pDst) - { - *pDst = VMA_CORRUPTION_DETECTION_MAGIC_VALUE; - } -#else - // no-op -#endif -} - -static bool VmaValidateMagicValue(const void* pData, VkDeviceSize offset) -{ -#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION - const uint32_t* pSrc = (const uint32_t*)((const char*)pData + offset); - const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); - for (size_t i = 0; i < numberCount; ++i, ++pSrc) - { - if (*pSrc != VMA_CORRUPTION_DETECTION_MAGIC_VALUE) - { - return false; - } - } -#endif - return true; -} - -/* -Fills structure with parameters of an example buffer to be used for transfers -during GPU memory defragmentation. -*/ -static void VmaFillGpuDefragmentationBufferCreateInfo(VkBufferCreateInfo& outBufCreateInfo) -{ - memset(&outBufCreateInfo, 0, sizeof(outBufCreateInfo)); - outBufCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO; - outBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; - outBufCreateInfo.size = (VkDeviceSize)VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE; // Example size. -} - - -/* -Performs binary search and returns iterator to first element that is greater or -equal to (key), according to comparison (cmp). - -Cmp should return true if first argument is less than second argument. - -Returned value is the found element, if present in the collection or place where -new element with value (key) should be inserted. -*/ -template -static IterT VmaBinaryFindFirstNotLess(IterT beg, IterT end, const KeyT& key, const CmpLess& cmp) -{ - size_t down = 0; - size_t up = size_t(end - beg); - while (down < up) - { - const size_t mid = down + (up - down) / 2; // Overflow-safe midpoint calculation - if (cmp(*(beg + mid), key)) - { - down = mid + 1; - } - else - { - up = mid; - } - } - return beg + down; -} - -template -IterT VmaBinaryFindSorted(const IterT& beg, const IterT& end, const KeyT& value, const CmpLess& cmp) -{ - IterT it = VmaBinaryFindFirstNotLess( - beg, end, value, cmp); - if (it == end || - (!cmp(*it, value) && !cmp(value, *it))) - { - return it; - } - return end; -} - -/* -Returns true if all pointers in the array are not-null and unique. -Warning! O(n^2) complexity. Use only inside VMA_HEAVY_ASSERT. -T must be pointer type, e.g. VmaAllocation, VmaPool. -*/ -template -static bool VmaValidatePointerArray(uint32_t count, const T* arr) -{ - for (uint32_t i = 0; i < count; ++i) - { - const T iPtr = arr[i]; - if (iPtr == VMA_NULL) - { - return false; - } - for (uint32_t j = i + 1; j < count; ++j) - { - if (iPtr == arr[j]) - { - return false; - } - } - } - return true; -} - -template -static inline void VmaPnextChainPushFront(MainT* mainStruct, NewT* newStruct) -{ - newStruct->pNext = mainStruct->pNext; - mainStruct->pNext = newStruct; -} // Finds structure with s->sType == sType in mainStruct->pNext chain. // Returns pointer to it. If not found, returns null. template -static inline const FindT* VmaPnextChainFind(const MainT* mainStruct, VkStructureType sType) +inline const FindT* VmaPnextChainFind(const MainT* mainStruct, VkStructureType sType) { for(const VkBaseInStructure* s = (const VkBaseInStructure*)mainStruct->pNext; s != VMA_NULL; s = s->pNext) @@ -4000,9 +3773,423 @@ VmaBufferImageUsage::VmaBufferImageUsage(const VkImageCreateInfo &createInfo) // VkImageUsageFlags2CreateInfoKHR, like the one for buffers... } +#endif // _VMA_BUFFER_IMAGE_USAGE + +#ifndef _VMA_FUNCTIONS + +namespace +{ + +/* +Returns number of bits set to 1 in (v). + +On specific platforms and compilers you can use intrinsics like: + +Visual Studio: + return __popcnt(v); +GCC, Clang: + return static_cast(__builtin_popcount(v)); + +Define macro VMA_COUNT_BITS_SET to provide your optimized implementation. +But you need to check in runtime whether user's CPU supports these, as some old processors don't. +*/ +inline uint32_t VmaCountBitsSet(uint32_t v) +{ +#if VMA_CPP20 + return std::popcount(v); +#else + uint32_t c = v - ((v >> 1) & 0x55555555); + c = ((c >> 2) & 0x33333333) + (c & 0x33333333); + c = ((c >> 4) + c) & 0x0F0F0F0F; + c = ((c >> 8) + c) & 0x00FF00FF; + c = ((c >> 16) + c) & 0x0000FFFF; + return c; +#endif +} + +inline uint8_t VmaBitScanLSB(uint64_t mask) +{ +#if defined(_MSC_VER) && defined(_WIN64) + unsigned long pos; + if (_BitScanForward64(&pos, mask)) + return static_cast(pos); + return UINT8_MAX; +#elif VMA_CPP20 + if(mask != 0) + return static_cast(std::countr_zero(mask)); + return UINT8_MAX; +#elif defined __GNUC__ || defined __clang__ + return static_cast(__builtin_ffsll(mask)) - 1U; +#else + uint8_t pos = 0; + uint64_t bit = 1; + do + { + if (mask & bit) + return pos; + bit <<= 1; + } while (pos++ < 63); + return UINT8_MAX; +#endif +} + +inline uint8_t VmaBitScanLSB(uint32_t mask) +{ +#ifdef _MSC_VER + unsigned long pos; + if (_BitScanForward(&pos, mask)) + return static_cast(pos); + return UINT8_MAX; +#elif VMA_CPP20 + if(mask != 0) + return static_cast(std::countr_zero(mask)); + return UINT8_MAX; +#elif defined __GNUC__ || defined __clang__ + return static_cast(__builtin_ffs(mask)) - 1U; +#else + uint8_t pos = 0; + uint32_t bit = 1; + do + { + if (mask & bit) + return pos; + bit <<= 1; + } while (pos++ < 31); + return UINT8_MAX; +#endif +} + +inline uint8_t VmaBitScanMSB(uint64_t mask) +{ +#if defined(_MSC_VER) && defined(_WIN64) + unsigned long pos; + if (_BitScanReverse64(&pos, mask)) + return static_cast(pos); +#elif VMA_CPP20 + if(mask != 0) + return 63 - static_cast(std::countl_zero(mask)); +#elif defined __GNUC__ || defined __clang__ + if (mask != 0) + return 63 - static_cast(__builtin_clzll(mask)); +#else + uint8_t pos = 63; + uint64_t bit = 1ULL << 63; + do + { + if (mask & bit) + return pos; + bit >>= 1; + } while (pos-- > 0); +#endif + return UINT8_MAX; +} + +inline uint8_t VmaBitScanMSB(uint32_t mask) +{ +#ifdef _MSC_VER + unsigned long pos; + if (_BitScanReverse(&pos, mask)) + return static_cast(pos); +#elif VMA_CPP20 + if(mask != 0) + return 31 - static_cast(std::countl_zero(mask)); +#elif defined __GNUC__ || defined __clang__ + if (mask != 0) + return 31 - static_cast(__builtin_clz(mask)); +#else + uint8_t pos = 31; + uint32_t bit = 1UL << 31; + do + { + if (mask & bit) + return pos; + bit >>= 1; + } while (pos-- > 0); +#endif + return UINT8_MAX; +} + +/* +Returns true if given number is a power of two. +T must be unsigned integer number or signed integer but always nonnegative. +For 0 returns true. +*/ +template +inline bool VmaIsPow2(T x) +{ + return (x & (x - 1)) == 0; +} + +// Aligns given value up to nearest multiply of align value. For example: VmaAlignUp(11, 8) = 16. +// Use types like uint32_t, uint64_t as T. +template +inline T VmaAlignUp(T val, T alignment) +{ + VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); + return (val + alignment - 1) & ~(alignment - 1); +} + +// Aligns given value down to nearest multiply of align value. For example: VmaAlignDown(11, 8) = 8. +// Use types like uint32_t, uint64_t as T. +template +inline T VmaAlignDown(T val, T alignment) +{ + VMA_HEAVY_ASSERT(VmaIsPow2(alignment)); + return val & ~(alignment - 1); +} + +// Division with mathematical rounding to nearest number. +template +inline T VmaRoundDiv(T x, T y) +{ + return (x + (y / (T)2)) / y; +} + +// Divide by 'y' and round up to nearest integer. +template +inline T VmaDivideRoundingUp(T x, T y) +{ + return (x + y - (T)1) / y; +} + +// Returns smallest power of 2 greater or equal to v. +inline uint32_t VmaNextPow2(uint32_t v) +{ + v--; + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v++; + return v; +} + +inline uint64_t VmaNextPow2(uint64_t v) +{ + v--; + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v |= v >> 32; + v++; + return v; +} + +// Returns largest power of 2 less or equal to v. +inline uint32_t VmaPrevPow2(uint32_t v) +{ + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v = v ^ (v >> 1); + return v; +} + +inline uint64_t VmaPrevPow2(uint64_t v) +{ + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + v |= v >> 32; + v = v ^ (v >> 1); + return v; +} + +inline bool VmaStrIsEmpty(const char* pStr) +{ + return pStr == VMA_NULL || *pStr == '\0'; +} + +/* +Returns true if two memory blocks occupy overlapping pages. +ResourceA must be in less memory offset than ResourceB. + +Algorithm is based on "Vulkan 1.0.39 - A Specification (with all registered Vulkan extensions)" +chapter 11.6 "Resource Memory Association", paragraph "Buffer-Image Granularity". +*/ +inline bool VmaBlocksOnSamePage( + VkDeviceSize resourceAOffset, + VkDeviceSize resourceASize, + VkDeviceSize resourceBOffset, + VkDeviceSize pageSize) +{ + VMA_ASSERT(resourceAOffset + resourceASize <= resourceBOffset && resourceASize > 0 && pageSize > 0); + VkDeviceSize resourceAEnd = resourceAOffset + resourceASize - 1; + VkDeviceSize resourceAEndPage = resourceAEnd & ~(pageSize - 1); + VkDeviceSize resourceBStart = resourceBOffset; + VkDeviceSize resourceBStartPage = resourceBStart & ~(pageSize - 1); + return resourceAEndPage == resourceBStartPage; +} + +/* +Returns true if given suballocation types could conflict and must respect +VkPhysicalDeviceLimits::bufferImageGranularity. They conflict if one is buffer +or linear image and another one is optimal image. If type is unknown, behave +conservatively. +*/ +inline bool VmaIsBufferImageGranularityConflict( + VmaSuballocationType suballocType1, + VmaSuballocationType suballocType2) +{ + if (suballocType1 > suballocType2) + { + std::swap(suballocType1, suballocType2); + } + + switch (suballocType1) + { + case VMA_SUBALLOCATION_TYPE_FREE: + return false; + case VMA_SUBALLOCATION_TYPE_UNKNOWN: + return true; + case VMA_SUBALLOCATION_TYPE_BUFFER: + return + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; + case VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN: + return + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN || + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR || + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; + case VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR: + return + suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL; + case VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL: + return false; + default: + VMA_ASSERT(0); + return true; + } +} + +void VmaWriteMagicValue(void* pData, VkDeviceSize offset) +{ +#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION + uint32_t* pDst = (uint32_t*)((char*)pData + offset); + const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); + for (size_t i = 0; i < numberCount; ++i, ++pDst) + { + *pDst = VMA_CORRUPTION_DETECTION_MAGIC_VALUE; + } +#else + // no-op +#endif +} + +bool VmaValidateMagicValue(const void* pData, VkDeviceSize offset) +{ +#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION + const uint32_t* pSrc = (const uint32_t*)((const char*)pData + offset); + const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t); + for (size_t i = 0; i < numberCount; ++i, ++pSrc) + { + if (*pSrc != VMA_CORRUPTION_DETECTION_MAGIC_VALUE) + { + return false; + } + } +#endif + return true; +} + +/* +Fills structure with parameters of an example buffer to be used for transfers +during GPU memory defragmentation. +*/ +void VmaFillGpuDefragmentationBufferCreateInfo(VkBufferCreateInfo& outBufCreateInfo) +{ + memset(&outBufCreateInfo, 0, sizeof(outBufCreateInfo)); + outBufCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO; + outBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; + outBufCreateInfo.size = (VkDeviceSize)VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE; // Example size. +} + + +/* +Performs binary search and returns iterator to first element that is greater or +equal to (key), according to comparison (cmp). + +Cmp should return true if first argument is less than second argument. + +Returned value is the found element, if present in the collection or place where +new element with value (key) should be inserted. +*/ +template +IterT VmaBinaryFindFirstNotLess(IterT beg, IterT end, const KeyT& key, const CmpLess& cmp) +{ + size_t down = 0; + size_t up = size_t(end - beg); + while (down < up) + { + const size_t mid = down + (up - down) / 2; // Overflow-safe midpoint calculation + if (cmp(*(beg + mid), key)) + { + down = mid + 1; + } + else + { + up = mid; + } + } + return beg + down; +} + +template +IterT VmaBinaryFindSorted(const IterT& beg, const IterT& end, const KeyT& value, const CmpLess& cmp) +{ + IterT it = VmaBinaryFindFirstNotLess( + beg, end, value, cmp); + if (it == end || + (!cmp(*it, value) && !cmp(value, *it))) + { + return it; + } + return end; +} + +/* +Returns true if all pointers in the array are not-null and unique. +Warning! O(n^2) complexity. Use only inside VMA_HEAVY_ASSERT. +T must be pointer type, e.g. VmaAllocation, VmaPool. +*/ +template +bool VmaValidatePointerArray(uint32_t count, const T* arr) +{ + for (uint32_t i = 0; i < count; ++i) + { + const T iPtr = arr[i]; + if (iPtr == VMA_NULL) + { + return false; + } + for (uint32_t j = i + 1; j < count; ++j) + { + if (iPtr == arr[j]) + { + return false; + } + } + } + return true; +} + +template +inline void VmaPnextChainPushFront(MainT* mainStruct, NewT* newStruct) +{ + newStruct->pNext = mainStruct->pNext; + mainStruct->pNext = newStruct; +} + // This is the main algorithm that guides the selection of a memory type best for an allocation - // converts usage to required/preferred/not preferred flags. -static bool FindMemoryPreferences( +bool FindMemoryPreferences( bool isIntegratedGPU, const VmaAllocationCreateInfo& allocCreateInfo, VmaBufferImageUsage bufImgUsage, @@ -4078,8 +4265,11 @@ static bool FindMemoryPreferences( } else { - // Always CPU memory. - outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + if(hostAccessAllowTransferInstead) + outPreferredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + else + // Always CPU memory. + outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; } } // CPU sequential write - may be CPU or host-visible GPU memory, uncached and write-combined. @@ -4094,7 +4284,12 @@ static bool FindMemoryPreferences( } else { - outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + if(hostAccessAllowTransferInstead) + outPreferredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + else + // Always CPU memory. + outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT; + // Direct GPU access, CPU sequential write (e.g. a dynamic uniform buffer updated every frame) if(deviceAccess) { @@ -4154,7 +4349,7 @@ static bool FindMemoryPreferences( //////////////////////////////////////////////////////////////////////////////// // Memory allocation -static void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t size, size_t alignment) +inline void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t size, size_t alignment) { void* result = VMA_NULL; if ((pAllocationCallbacks != VMA_NULL) && @@ -4174,7 +4369,7 @@ static void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t return result; } -static void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr) +inline void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr) { if ((pAllocationCallbacks != VMA_NULL) && (pAllocationCallbacks->pfnFree != VMA_NULL)) @@ -4188,13 +4383,13 @@ static void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr } template -static T* VmaAllocate(const VkAllocationCallbacks* pAllocationCallbacks) +T* VmaAllocate(const VkAllocationCallbacks* pAllocationCallbacks) { return (T*)VmaMalloc(pAllocationCallbacks, sizeof(T), VMA_ALIGN_OF(T)); } template -static T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, size_t count) +T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, size_t count) { return (T*)VmaMalloc(pAllocationCallbacks, sizeof(T) * count, VMA_ALIGN_OF(T)); } @@ -4204,14 +4399,14 @@ static T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, si #define vma_new_array(allocator, type, count) new(VmaAllocateArray((allocator), (count)))(type) template -static void vma_delete(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr) +void vma_delete(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr) { ptr->~T(); VmaFree(pAllocationCallbacks, ptr); } template -static void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr, size_t count) +void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr, size_t count) { if (ptr != VMA_NULL) { @@ -4223,7 +4418,7 @@ static void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, } } -static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr) +char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr) { if (srcStr != VMA_NULL) { @@ -4236,7 +4431,7 @@ static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char } #if VMA_STATS_STRING_ENABLED -static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr, size_t strLen) +char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr, size_t strLen) { if (srcStr != VMA_NULL) { @@ -4249,7 +4444,7 @@ static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char } #endif // VMA_STATS_STRING_ENABLED -static void VmaFreeString(const VkAllocationCallbacks* allocs, char* str) +void VmaFreeString(const VkAllocationCallbacks* allocs, char* str) { if (str != VMA_NULL) { @@ -4287,11 +4482,17 @@ bool VmaVectorRemoveSorted(VectorT& vector, const typename VectorT::value_type& } return false; } + +} // namespace + #endif // _VMA_FUNCTIONS #ifndef _VMA_STATISTICS_FUNCTIONS -static void VmaClearStatistics(VmaStatistics& outStats) +namespace +{ + +void VmaClearStatistics(VmaStatistics& outStats) { outStats.blockCount = 0; outStats.allocationCount = 0; @@ -4299,7 +4500,7 @@ static void VmaClearStatistics(VmaStatistics& outStats) outStats.allocationBytes = 0; } -static void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src) +void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src) { inoutStats.blockCount += src.blockCount; inoutStats.allocationCount += src.allocationCount; @@ -4307,7 +4508,7 @@ static void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src inoutStats.allocationBytes += src.allocationBytes; } -static void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats) +void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats) { VmaClearStatistics(outStats.statistics); outStats.unusedRangeCount = 0; @@ -4317,7 +4518,7 @@ static void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats) outStats.unusedRangeSizeMax = 0; } -static void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats, VkDeviceSize size) +void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats, VkDeviceSize size) { inoutStats.statistics.allocationCount++; inoutStats.statistics.allocationBytes += size; @@ -4325,14 +4526,14 @@ static void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats inoutStats.allocationSizeMax = VMA_MAX(inoutStats.allocationSizeMax, size); } -static void VmaAddDetailedStatisticsUnusedRange(VmaDetailedStatistics& inoutStats, VkDeviceSize size) +void VmaAddDetailedStatisticsUnusedRange(VmaDetailedStatistics& inoutStats, VkDeviceSize size) { inoutStats.unusedRangeCount++; inoutStats.unusedRangeSizeMin = VMA_MIN(inoutStats.unusedRangeSizeMin, size); inoutStats.unusedRangeSizeMax = VMA_MAX(inoutStats.unusedRangeSizeMax, size); } -static void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const VmaDetailedStatistics& src) +void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const VmaDetailedStatistics& src) { VmaAddStatistics(inoutStats.statistics, src.statistics); inoutStats.unusedRangeCount += src.unusedRangeCount; @@ -4342,6 +4543,8 @@ static void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const Vm inoutStats.unusedRangeSizeMax = VMA_MAX(inoutStats.unusedRangeSizeMax, src.unusedRangeSizeMax); } +} // namespace + #endif // _VMA_STATISTICS_FUNCTIONS #ifndef _VMA_MUTEX_LOCK @@ -4441,8 +4644,8 @@ struct VmaStlAllocator VmaStlAllocator(const VmaStlAllocator&) = default; VmaStlAllocator& operator=(const VmaStlAllocator&) = delete; - T* allocate(size_t n) { return VmaAllocateArray(m_pCallbacks, n); } - void deallocate(T* p, size_t n) { VmaFree(m_pCallbacks, p); } + T* allocate(size_t n); + void deallocate(T* p, size_t n); template bool operator==(const VmaStlAllocator& rhs) const @@ -4455,6 +4658,12 @@ struct VmaStlAllocator return m_pCallbacks != rhs.m_pCallbacks; } }; + +template +T* VmaStlAllocator::allocate(size_t n) { return VmaAllocateArray(m_pCallbacks, n); } + +template +void VmaStlAllocator::deallocate(T* p, size_t n) { VmaFree(m_pCallbacks, p); } #endif // _VMA_STL_ALLOCATOR #ifndef _VMA_VECTOR @@ -4476,7 +4685,7 @@ public: VmaVector(size_t count, const T& value, const AllocatorT& allocator) : VmaVector(count, allocator) {} VmaVector(const VmaVector& src); VmaVector& operator=(const VmaVector& rhs); - ~VmaVector() { VmaFree(m_Allocator.m_pCallbacks, m_pArray); } + ~VmaVector(); bool empty() const { return m_Count == 0; } size_t size() const { return m_Count; } @@ -4517,6 +4726,9 @@ private: }; #ifndef _VMA_VECTOR_FUNCTIONS +template +VmaVector::~VmaVector() { VmaFree(m_Allocator.m_pCallbacks, m_pArray); } + template VmaVector::VmaVector(const AllocatorT& allocator) : m_Allocator(allocator), @@ -4600,7 +4812,8 @@ void VmaVector::resize(size_t newCount) if (newCapacity != m_Capacity) { - T* const newArray = newCapacity ? VmaAllocateArray(m_Allocator.m_pCallbacks, newCapacity) : VMA_NULL; + VMA_HEAVY_ASSERT(newCapacity > 0); + T* const newArray = VmaAllocateArray(m_Allocator.m_pCallbacks, newCapacity); const size_t elementsToCopy = VMA_MIN(m_Count, newCount); if (elementsToCopy != 0) { @@ -4657,17 +4870,23 @@ void VmaVector::remove(size_t index) } #endif // _VMA_VECTOR_FUNCTIONS +namespace +{ + template -static void VmaVectorInsert(VmaVector& vec, size_t index, const T& item) +void VmaVectorInsert(VmaVector& vec, size_t index, const T& item) { vec.insert(index, item); } template -static void VmaVectorRemove(VmaVector& vec, size_t index) +void VmaVectorRemove(VmaVector& vec, size_t index) { vec.remove(index); } + +} // namespace + #endif // _VMA_VECTOR #ifndef _VMA_SMALL_VECTOR @@ -6065,7 +6284,10 @@ void VmaJsonWriter::WriteIndent(bool oneLess) } #endif // _VMA_JSON_WRITER_FUNCTIONS -static void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedStatistics& stat) +namespace +{ + +void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedStatistics& stat) { json.BeginObject(); @@ -6096,6 +6318,9 @@ static void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedSta } json.EndObject(); } + +} // namespace + #endif // _VMA_JSON_WRITER #ifndef _VMA_MAPPING_HYSTERESIS @@ -6176,7 +6401,7 @@ public: } private: - static const int32_t COUNTER_MIN_EXTRA_MAPPING = 7; + static constexpr int32_t COUNTER_MIN_EXTRA_MAPPING = 7; uint32_t m_MinorCounter = 0; uint32_t m_MajorCounter = 0; @@ -6203,40 +6428,45 @@ class VmaWin32Handle { public: VmaWin32Handle() noexcept : m_hHandle(VMA_NULL) { } - explicit VmaWin32Handle(HANDLE hHandle) noexcept : m_hHandle(hHandle) { } - ~VmaWin32Handle() noexcept { if (m_hHandle != VMA_NULL) { ::CloseHandle(m_hHandle); } } + explicit VmaWin32Handle(HANDLE hHandle) noexcept + : m_hHandle(hHandle) + , m_IsNTHandle(IsNTHandle(hHandle)) + { + } + ~VmaWin32Handle() noexcept { if (m_hHandle != VMA_NULL && m_IsNTHandle) { ::CloseHandle(m_hHandle); } } VMA_CLASS_NO_COPY_NO_MOVE(VmaWin32Handle) public: // Strengthened - VkResult GetHandle(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, HANDLE hTargetProcess, bool useMutex, HANDLE* pHandle) noexcept + VkResult GetHandle(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, bool useMutex, HANDLE* pHandle) noexcept { *pHandle = VMA_NULL; // Try to get handle first. - if (m_hHandle != VMA_NULL) - { - *pHandle = Duplicate(hTargetProcess); - return VK_SUCCESS; - } - VkResult res = VK_SUCCESS; - // If failed, try to create it. + if (m_hHandle == VMA_NULL) { VmaMutexLockWrite lock(m_Mutex, useMutex); if (m_hHandle == VMA_NULL) { - res = Create(device, memory, pvkGetMemoryWin32HandleKHR, &m_hHandle); + res = Create(device, memory, pvkGetMemoryWin32HandleKHR, handleType, &m_hHandle); + if (res != VK_SUCCESS) { + m_hHandle = VMA_NULL; + return res; + } + m_IsNTHandle = IsNTHandle(m_hHandle); } } - - *pHandle = Duplicate(hTargetProcess); + if (res == VK_SUCCESS) { + // KMT handle is returned as is. + *pHandle = m_IsNTHandle ? Duplicate(hTargetProcess) : m_hHandle; + } return res; } operator bool() const noexcept { return m_hHandle != VMA_NULL; } private: // Not atomic - static VkResult Create(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, HANDLE* pHandle) noexcept + static VkResult Create(VkDevice device, VkDeviceMemory memory, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE* pHandle) noexcept { VkResult res = VK_ERROR_FEATURE_NOT_PRESENT; if (pvkGetMemoryWin32HandleKHR != VMA_NULL) @@ -6244,7 +6474,7 @@ private: VkMemoryGetWin32HandleInfoKHR handleInfo{ }; handleInfo.sType = VK_STRUCTURE_TYPE_MEMORY_GET_WIN32_HANDLE_INFO_KHR; handleInfo.memory = memory; - handleInfo.handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR; + handleInfo.handleType = handleType; res = pvkGetMemoryWin32HandleKHR(device, &handleInfo, pHandle); } return res; @@ -6262,9 +6492,15 @@ private: } return hDupHandle; } + static bool IsNTHandle(HANDLE hHandle) noexcept + { + DWORD flags = 0; + return (hHandle != VMA_NULL) ? (::GetHandleInformation(hHandle, &flags) != 0) : false; + } private: HANDLE m_hHandle; VMA_RW_MUTEX m_Mutex; // Protects access m_Handle + bool m_IsNTHandle = false; // True if m_Handle is NT handle, false if it's a KMT handle. }; #else class VmaWin32Handle @@ -6272,6 +6508,7 @@ class VmaWin32Handle // ABI compatibility void* placeholder = VMA_NULL; VMA_RW_MUTEX placeholder2; + bool placeholder3 = false; }; #endif // VMA_EXTERNAL_MEMORY_WIN32 @@ -6346,6 +6583,7 @@ public: VkResult CreateWin32Handle( const VmaAllocator hAllocator, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, + VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* pHandle)noexcept; #endif // VMA_EXTERNAL_MEMORY_WIN32 @@ -6460,7 +6698,7 @@ public: #endif #if VMA_EXTERNAL_MEMORY_WIN32 - VkResult GetWin32Handle(VmaAllocator hAllocator, HANDLE hTargetProcess, HANDLE* hHandle) noexcept; + VkResult GetWin32Handle(VmaAllocator hAllocator, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* hHandle) noexcept; #endif // VMA_EXTERNAL_MEMORY_WIN32 private: @@ -6966,7 +7204,7 @@ public: bool FinishValidation(ValidationContext& ctx) const; private: - static const uint16_t MAX_LOW_BUFFER_IMAGE_GRANULARITY = 256; + static constexpr uint16_t MAX_LOW_BUFFER_IMAGE_GRANULARITY = 256; struct RegionInfo { @@ -8907,11 +9145,11 @@ private: // According to original paper it should be preferable 4 or 5: // M. Masmano, I. Ripoll, A. Crespo, and J. Real "TLSF: a New Dynamic Memory Allocator for Real-Time Systems" // http://www.gii.upv.es/tlsf/files/ecrts04_tlsf.pdf - static const uint8_t SECOND_LEVEL_INDEX = 5; - static const uint16_t SMALL_BUFFER_SIZE = 256; - static const uint32_t INITIAL_BLOCK_ALLOC_COUNT = 16; - static const uint8_t MEMORY_CLASS_SHIFT = 7; - static const uint8_t MAX_MEMORY_CLASSES = 65 - MEMORY_CLASS_SHIFT; + static constexpr uint8_t SECOND_LEVEL_INDEX = 5; + static constexpr uint16_t SMALL_BUFFER_SIZE = 256; + static constexpr uint32_t INITIAL_BLOCK_ALLOC_COUNT = 16; + static constexpr uint8_t MEMORY_CLASS_SHIFT = 7; + static constexpr uint8_t MAX_MEMORY_CLASSES = 65 - MEMORY_CLASS_SHIFT; class Block { @@ -9927,7 +10165,7 @@ public: private: // Max number of allocations to ignore due to size constraints before ending single pass - static const uint8_t MAX_ALLOCS_TO_IGNORE = 16; + static constexpr uint8_t MAX_ALLOCS_TO_IGNORE = 16; enum class CounterStatus { Pass, Ignore, End }; struct FragmentedBlock @@ -10385,14 +10623,32 @@ public: VmaBufferImageUsage bufImgUsage, uint32_t* pMemoryTypeIndex) const; + // Common code for public functions vmaCreateBuffer, vmaCreateBufferWithAlignment, etc. + VkResult CreateBuffer( + const VkBufferCreateInfo* pBufferCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, // pNext chain for VkMemoryAllocateInfo. + VkBuffer* pBuffer, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo); + // Common code for public functions vmaCreateImage, vmaCreateDedicatedImage. + VkResult CreateImage( + const VkImageCreateInfo* pImageCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, // pNext chain for VkMemoryAllocateInfo. + VkImage* pImage, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo); + // Main allocation function. VkResult AllocateMemory( - const VkMemoryRequirements& vkMemReq, + VkMemoryRequirements vkMemReq, bool requiresDedicatedAllocation, bool prefersDedicatedAllocation, VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + void* pMemoryAllocateNext, // Optional pNext chain for VkMemoryAllocateInfo. const VmaAllocationCreateInfo& createInfo, VmaSuballocationType suballocType, size_t allocationCount, @@ -10538,6 +10794,7 @@ private: VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + void* pMemoryAllocateNext, // Optional pNext chain for VkMemoryAllocateInfo. const VmaAllocationCreateInfo& createInfo, uint32_t memTypeIndex, VmaSuballocationType suballocType, @@ -10577,7 +10834,7 @@ private: VmaBufferImageUsage dedicatedBufferImageUsage, size_t allocationCount, VmaAllocation* pAllocations, - const void* pNextChain = VMA_NULL); + const void* pNextChain); void FreeDedicatedMemory(VmaAllocation allocation); @@ -10609,30 +10866,32 @@ private: #ifndef _VMA_MEMORY_FUNCTIONS -static void* VmaMalloc(VmaAllocator hAllocator, size_t size, size_t alignment) +namespace +{ +inline void* VmaMalloc(VmaAllocator hAllocator, size_t size, size_t alignment) { return VmaMalloc(&hAllocator->m_AllocationCallbacks, size, alignment); } -static void VmaFree(VmaAllocator hAllocator, void* ptr) +inline void VmaFree(VmaAllocator hAllocator, void* ptr) { VmaFree(&hAllocator->m_AllocationCallbacks, ptr); } template -static T* VmaAllocate(VmaAllocator hAllocator) +T* VmaAllocate(VmaAllocator hAllocator) { return (T*)VmaMalloc(hAllocator, sizeof(T), VMA_ALIGN_OF(T)); } template -static T* VmaAllocateArray(VmaAllocator hAllocator, size_t count) +T* VmaAllocateArray(VmaAllocator hAllocator, size_t count) { return (T*)VmaMalloc(hAllocator, sizeof(T) * count, VMA_ALIGN_OF(T)); } template -static void vma_delete(VmaAllocator hAllocator, T* ptr) +void vma_delete(VmaAllocator hAllocator, T* ptr) { if(ptr != VMA_NULL) { @@ -10642,7 +10901,7 @@ static void vma_delete(VmaAllocator hAllocator, T* ptr) } template -static void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count) +void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count) { if(ptr != VMA_NULL) { @@ -10651,6 +10910,7 @@ static void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count) VmaFree(hAllocator, ptr); } } +} // namespace #endif // _VMA_MEMORY_FUNCTIONS #ifndef _VMA_DEVICE_MEMORY_BLOCK_FUNCTIONS @@ -10904,10 +11164,10 @@ VkResult VmaDeviceMemoryBlock::BindImageMemory( } #if VMA_EXTERNAL_MEMORY_WIN32 -VkResult VmaDeviceMemoryBlock::CreateWin32Handle(const VmaAllocator hAllocator, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, HANDLE hTargetProcess, HANDLE* pHandle) noexcept +VkResult VmaDeviceMemoryBlock::CreateWin32Handle(const VmaAllocator hAllocator, PFN_vkGetMemoryWin32HandleKHR pvkGetMemoryWin32HandleKHR, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* pHandle) noexcept { VMA_ASSERT(pHandle); - return m_Handle.GetHandle(hAllocator->m_hDevice, m_hMemory, pvkGetMemoryWin32HandleKHR, hTargetProcess, hAllocator->m_UseMutex, pHandle); + return m_Handle.GetHandle(hAllocator->m_hDevice, m_hMemory, pvkGetMemoryWin32HandleKHR, handleType, hTargetProcess, hAllocator->m_UseMutex, pHandle); } #endif // VMA_EXTERNAL_MEMORY_WIN32 #endif // _VMA_DEVICE_MEMORY_BLOCK_FUNCTIONS @@ -11225,16 +11485,16 @@ void VmaAllocation_T::PrintParameters(class VmaJsonWriter& json) const } } #if VMA_EXTERNAL_MEMORY_WIN32 -VkResult VmaAllocation_T::GetWin32Handle(VmaAllocator hAllocator, HANDLE hTargetProcess, HANDLE* pHandle) noexcept +VkResult VmaAllocation_T::GetWin32Handle(VmaAllocator hAllocator, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* pHandle) noexcept { auto pvkGetMemoryWin32HandleKHR = hAllocator->GetVulkanFunctions().vkGetMemoryWin32HandleKHR; switch (m_Type) { case ALLOCATION_TYPE_BLOCK: - return m_BlockAllocation.m_Block->CreateWin32Handle(hAllocator, pvkGetMemoryWin32HandleKHR, hTargetProcess, pHandle); + return m_BlockAllocation.m_Block->CreateWin32Handle(hAllocator, pvkGetMemoryWin32HandleKHR, handleType, hTargetProcess, pHandle); case ALLOCATION_TYPE_DEDICATED: EnsureExtraData(hAllocator); - return m_DedicatedAllocation.m_ExtraData->m_Handle.GetHandle(hAllocator->m_hDevice, m_DedicatedAllocation.m_hMemory, pvkGetMemoryWin32HandleKHR, hTargetProcess, hAllocator->m_UseMutex, pHandle); + return m_DedicatedAllocation.m_ExtraData->m_Handle.GetHandle(hAllocator->m_hDevice, m_DedicatedAllocation.m_hMemory, pvkGetMemoryWin32HandleKHR, handleType, hTargetProcess, hAllocator->m_UseMutex, pHandle); default: VMA_ASSERT(0); return VK_ERROR_FEATURE_NOT_PRESENT; @@ -11348,11 +11608,13 @@ bool VmaBlockVector::IsEmpty() bool VmaBlockVector::IsCorruptionDetectionEnabled() const { - const uint32_t requiredMemFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; - return (VMA_DEBUG_DETECT_CORRUPTION != 0) && - (VMA_DEBUG_MARGIN > 0) && - (m_Algorithm == 0 || m_Algorithm == VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT) && +#if (VMA_DEBUG_DETECT_CORRUPTION == 0) || (VMA_DEBUG_MARGIN == 0) + return false; +#else + constexpr uint32_t requiredMemFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; + return (m_Algorithm == 0 || m_Algorithm == VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT) && (m_hAllocator->m_MemProps.memoryTypes[m_MemoryTypeIndex].propertyFlags & requiredMemFlags) == requiredMemFlags; +#endif } VkResult VmaBlockVector::Allocate( @@ -11802,10 +12064,11 @@ VkResult VmaBlockVector::CommitAllocationRequest( else (*pAllocation)->SetUserData(m_hAllocator, pUserData); m_hAllocator->m_Budget.AddAllocation(m_hAllocator->MemoryTypeIndexToHeapIndex(m_MemoryTypeIndex), allocRequest.size); - if (VMA_DEBUG_INITIALIZE_ALLOCATIONS) - { - m_hAllocator->FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); - } + +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + m_hAllocator->FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); +#endif + if (IsCorruptionDetectionEnabled()) { VkResult res = pBlock->WriteMagicValueAfterAllocation(m_hAllocator, (*pAllocation)->GetOffset(), allocRequest.size); @@ -13496,6 +13759,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + void* pMemoryAllocateNext, const VmaAllocationCreateInfo& createInfo, uint32_t memTypeIndex, VmaSuballocationType suballocType, @@ -13516,6 +13780,14 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( if(res != VK_SUCCESS) return res; + const void* allocateNextPtr = blockVector.GetAllocationNextPtr(); + if(pMemoryAllocateNext != VMA_NULL) + { + VMA_ASSERT(allocateNextPtr == VMA_NULL && + "You shouldn't create a dedicated allocation with a custom pMemoryAllocateNext if the pNext chain is already provided for this pool."); + allocateNextPtr = pMemoryAllocateNext; + } + if((finalCreateInfo.flags & VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT) != 0) { return AllocateDedicatedMemory( @@ -13536,7 +13808,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( dedicatedBufferImageUsage, allocationCount, pAllocations, - blockVector.GetAllocationNextPtr()); + allocateNextPtr); } const bool canAllocateDedicated = @@ -13579,7 +13851,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( dedicatedBufferImageUsage, allocationCount, pAllocations, - blockVector.GetAllocationNextPtr()); + allocateNextPtr); if(res == VK_SUCCESS) { // Succeeded: AllocateDedicatedMemory function already filled pMemory, nothing more to do here. @@ -13620,7 +13892,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( dedicatedBufferImageUsage, allocationCount, pAllocations, - blockVector.GetAllocationNextPtr()); + allocateNextPtr); if(res == VK_SUCCESS) { // Succeeded: AllocateDedicatedMemory function already filled pMemory, nothing more to do here. @@ -13628,6 +13900,7 @@ VkResult VmaAllocator_T::AllocateMemoryOfType( return VK_SUCCESS; } } + // Everything failed: Return error code. VMA_DEBUG_LOG(" vkAllocateMemory FAILED"); return res; @@ -13825,10 +14098,10 @@ VkResult VmaAllocator_T::AllocateDedicatedMemoryPage( else (*pAllocation)->SetUserData(this, pUserData); m_Budget.AddAllocation(MemoryTypeIndexToHeapIndex(memTypeIndex), size); - if(VMA_DEBUG_INITIALIZE_ALLOCATIONS) - { - FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); - } + +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + FillAllocation(*pAllocation, VMA_ALLOCATION_FILL_PATTERN_CREATED); +#endif return VK_SUCCESS; } @@ -14027,11 +14300,12 @@ VkResult VmaAllocator_T::CalcAllocationParams( return VK_ERROR_FEATURE_NOT_PRESENT; } - if(VMA_DEBUG_ALWAYS_DEDICATED_MEMORY && - (inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) != 0) +#if VMA_DEBUG_ALWAYS_DEDICATED_MEMORY + if((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) != 0) { inoutCreateInfo.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; } +#endif // Non-auto USAGE values imply HOST_ACCESS flags. // And so does VMA_MEMORY_USAGE_UNKNOWN because it is used with custom pools. @@ -14050,13 +14324,176 @@ VkResult VmaAllocator_T::CalcAllocationParams( return VK_SUCCESS; } +VkResult VmaAllocator_T::CreateBuffer( + const VkBufferCreateInfo* pBufferCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkBuffer* pBuffer, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + *pBuffer = VK_NULL_HANDLE; + *pAllocation = VK_NULL_HANDLE; + + if (pBufferCreateInfo->size == 0) + { + return VK_ERROR_INITIALIZATION_FAILED; + } + if ((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 && + !m_UseKhrBufferDeviceAddress) + { + VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used."); + return VK_ERROR_INITIALIZATION_FAILED; + } + + // 1. Create VkBuffer. + VkResult res = (*m_VulkanFunctions.vkCreateBuffer)(m_hDevice, pBufferCreateInfo, + GetAllocationCallbacks(), pBuffer); + if (res >= 0) + { + // 2. vkGetBufferMemoryRequirements. + VkMemoryRequirements vkMemReq = {}; + bool requiresDedicatedAllocation = false; + bool prefersDedicatedAllocation = false; + GetBufferMemoryRequirements(*pBuffer, vkMemReq, + requiresDedicatedAllocation, prefersDedicatedAllocation); + + // 3. Allocate memory using allocator. + res = AllocateMemory( + vkMemReq, + requiresDedicatedAllocation, + prefersDedicatedAllocation, + *pBuffer, // dedicatedBuffer + VK_NULL_HANDLE, // dedicatedImage + VmaBufferImageUsage(*pBufferCreateInfo, m_UseKhrMaintenance5), // dedicatedBufferImageUsage + pMemoryAllocateNext, + *pAllocationCreateInfo, + VMA_SUBALLOCATION_TYPE_BUFFER, + 1, // allocationCount + pAllocation); + if (res >= 0) + { + // 3. Bind buffer with memory. + if ((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) + { + res = BindBufferMemory(*pAllocation, 0, *pBuffer, VMA_NULL); + } + if (res >= 0) + { + // All steps succeeded. +#if VMA_STATS_STRING_ENABLED + (*pAllocation)->InitBufferUsage(*pBufferCreateInfo, m_UseKhrMaintenance5); +#endif + if (pAllocationInfo != VMA_NULL) + { + GetAllocationInfo(*pAllocation, pAllocationInfo); + } + + return VK_SUCCESS; + } + FreeMemory(1, pAllocation); + *pAllocation = VK_NULL_HANDLE; + (*m_VulkanFunctions.vkDestroyBuffer)(m_hDevice, *pBuffer, GetAllocationCallbacks()); + *pBuffer = VK_NULL_HANDLE; + return res; + } + (*m_VulkanFunctions.vkDestroyBuffer)(m_hDevice, *pBuffer, GetAllocationCallbacks()); + *pBuffer = VK_NULL_HANDLE; + return res; + } + return res; +} + +VkResult VmaAllocator_T::CreateImage( + const VkImageCreateInfo* pImageCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkImage* pImage, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + *pImage = VK_NULL_HANDLE; + *pAllocation = VK_NULL_HANDLE; + + if (pImageCreateInfo->extent.width == 0 || + pImageCreateInfo->extent.height == 0 || + pImageCreateInfo->extent.depth == 0 || + pImageCreateInfo->mipLevels == 0 || + pImageCreateInfo->arrayLayers == 0) + { + return VK_ERROR_INITIALIZATION_FAILED; + } + + // 1. Create VkImage. + VkResult res = (*m_VulkanFunctions.vkCreateImage)(m_hDevice, pImageCreateInfo, + GetAllocationCallbacks(), pImage); + if (res == VK_SUCCESS) + { + VmaSuballocationType suballocType = pImageCreateInfo->tiling == VK_IMAGE_TILING_OPTIMAL ? + VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL : + VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR; + + // 2. Allocate memory using allocator. + VkMemoryRequirements vkMemReq = {}; + bool requiresDedicatedAllocation = false; + bool prefersDedicatedAllocation = false; + GetImageMemoryRequirements(*pImage, vkMemReq, + requiresDedicatedAllocation, prefersDedicatedAllocation); + + res = AllocateMemory( + vkMemReq, + requiresDedicatedAllocation, + prefersDedicatedAllocation, + VK_NULL_HANDLE, // dedicatedBuffer + *pImage, // dedicatedImage + VmaBufferImageUsage(*pImageCreateInfo), // dedicatedBufferImageUsage + pMemoryAllocateNext, + *pAllocationCreateInfo, + suballocType, + 1, // allocationCount + pAllocation); + if (res == VK_SUCCESS) + { + // 3. Bind image with memory. + if ((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) + { + res = BindImageMemory(*pAllocation, 0, *pImage, VMA_NULL); + } + if (res == VK_SUCCESS) + { + // All steps succeeded. +#if VMA_STATS_STRING_ENABLED + (*pAllocation)->InitImageUsage(*pImageCreateInfo); +#endif + if (pAllocationInfo != VMA_NULL) + { + GetAllocationInfo(*pAllocation, pAllocationInfo); + } + + return VK_SUCCESS; + } + FreeMemory(1, pAllocation); + *pAllocation = VK_NULL_HANDLE; + (*m_VulkanFunctions.vkDestroyImage)(m_hDevice, *pImage, GetAllocationCallbacks()); + *pImage = VK_NULL_HANDLE; + return res; + } + (*m_VulkanFunctions.vkDestroyImage)(m_hDevice, *pImage, GetAllocationCallbacks()); + *pImage = VK_NULL_HANDLE; + return res; + } + return res; +} + VkResult VmaAllocator_T::AllocateMemory( - const VkMemoryRequirements& vkMemReq, + VkMemoryRequirements vkMemReq, bool requiresDedicatedAllocation, bool prefersDedicatedAllocation, VkBuffer dedicatedBuffer, VkImage dedicatedImage, VmaBufferImageUsage dedicatedBufferImageUsage, + // pNext chain for VkMemoryAllocateInfo. When used, must specify requiresDedicatedAllocation = true. + void* pMemoryAllocateNext, const VmaAllocationCreateInfo& createInfo, VmaSuballocationType suballocType, size_t allocationCount, @@ -14064,8 +14501,15 @@ VkResult VmaAllocator_T::AllocateMemory( { memset(pAllocations, 0, sizeof(VmaAllocation) * allocationCount); + vkMemReq.alignment = VMA_MAX(vkMemReq.alignment, createInfo.minAlignment); VMA_ASSERT(VmaIsPow2(vkMemReq.alignment)); + // If using custom pNext chain for VkMemoryAllocateInfo, must require dedicated allocations. + if(pMemoryAllocateNext != VMA_NULL) + { + requiresDedicatedAllocation = true; + } + if(vkMemReq.size == 0) { return VK_ERROR_INITIALIZATION_FAILED; @@ -14087,6 +14531,7 @@ VkResult VmaAllocator_T::AllocateMemory( dedicatedBuffer, dedicatedImage, dedicatedBufferImageUsage, + pMemoryAllocateNext, createInfoFinal, blockVector.GetMemoryTypeIndex(), suballocType, @@ -14116,6 +14561,7 @@ VkResult VmaAllocator_T::AllocateMemory( dedicatedBuffer, dedicatedImage, dedicatedBufferImageUsage, + pMemoryAllocateNext, createInfoFinal, memTypeIndex, suballocType, @@ -14150,10 +14596,9 @@ void VmaAllocator_T::FreeMemory( if(allocation != VK_NULL_HANDLE) { - if(VMA_DEBUG_INITIALIZE_ALLOCATIONS) - { - FillAllocation(allocation, VMA_ALLOCATION_FILL_PATTERN_DESTROYED); - } +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + FillAllocation(allocation, VMA_ALLOCATION_FILL_PATTERN_DESTROYED); +#endif switch(allocation->GetType()) { @@ -14997,8 +15442,8 @@ void VmaAllocator_T::UpdateVulkanBudget() void VmaAllocator_T::FillAllocation(VmaAllocation hAllocation, uint8_t pattern) { - if(VMA_DEBUG_INITIALIZE_ALLOCATIONS && - hAllocation->IsMappingAllowed() && +#if VMA_DEBUG_INITIALIZE_ALLOCATIONS + if(hAllocation->IsMappingAllowed() && (m_MemProps.memoryTypes[hAllocation->GetMemoryTypeIndex()].propertyFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) != 0) { void* pData = VMA_NULL; @@ -15014,6 +15459,7 @@ void VmaAllocator_T::FillAllocation(VmaAllocation hAllocation, uint8_t pattern) VMA_ASSERT(0 && "VMA_DEBUG_INITIALIZE_ALLOCATIONS is enabled, but couldn't map memory to fill allocation."); } } +#endif // #if VMA_DEBUG_INITIALIZE_ALLOCATIONS } uint32_t VmaAllocator_T::GetGpuDefragmentationMemoryTypeBits() @@ -15543,7 +15989,8 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( VkResult res = VK_SUCCESS; #if VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 - if(funcs->vkGetDeviceBufferMemoryRequirements) + if (funcs->vkGetDeviceBufferMemoryRequirements && + (allocator->m_UseKhrMaintenance4 || allocator->m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))) { // Can query straight from VkBufferCreateInfo :) VkDeviceBufferMemoryRequirementsKHR devBufMemReq = {VK_STRUCTURE_TYPE_DEVICE_BUFFER_MEMORY_REQUIREMENTS_KHR}; @@ -15552,29 +15999,27 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo( VkMemoryRequirements2 memReq = {VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2}; (*funcs->vkGetDeviceBufferMemoryRequirements)(hDev, &devBufMemReq, &memReq); - res = allocator->FindMemoryTypeIndex( + return allocator->FindMemoryTypeIndex( memReq.memoryRequirements.memoryTypeBits, pAllocationCreateInfo, VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), pMemoryTypeIndex); } - else #endif // VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 + + // Must create a dummy buffer to query :( + VkBuffer hBuffer = VK_NULL_HANDLE; + res = funcs->vkCreateBuffer( + hDev, pBufferCreateInfo, allocator->GetAllocationCallbacks(), &hBuffer); + if(res == VK_SUCCESS) { - // Must create a dummy buffer to query :( - VkBuffer hBuffer = VK_NULL_HANDLE; - res = funcs->vkCreateBuffer( - hDev, pBufferCreateInfo, allocator->GetAllocationCallbacks(), &hBuffer); - if(res == VK_SUCCESS) - { - VkMemoryRequirements memReq = {}; - funcs->vkGetBufferMemoryRequirements(hDev, hBuffer, &memReq); + VkMemoryRequirements memReq = {}; + funcs->vkGetBufferMemoryRequirements(hDev, hBuffer, &memReq); - res = allocator->FindMemoryTypeIndex( - memReq.memoryTypeBits, pAllocationCreateInfo, - VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), pMemoryTypeIndex); + res = allocator->FindMemoryTypeIndex( + memReq.memoryTypeBits, pAllocationCreateInfo, + VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), pMemoryTypeIndex); - funcs->vkDestroyBuffer( - hDev, hBuffer, allocator->GetAllocationCallbacks()); - } + funcs->vkDestroyBuffer( + hDev, hBuffer, allocator->GetAllocationCallbacks()); } return res; } @@ -15595,7 +16040,8 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo( VkResult res = VK_SUCCESS; #if VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 - if(funcs->vkGetDeviceImageMemoryRequirements) + if(funcs->vkGetDeviceImageMemoryRequirements && + (allocator->m_UseKhrMaintenance4 || allocator->m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))) { // Can query straight from VkImageCreateInfo :) VkDeviceImageMemoryRequirementsKHR devImgMemReq = {VK_STRUCTURE_TYPE_DEVICE_IMAGE_MEMORY_REQUIREMENTS_KHR}; @@ -15606,29 +16052,27 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo( VkMemoryRequirements2 memReq = {VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2}; (*funcs->vkGetDeviceImageMemoryRequirements)(hDev, &devImgMemReq, &memReq); - res = allocator->FindMemoryTypeIndex( + return allocator->FindMemoryTypeIndex( memReq.memoryRequirements.memoryTypeBits, pAllocationCreateInfo, VmaBufferImageUsage(*pImageCreateInfo), pMemoryTypeIndex); } - else #endif // VMA_KHR_MAINTENANCE4 || VMA_VULKAN_VERSION >= 1003000 + + // Must create a dummy image to query :( + VkImage hImage = VK_NULL_HANDLE; + res = funcs->vkCreateImage( + hDev, pImageCreateInfo, allocator->GetAllocationCallbacks(), &hImage); + if(res == VK_SUCCESS) { - // Must create a dummy image to query :( - VkImage hImage = VK_NULL_HANDLE; - res = funcs->vkCreateImage( - hDev, pImageCreateInfo, allocator->GetAllocationCallbacks(), &hImage); - if(res == VK_SUCCESS) - { - VkMemoryRequirements memReq = {}; - funcs->vkGetImageMemoryRequirements(hDev, hImage, &memReq); + VkMemoryRequirements memReq = {}; + funcs->vkGetImageMemoryRequirements(hDev, hImage, &memReq); - res = allocator->FindMemoryTypeIndex( - memReq.memoryTypeBits, pAllocationCreateInfo, - VmaBufferImageUsage(*pImageCreateInfo), pMemoryTypeIndex); + res = allocator->FindMemoryTypeIndex( + memReq.memoryTypeBits, pAllocationCreateInfo, + VmaBufferImageUsage(*pImageCreateInfo), pMemoryTypeIndex); - funcs->vkDestroyImage( - hDev, hImage, allocator->GetAllocationCallbacks()); - } + funcs->vkDestroyImage( + hDev, hImage, allocator->GetAllocationCallbacks()); } return res; } @@ -15748,6 +16192,42 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory( VK_NULL_HANDLE, // dedicatedBuffer VK_NULL_HANDLE, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext + *pCreateInfo, + VMA_SUBALLOCATION_TYPE_UNKNOWN, + 1, // allocationCount + pAllocation); + + if(pAllocationInfo != VMA_NULL && result == VK_SUCCESS) + { + allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); + } + + return result; +} + +VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateDedicatedMemory( + VmaAllocator allocator, + const VkMemoryRequirements* pVkMemoryRequirements, + const VmaAllocationCreateInfo* pCreateInfo, + void* pMemoryAllocateNext, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + VMA_ASSERT(allocator && pVkMemoryRequirements && pCreateInfo && pAllocation); + + VMA_DEBUG_LOG("vmaAllocateDedicatedMemory"); + + VMA_DEBUG_GLOBAL_MUTEX_LOCK + + VkResult result = allocator->AllocateMemory( + *pVkMemoryRequirements, + true, // requiresDedicatedAllocation + false, // prefersDedicatedAllocation + VK_NULL_HANDLE, // dedicatedBuffer + VK_NULL_HANDLE, // dedicatedImage + VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + pMemoryAllocateNext, *pCreateInfo, VMA_SUBALLOCATION_TYPE_UNKNOWN, 1, // allocationCount @@ -15787,6 +16267,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryPages( VK_NULL_HANDLE, // dedicatedBuffer VK_NULL_HANDLE, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext *pCreateInfo, VMA_SUBALLOCATION_TYPE_UNKNOWN, allocationCount, @@ -15830,6 +16311,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForBuffer( buffer, // dedicatedBuffer VK_NULL_HANDLE, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext *pCreateInfo, VMA_SUBALLOCATION_TYPE_BUFFER, 1, // allocationCount @@ -15869,6 +16351,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForImage( VK_NULL_HANDLE, // dedicatedBuffer image, // dedicatedImage VmaBufferImageUsage::UNKNOWN, // dedicatedBufferImageUsage + VMA_NULL, // pMemoryAllocateNext *pCreateInfo, VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN, 1, // allocationCount @@ -16264,86 +16747,13 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer( VmaAllocationInfo* pAllocationInfo) { VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && pBuffer && pAllocation); - - if(pBufferCreateInfo->size == 0) - { - return VK_ERROR_INITIALIZATION_FAILED; - } - if((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 && - !allocator->m_UseKhrBufferDeviceAddress) - { - VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used."); - return VK_ERROR_INITIALIZATION_FAILED; - } - VMA_DEBUG_LOG("vmaCreateBuffer"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - VMA_DEBUG_GLOBAL_MUTEX_LOCK + return allocator->CreateBuffer(pBufferCreateInfo, pAllocationCreateInfo, + VMA_NULL, // pMemoryAllocateNext + pBuffer, pAllocation, pAllocationInfo); - *pBuffer = VK_NULL_HANDLE; - *pAllocation = VK_NULL_HANDLE; - - // 1. Create VkBuffer. - VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)( - allocator->m_hDevice, - pBufferCreateInfo, - allocator->GetAllocationCallbacks(), - pBuffer); - if(res >= 0) - { - // 2. vkGetBufferMemoryRequirements. - VkMemoryRequirements vkMemReq = {}; - bool requiresDedicatedAllocation = false; - bool prefersDedicatedAllocation = false; - allocator->GetBufferMemoryRequirements(*pBuffer, vkMemReq, - requiresDedicatedAllocation, prefersDedicatedAllocation); - - // 3. Allocate memory using allocator. - res = allocator->AllocateMemory( - vkMemReq, - requiresDedicatedAllocation, - prefersDedicatedAllocation, - *pBuffer, // dedicatedBuffer - VK_NULL_HANDLE, // dedicatedImage - VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), // dedicatedBufferImageUsage - *pAllocationCreateInfo, - VMA_SUBALLOCATION_TYPE_BUFFER, - 1, // allocationCount - pAllocation); - - if(res >= 0) - { - // 3. Bind buffer with memory. - if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) - { - res = allocator->BindBufferMemory(*pAllocation, 0, *pBuffer, VMA_NULL); - } - if(res >= 0) - { - // All steps succeeded. - #if VMA_STATS_STRING_ENABLED - (*pAllocation)->InitBufferUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5); - #endif - if(pAllocationInfo != VMA_NULL) - { - allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); - } - - return VK_SUCCESS; - } - allocator->FreeMemory( - 1, // allocationCount - pAllocation); - *pAllocation = VK_NULL_HANDLE; - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - return res; } VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( @@ -16356,89 +16766,36 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment( VmaAllocationInfo* pAllocationInfo) { VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && VmaIsPow2(minAlignment) && pBuffer && pAllocation); - - if(pBufferCreateInfo->size == 0) - { - return VK_ERROR_INITIALIZATION_FAILED; - } - if((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 && - !allocator->m_UseKhrBufferDeviceAddress) - { - VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used."); - return VK_ERROR_INITIALIZATION_FAILED; - } - VMA_DEBUG_LOG("vmaCreateBufferWithAlignment"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - VMA_DEBUG_GLOBAL_MUTEX_LOCK + VmaAllocationCreateInfo allocCreateInfoCopy = *pAllocationCreateInfo; + allocCreateInfoCopy.minAlignment = VMA_MAX(allocCreateInfoCopy.minAlignment, minAlignment); - *pBuffer = VK_NULL_HANDLE; - *pAllocation = VK_NULL_HANDLE; + return allocator->CreateBuffer(pBufferCreateInfo, &allocCreateInfoCopy, + VMA_NULL, // pMemoryAllocateNext + pBuffer, pAllocation, pAllocationInfo); +} - // 1. Create VkBuffer. - VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)( - allocator->m_hDevice, - pBufferCreateInfo, - allocator->GetAllocationCallbacks(), - pBuffer); - if(res >= 0) - { - // 2. vkGetBufferMemoryRequirements. - VkMemoryRequirements vkMemReq = {}; - bool requiresDedicatedAllocation = false; - bool prefersDedicatedAllocation = false; - allocator->GetBufferMemoryRequirements(*pBuffer, vkMemReq, - requiresDedicatedAllocation, prefersDedicatedAllocation); +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedBuffer( + VmaAllocator allocator, + const VkBufferCreateInfo* pBufferCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkBuffer* pBuffer, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && pBuffer && pAllocation); + VMA_DEBUG_LOG("vmaCreateDedicatedBuffer"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - // 2a. Include minAlignment - vkMemReq.alignment = VMA_MAX(vkMemReq.alignment, minAlignment); + VmaAllocationCreateInfo allocCreateInfoCopy = *pAllocationCreateInfo; + allocCreateInfoCopy.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; - // 3. Allocate memory using allocator. - res = allocator->AllocateMemory( - vkMemReq, - requiresDedicatedAllocation, - prefersDedicatedAllocation, - *pBuffer, // dedicatedBuffer - VK_NULL_HANDLE, // dedicatedImage - VmaBufferImageUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5), // dedicatedBufferImageUsage - *pAllocationCreateInfo, - VMA_SUBALLOCATION_TYPE_BUFFER, - 1, // allocationCount - pAllocation); - - if(res >= 0) - { - // 3. Bind buffer with memory. - if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) - { - res = allocator->BindBufferMemory(*pAllocation, 0, *pBuffer, VMA_NULL); - } - if(res >= 0) - { - // All steps succeeded. - #if VMA_STATS_STRING_ENABLED - (*pAllocation)->InitBufferUsage(*pBufferCreateInfo, allocator->m_UseKhrMaintenance5); - #endif - if(pAllocationInfo != VMA_NULL) - { - allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); - } - - return VK_SUCCESS; - } - allocator->FreeMemory( - 1, // allocationCount - pAllocation); - *pAllocation = VK_NULL_HANDLE; - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks()); - *pBuffer = VK_NULL_HANDLE; - return res; - } - return res; + return allocator->CreateBuffer(pBufferCreateInfo, &allocCreateInfoCopy, + pMemoryAllocateNext, // pMemoryAllocateNext + pBuffer, pAllocation, pAllocationInfo); } VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingBuffer( @@ -16534,90 +16891,37 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage( VmaAllocationInfo* pAllocationInfo) { VMA_ASSERT(allocator && pImageCreateInfo && pAllocationCreateInfo && pImage && pAllocation); - VMA_ASSERT((pImageCreateInfo->flags & VK_IMAGE_CREATE_DISJOINT_BIT_COPY) == 0 && "vmaCreateImage() doesn't support disjoint multi-planar images. Please allocate memory for the planes using vmaAllocateMemory() and bind them using vmaBindImageMemory2()."); - - if(pImageCreateInfo->extent.width == 0 || - pImageCreateInfo->extent.height == 0 || - pImageCreateInfo->extent.depth == 0 || - pImageCreateInfo->mipLevels == 0 || - pImageCreateInfo->arrayLayers == 0) - { - return VK_ERROR_INITIALIZATION_FAILED; - } - VMA_DEBUG_LOG("vmaCreateImage"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - VMA_DEBUG_GLOBAL_MUTEX_LOCK + return allocator->CreateImage(pImageCreateInfo, pAllocationCreateInfo, + VMA_NULL, // pMemoryAllocateNext + pImage, pAllocation, pAllocationInfo); +} - *pImage = VK_NULL_HANDLE; - *pAllocation = VK_NULL_HANDLE; +VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateDedicatedImage( + VmaAllocator allocator, + const VkImageCreateInfo* pImageCreateInfo, + const VmaAllocationCreateInfo* pAllocationCreateInfo, + void* pMemoryAllocateNext, + VkImage* pImage, + VmaAllocation* pAllocation, + VmaAllocationInfo* pAllocationInfo) +{ + VMA_ASSERT(allocator && pImageCreateInfo && pAllocationCreateInfo && pImage && pAllocation); + VMA_ASSERT((pImageCreateInfo->flags & VK_IMAGE_CREATE_DISJOINT_BIT_COPY) == 0 && + "vmaCreateDedicatedImage() doesn't support disjoint multi-planar images. Please allocate memory for the planes using vmaAllocateMemory() and bind them using vmaBindImageMemory2()."); + VMA_DEBUG_LOG("vmaCreateDedicatedImage"); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; - // 1. Create VkImage. - VkResult res = (*allocator->GetVulkanFunctions().vkCreateImage)( - allocator->m_hDevice, - pImageCreateInfo, - allocator->GetAllocationCallbacks(), - pImage); - if(res == VK_SUCCESS) - { - VmaSuballocationType suballocType = pImageCreateInfo->tiling == VK_IMAGE_TILING_OPTIMAL ? - VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL : - VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR; + VmaAllocationCreateInfo allocCreateInfoCopy = *pAllocationCreateInfo; + allocCreateInfoCopy.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; - // 2. Allocate memory using allocator. - VkMemoryRequirements vkMemReq = {}; - bool requiresDedicatedAllocation = false; - bool prefersDedicatedAllocation = false; - allocator->GetImageMemoryRequirements(*pImage, vkMemReq, - requiresDedicatedAllocation, prefersDedicatedAllocation); - - res = allocator->AllocateMemory( - vkMemReq, - requiresDedicatedAllocation, - prefersDedicatedAllocation, - VK_NULL_HANDLE, // dedicatedBuffer - *pImage, // dedicatedImage - VmaBufferImageUsage(*pImageCreateInfo), // dedicatedBufferImageUsage - *pAllocationCreateInfo, - suballocType, - 1, // allocationCount - pAllocation); - - if(res == VK_SUCCESS) - { - // 3. Bind image with memory. - if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0) - { - res = allocator->BindImageMemory(*pAllocation, 0, *pImage, VMA_NULL); - } - if(res == VK_SUCCESS) - { - // All steps succeeded. - #if VMA_STATS_STRING_ENABLED - (*pAllocation)->InitImageUsage(*pImageCreateInfo); - #endif - if(pAllocationInfo != VMA_NULL) - { - allocator->GetAllocationInfo(*pAllocation, pAllocationInfo); - } - - return VK_SUCCESS; - } - allocator->FreeMemory( - 1, // allocationCount - pAllocation); - *pAllocation = VK_NULL_HANDLE; - (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks()); - *pImage = VK_NULL_HANDLE; - return res; - } - (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks()); - *pImage = VK_NULL_HANDLE; - return res; - } - return res; + return allocator->CreateImage(pImageCreateInfo, &allocCreateInfoCopy, + pMemoryAllocateNext, // pMemoryAllocateNext + pImage, pAllocation, pAllocationInfo); } VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingImage( @@ -16837,7 +17141,20 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle(VmaAllocator VMA_NOT { VMA_ASSERT(allocator && allocation && pHandle); VMA_DEBUG_GLOBAL_MUTEX_LOCK; - return allocation->GetWin32Handle(allocator, hTargetProcess, pHandle); + return allocation->GetWin32Handle(allocator, VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT, hTargetProcess, pHandle); +} +VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle2(VmaAllocator VMA_NOT_NULL allocator, + VmaAllocation VMA_NOT_NULL allocation, VkExternalMemoryHandleTypeFlagBits handleType, HANDLE hTargetProcess, HANDLE* VMA_NOT_NULL pHandle) +{ + VMA_ASSERT(allocator && allocation && pHandle); + VMA_ASSERT(handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_HEAP_BIT_KHR || + handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KHR); + VMA_DEBUG_GLOBAL_MUTEX_LOCK; + return allocation->GetWin32Handle(allocator, handleType, hTargetProcess, pHandle); } #endif // VMA_EXTERNAL_MEMORY_WIN32 #endif // VMA_STATS_STRING_ENABLED @@ -16845,7 +17162,7 @@ VMA_CALL_PRE VkResult VMA_CALL_POST vmaGetMemoryWin32Handle(VmaAllocator VMA_NOT #endif // VMA_IMPLEMENTATION /** -\page faq Frequenty asked questions +\page faq Frequently asked questions What is VMA? @@ -17006,7 +17323,6 @@ Among many extensions available for Vulkan, only a few interact with memory mana VMA can automatically take advantage of them. Some of them are: VK_EXT_memory_budget, VK_EXT_memory_priority, VK_KHR_external_memory_win32, and VK_KHR_maintenance* extensions that are later promoted to the new versions of the core Vulkan API. - To use them, it is your responsibility to validate if they are available on the current system and if so, enable them while creating the Vulkan device object. You also need to pass appropriate #VmaAllocatorCreateFlagBits to inform VMA that they are enabled. @@ -17057,8 +17373,8 @@ You can access it in multiple ways: Is it a mature project? -Yes! The library is in development since June 2017, has over 1000 commits, over 400 issue tickets -and pull requests (most of them resolved), and over 70 contributors. +Yes! The library is in development since June 2017, has over 1000 commits, over 500 issue tickets +and pull requests (most of them resolved), and over 80 contributors. It is distributed together with Vulkan SDK. It is used by many software projects, including some large and popular ones like Qt or Blender, as well as some AAA games. @@ -18008,7 +18324,7 @@ Many of the common concerns can be addressed in a different way than using custo - If you want to select specific memory type for your allocation, you can set VmaAllocationCreateInfo::memoryTypeBits to `(1U << myMemoryTypeIndex)` instead. - If you need to create a buffer with certain minimum alignment, you can still do it - using default pools with dedicated function vmaCreateBufferWithAlignment(). + using default pools by specifying VmaAllocationCreateInfo::minAlignment. \section linear_algorithm Linear allocation algorithm @@ -18636,31 +18952,163 @@ individual names to allocations using vmaSetAllocationName(), can greatly aid in \page other_api_interop Interop with other graphics APIs -VMA provides some features that help with interoperability with other graphics APIs, e.g. OpenGL. +VMA provides some features that help with interoperability with other graphics APIs, e.g. OpenGL, Direct3D 11, Direct3D 12. -\section opengl_interop_exporting_memory Exporting memory +\section other_api_interop_exporting_memory Exporting memory -If you want to attach `VkExportMemoryAllocateInfoKHR` or other structure to `pNext` chain of memory allocations made by the library: +On Windows, the VK_KHR_external_memory_win32 device extension allows exporting a Win32 `HANDLE` +of a `VkDeviceMemory` block, to be able to reference the memory on other Vulkan logical devices or instances, +in multiple processes, and/or in multiple APIs. +VMA offers support for it. -You can create \ref custom_memory_pools for such allocations. -Define and fill in your `VkExportMemoryAllocateInfoKHR` structure and attach it to VmaPoolCreateInfo::pMemoryAllocateNext -while creating the custom pool. -Please note that the structure must remain alive and unchanged for the whole lifetime of the #VmaPool, -not only while creating it, as no copy of the structure is made, -but its original pointer is used for each allocation instead. +\subsection other_api_interop_exporting_initialization Initialization + +1) Make sure the extension is defined in the code by including following header before including VMA: + +\code +#include +\endcode + +2) Check if "VK_KHR_external_memory_win32" is available among device extensions. +Enable it when creating the `VkDevice` object. + +3) Enable the usage of this extension in VMA by setting flag #VMA_ALLOCATOR_CREATE_KHR_EXTERNAL_MEMORY_WIN32_BIT +when calling vmaCreateAllocator(). + +4) Make sure that VMA has access to the `vkGetMemoryWin32HandleKHR` function by either enabling `VMA_DYNAMIC_VULKAN_FUNCTIONS` macro +or setting VmaVulkanFunctions::vkGetMemoryWin32HandleKHR explicitly. +For more information, see \ref quick_start_initialization_importing_vulkan_functions. + +\subsection other_api_interop_exporting_preparations Preparations + +You can find example usage among tests, in file "Tests.cpp", function `TestWin32Handles()`. + +To use the extenion, buffers need to be created with `VkExternalMemoryBufferCreateInfoKHR` attached to their `pNext` chain, +and memory allocations need to be made with `VkExportMemoryAllocateInfoKHR` attached to their `pNext` chain. +To make use of them, you need to use \ref custom_memory_pools. Example: + +\code +constexpr VkExternalMemoryHandleTypeFlagsKHR handleType = + VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR; + +// Define an example buffer and allocation parameters. +VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { + VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, + nullptr, + handleType +}; +VkBufferCreateInfo exampleBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; +exampleBufCreateInfo.size = 0x10000; // Doesn't matter here. +exampleBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; +exampleBufCreateInfo.pNext = &externalMemBufCreateInfo; + +VmaAllocationCreateInfo exampleAllocCreateInfo = {}; +exampleAllocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO; + +// Find memory type index to use for the custom pool. +uint32_t memTypeIndex; +VkResult res = vmaFindMemoryTypeIndexForBufferInfo(g_Allocator, + &exampleBufCreateInfo, &exampleAllocCreateInfo, &memTypeIndex); +// Check res... + +// Create a custom pool. +constexpr static VkExportMemoryAllocateInfoKHR exportMemAllocInfo = { + VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR, + nullptr, + handleType +}; +VmaPoolCreateInfo poolCreateInfo = {}; +poolCreateInfo.memoryTypeIndex = memTypeIndex; +poolCreateInfo.pMemoryAllocateNext = (void*)&exportMemAllocInfo; + +VmaPool pool; +res = vmaCreatePool(g_Allocator, &poolCreateInfo, &pool); +// Check res... + +// YOUR OTHER CODE COMES HERE.... + +// At the end, don't forget to destroy it! +vmaDestroyPool(g_Allocator, pool); +\endcode + +Note that the structure passed as VmaPoolCreateInfo::pMemoryAllocateNext must remain alive and unchanged +for the whole lifetime of the custom pool, because it will be used when the pool allocates a new device memory block. +No copy is made internally. This is why variable `exportMemAllocInfo` is defined as static. If you want to export all memory allocated by VMA from certain memory types, -also dedicated allocations or other allocations made from default pools, +including dedicated allocations and allocations made from default pools, an alternative solution is to fill in VmaAllocatorCreateInfo::pTypeExternalMemoryHandleTypes. It should point to an array with `VkExternalMemoryHandleTypeFlagsKHR` to be automatically passed by the library through `VkExportMemoryAllocateInfoKHR` on each allocation made from a specific memory type. -Please note that new versions of the library also support dedicated allocations created in custom pools. - You should not mix these two methods in a way that allows to apply both to the same memory type. Otherwise, `VkExportMemoryAllocateInfoKHR` structure would be attached twice to the `pNext` chain of `VkMemoryAllocateInfo`. +\subsection other_api_interop_exporting_memory_allocation Memory allocation -\section opengl_interop_custom_alignment Custom alignment +Finally, you can create a buffer with an allocation out of the custom pool. +The buffer should use same flags as the sample buffer used to find the memory type. +It should also specify `VkExternalMemoryBufferCreateInfoKHR` in its `pNext` chain. + +\code +VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { + VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, + nullptr, + handleType +}; +VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; +bufCreateInfo.size = // Your desired buffer size. +bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; +bufCreateInfo.pNext = &externalMemBufCreateInfo; + +VmaAllocationCreateInfo allocCreateInfo = {}; +allocCreateInfo.pool = pool; // It is enough to set this one member. + +VkBuffer buf; +VmaAllocation alloc; +res = vmaCreateBuffer(g_Allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, nullptr); +// Check res... + +// YOUR OTHER CODE COMES HERE.... + +// At the end, don't forget to destroy it! +vmaDestroyBuffer(g_Allocator, buf, alloc); +\endcode + +If you need each allocation to have its own device memory block and start at offset 0, you can still do +by using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT flag. It works also with custom pools. + +Alternatively, you can use convenient functions vmaCreateDedicatedBuffer(), vmaCreateDedicatedImage() that +always allocate dedicated memory for the buffer/image created, and also allow specifying custom `pNext` chain +for the `VkMemoryAllocateInfo` structure. + +\subsection other_api_interop_exporting_exporting_win32_handle Exporting Win32 handle + +After the allocation is created, you can acquire a Win32 `HANDLE` to the `VkDeviceMemory` block it belongs to. +VMA function vmaGetMemoryWin32Handle2() is a replacement of the Vulkan function `vkGetMemoryWin32HandleKHR`. + +\code +HANDLE handle; +res = vmaGetMemoryWin32Handle2(g_Allocator, alloc, handleType, nullptr, &handle); +// Check res... + +// YOUR OTHER CODE COMES HERE.... + +// At the end, you must close the handle. +CloseHandle(handle); +\endcode + +Documentation of the VK_KHR_external_memory_win32 extension states that: + +> If handleType is defined as an NT handle, vkGetMemoryWin32HandleKHR must be called no more than once for each valid unique combination of memory and handleType. + +This is ensured automatically inside VMA. +If `VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT` is used as the handle type, +or other NT handle types, +the library fetches the handle on first use, remembers it internally, and closes it when the memory block or dedicated allocation is destroyed. +Every time you call vmaGetMemoryWin32Handle2(), VMA calls `DuplicateHandle` and returns a new handle that you need to close. +For further information, please check the documentation of this function. + +\subsection other_api_interop_exporting_custom_alignment Custom alignment Buffers or images exported to a different API like OpenGL may require a different alignment, higher than the one used by the library automatically, queried from functions like `vkGetBufferMemoryRequirements`. @@ -18672,20 +19120,59 @@ to be made out of this pool. The alignment actually used will be the maximum of this member and the alignment returned for the specific buffer or image from a function like `vkGetBufferMemoryRequirements`, which is called by VMA automatically. -If you want to create a buffer with a specific minimum alignment out of default pools, -use special function vmaCreateBufferWithAlignment(), which takes additional parameter `minAlignment`. +If you want to create a buffer/image/allocate memory with a specific minimum alignment out of default pools, +you can use VmaAllocationCreateInfo::minAlignment. Note the problem of alignment affects only resources placed inside bigger `VkDeviceMemory` blocks and not dedicated allocations, as these, by definition, always have alignment = 0 because the resource is bound to the beginning of its dedicated block. You can ensure that an allocation is created as dedicated by using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT. Contrary to Direct3D 12, Vulkan doesn't have a concept of alignment of the entire memory block passed on its allocation. -\section opengl_interop_extended_allocation_information Extended allocation information +\subsection other_api_interop_exporting_extended_allocation_information Extended allocation information If you want to rely on VMA to allocate your buffers and images inside larger memory blocks, but you need to know the size of the entire block and whether the allocation was made with its own dedicated memory, use function vmaGetAllocationInfo2() to retrieve -extended allocation information in structure #VmaAllocationInfo2. +extended allocation information in structure #VmaAllocationInfo2, which provides extra members: +`blockSize` and `dedicatedMemory`. + +\section other_api_interop_importing_memory Importing memory + +Importing external memory requires attaching an extra structure like `VkImportMemoryWin32HandleInfoKHR` +to the `pNext` chain of `VkMemoryAllocateInfo` structure. +VMA offers support for it by providing functions that allocate memory, create a buffer or an image +always with a dedicated `VkDeviceMemory` block and accept custom `pNext` pointer: +vmaAllocateDedicatedMemory(), vmaCreateDedicatedBuffer(), vmaCreateDedicatedImage(). +Example: + +\code +constexpr VkExternalMemoryHandleTypeFlagBits handleType = + VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT; + +VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { + VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR }; +externalMemBufCreateInfo.handleTypes = handleType; + +VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; +bufCreateInfo.pNext = &externalMemBufCreateInfo; // !!! +bufCreateInfo.size = ... +bufCreateInfo.usage = ... + +VkImportMemoryWin32HandleInfoKHR importInfo = { + VK_STRUCTURE_TYPE_IMPORT_MEMORY_WIN32_HANDLE_INFO_KHR }; +importInfo.handleType = handleType; +importInfo.handle = myExternalHandleToImport; + +VmaAllocationCreateInfo allocCreateInfo = {}; +allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO; + +VkBuffer buf = VK_NULL_HANDLE; +VmaAllocation alloc = VK_NULL_HANDLE; +VkResult res = vmaCreateDedicatedBuffer(allocator, &bufCreateInfo, &allocCreateInfo, + &importInfo, // pMemoryAllocateNext !!! + &buf, &alloc, nullptr); +// Check res... +\endcode @@ -18865,11 +19352,9 @@ vmaGetAllocationMemoryProperties(allocator, alloc, &memPropFlags); if(memPropFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) { - // Allocation ended up in a mappable memory and is already mapped - write to it directly. - - // [Executed in runtime]: - memcpy(allocInfo.pMappedData, myData, myDataSize); - result = vmaFlushAllocation(allocator, alloc, 0, VK_WHOLE_SIZE); + // The Allocation ended up in a mappable memory. + // Calling vmaCopyMemoryToAllocation() does vmaMapMemory(), memcpy(), vmaUnmapMemory(), and vmaFlushAllocation(). + result = vmaCopyMemoryToAllocation(allocator, myData, alloc, 0, myDataSize); // Check result... VkBufferMemoryBarrier bufMemBarrier = { VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER }; @@ -18881,6 +19366,7 @@ if(memPropFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) bufMemBarrier.offset = 0; bufMemBarrier.size = VK_WHOLE_SIZE; + // It's important to insert a buffer memory barrier here to ensure writing to the buffer has finished. vkCmdPipelineBarrier(cmdBuf, VK_PIPELINE_STAGE_HOST_BIT, VK_PIPELINE_STAGE_VERTEX_SHADER_BIT, 0, 0, nullptr, 1, &bufMemBarrier, 0, nullptr); } @@ -18903,9 +19389,8 @@ else &stagingBuf, &stagingAlloc, &stagingAllocInfo); // Check result... - // [Executed in runtime]: - memcpy(stagingAllocInfo.pMappedData, myData, myDataSize); - result = vmaFlushAllocation(allocator, stagingAlloc, 0, VK_WHOLE_SIZE); + // Calling vmaCopyMemoryToAllocation() does vmaMapMemory(), memcpy(), vmaUnmapMemory(), and vmaFlushAllocation(). + result = vmaCopyMemoryToAllocation(allocator, myData, stagingAlloc, 0, myDataSize); // Check result... VkBufferMemoryBarrier bufMemBarrier = { VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER }; @@ -18917,6 +19402,7 @@ else bufMemBarrier.offset = 0; bufMemBarrier.size = VK_WHOLE_SIZE; + // Insert a buffer memory barrier to make sure writing to the staging buffer has finished. vkCmdPipelineBarrier(cmdBuf, VK_PIPELINE_STAGE_HOST_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 1, &bufMemBarrier, 0, nullptr); @@ -18937,6 +19423,7 @@ else bufMemBarrier2.offset = 0; bufMemBarrier2.size = VK_WHOLE_SIZE; + // Make sure copying from staging buffer to the actual buffer has finished by inserting a buffer memory barrier. vkCmdPipelineBarrier(cmdBuf, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_VERTEX_SHADER_BIT, 0, 0, nullptr, 1, &bufMemBarrier2, 0, nullptr); } @@ -19241,145 +19728,6 @@ Example use of this extension can be found in the code of the sample and test su accompanying this library. -\page vk_khr_external_memory_win32 VK_KHR_external_memory_win32 - -On Windows, the VK_KHR_external_memory_win32 device extension allows exporting a Win32 `HANDLE` -of a `VkDeviceMemory` block, to be able to reference the memory on other Vulkan logical devices or instances, -in multiple processes, and/or in multiple APIs. -VMA offers support for it. - -\section vk_khr_external_memory_win32_initialization Initialization - -1) Make sure the extension is defined in the code by including following header before including VMA: - -\code -#include -\endcode - -2) Check if "VK_KHR_external_memory_win32" is available among device extensions. -Enable it when creating the `VkDevice` object. - -3) Enable the usage of this extension in VMA by setting flag #VMA_ALLOCATOR_CREATE_KHR_EXTERNAL_MEMORY_WIN32_BIT -when calling vmaCreateAllocator(). - -4) Make sure that VMA has access to the `vkGetMemoryWin32HandleKHR` function by either enabling `VMA_DYNAMIC_VULKAN_FUNCTIONS` macro -or setting VmaVulkanFunctions::vkGetMemoryWin32HandleKHR explicitly. -For more information, see \ref quick_start_initialization_importing_vulkan_functions. - -\section vk_khr_external_memory_win32_preparations Preparations - -You can find example usage among tests, in file "Tests.cpp", function `TestWin32Handles()`. - -To use the extenion, buffers need to be created with `VkExternalMemoryBufferCreateInfoKHR` attached to their `pNext` chain, -and memory allocations need to be made with `VkExportMemoryAllocateInfoKHR` attached to their `pNext` chain. -To make use of them, you need to use \ref custom_memory_pools. Example: - -\code -// Define an example buffer and allocation parameters. -VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { - VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, - nullptr, - VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT -}; -VkBufferCreateInfo exampleBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; -exampleBufCreateInfo.size = 0x10000; // Doesn't matter here. -exampleBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; -exampleBufCreateInfo.pNext = &externalMemBufCreateInfo; - -VmaAllocationCreateInfo exampleAllocCreateInfo = {}; -exampleAllocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO; - -// Find memory type index to use for the custom pool. -uint32_t memTypeIndex; -VkResult res = vmaFindMemoryTypeIndexForBufferInfo(g_Allocator, - &exampleBufCreateInfo, &exampleAllocCreateInfo, &memTypeIndex); -// Check res... - -// Create a custom pool. -constexpr static VkExportMemoryAllocateInfoKHR exportMemAllocInfo = { - VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR, - nullptr, - VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT -}; -VmaPoolCreateInfo poolCreateInfo = {}; -poolCreateInfo.memoryTypeIndex = memTypeIndex; -poolCreateInfo.pMemoryAllocateNext = (void*)&exportMemAllocInfo; - -VmaPool pool; -res = vmaCreatePool(g_Allocator, &poolCreateInfo, &pool); -// Check res... - -// YOUR OTHER CODE COMES HERE.... - -// At the end, don't forget to destroy it! -vmaDestroyPool(g_Allocator, pool); -\endcode - -Note that the structure passed as VmaPoolCreateInfo::pMemoryAllocateNext must remain alive and unchanged -for the whole lifetime of the custom pool, because it will be used when the pool allocates a new device memory block. -No copy is made internally. This is why variable `exportMemAllocInfo` is defined as `static`. - -\section vk_khr_external_memory_win32_memory_allocation Memory allocation - -Finally, you can create a buffer with an allocation out of the custom pool. -The buffer should use same flags as the sample buffer used to find the memory type. -It should also specify `VkExternalMemoryBufferCreateInfoKHR` in its `pNext` chain. - -\code -VkExternalMemoryBufferCreateInfoKHR externalMemBufCreateInfo = { - VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO_KHR, - nullptr, - VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT -}; -VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; -bufCreateInfo.size = // Your desired buffer size. -bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; -bufCreateInfo.pNext = &externalMemBufCreateInfo; - -VmaAllocationCreateInfo allocCreateInfo = {}; -allocCreateInfo.pool = pool; // It is enough to set this one member. - -VkBuffer buf; -VmaAllocation alloc; -res = vmaCreateBuffer(g_Allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, nullptr); -// Check res... - -// YOUR OTHER CODE COMES HERE.... - -// At the end, don't forget to destroy it! -vmaDestroyBuffer(g_Allocator, buf, alloc); -\endcode - -If you need each allocation to have its own device memory block and start at offset 0, you can still do -by using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT flag. It works also with custom pools. - -\section vk_khr_external_memory_win32_exporting_win32_handle Exporting Win32 handle - -After the allocation is created, you can acquire a Win32 `HANDLE` to the `VkDeviceMemory` block it belongs to. -VMA function vmaGetMemoryWin32Handle() is a replacement of the Vulkan function `vkGetMemoryWin32HandleKHR`. - -\code -HANDLE handle; -res = vmaGetMemoryWin32Handle(g_Allocator, alloc, nullptr, &handle); -// Check res... - -// YOUR OTHER CODE COMES HERE.... - -// At the end, you must close the handle. -CloseHandle(handle); -\endcode - -Documentation of the VK_KHR_external_memory_win32 extension states that: - -> If handleType is defined as an NT handle, vkGetMemoryWin32HandleKHR must be called no more than once for each valid unique combination of memory and handleType. - -This is ensured automatically inside VMA. -The library fetches the handle on first use, remembers it internally, and closes it when the memory block or dedicated allocation is destroyed. -Every time you call vmaGetMemoryWin32Handle(), VMA calls `DuplicateHandle` and returns a new handle that you need to close. - -For further information, please check documentation of the vmaGetMemoryWin32Handle() function. - - \page enabling_buffer_device_address Enabling buffer device address Device extension VK_KHR_buffer_device_address @@ -19475,9 +19823,6 @@ When using this library, you can meet following types of warnings issued by Vulkan validation layer. They don't necessarily indicate a bug, so you may need to just ignore them. -- *vkBindBufferMemory(): Binding memory to buffer 0xeb8e4 but vkGetBufferMemoryRequirements() has not been called on that buffer.* - - It happens when VK_KHR_dedicated_allocation extension is enabled. - `vkGetBufferMemoryRequirements2KHR` function is used instead, while validation layer seems to be unaware of it. - *Mapping an image with layout VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL can result in undefined behavior if this memory is used by the device. Only GENERAL or PREINITIALIZED should be used.* - It happens when you map a buffer or image, because the library maps entire `VkDeviceMemory` block, where different types of images and buffers may end