PrxDecrypter: require a whole header before decrypting

Every decrypt type reads the tag at 0xD0, the compressed size at 0xB0 and key
data as far as 0x150, and writes a KIRK header into outbuf at a fixed offset
derived from sizeof(PSP_Header) - all without checking that either buffer is
that big. A PRX declaring a tiny psp_size therefore read past the end of its
input, wrote a 0xE0-byte header past the end of an equally tiny output buffer,
and handed KIRK "size - offset" as an unsigned underflow. The header write sits
behind the SHA-1 check, but the tag keys are compiled in and every hashed input
comes from the file, so that's arithmetic rather than luck. One size check at
the top of pspDecryptPRX covers all five types.

The module loader needed two things to go with it. Its "maybe it just isn't
encrypted" fallback checked for ELF magic at 0x150 of the *output* buffer, which
on the paths where decryption bails early has nothing written to it yet - so it
read uninitialized heap to decide, and then, if psp_size was under 0x150,
memcpy'd a negative length. It reads the input buffer now, which is what it goes
on to copy from anyway, and only when psp_size is big enough to hold what's
being tested.

Second, the returned size is just comp_size out of the file header, checked
against the allocated buffer by a _dbg_assert_ that isn't there in release. That
check is a real one now, folded into the existing sanity test next to it.

pspautotests cpu 11/11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GZq8ZtJmFY7bkX5FVkr3P9

Core/HLE/sceKernelModule.cpp

@@ -1159,15 +1159,19 @@ static PSPModule *__KernelLoadELFFromPtr(const u8 *ptr, size_t elfSize, u32 load
 		elfSize = maxElfSize;
 		ptr = newptr;
 		int decryptedSize = pspDecryptPRX(in, (u8*)ptr, head->psp_size);
-		_dbg_assert_(decryptedSize <= (int)maxElfSize);
-		if (decryptedSize <= 0 && Read32(ptr + 0x150) == ELF_MAGIC) {
+		// If decryption got us nowhere, the PRX may simply not be encrypted - in which case the ELF
+		// starts right after the header. Check the source buffer, not the destination: on the paths
+		// where decryption bails early nothing has been written to newptr yet, so this used to read
+		// uninitialized heap to decide. psp_size is known to be <= the data we actually have.
+		if (decryptedSize <= 0 && head->psp_size >= 0x150 + sizeof(u32) && Read32(in + 0x150) == ELF_MAGIC) {
 			decryptedSize = head->psp_size - 0x150;
 			memcpy(newptr, in + 0x150, decryptedSize);
 			// In this case it's definitely not compressed. Added assert below.
 		}
 
-		// Don't accept ELFs over 24MB - nor ones with negative size, of course.
-		if (decryptedSize < 0 || decryptedSize > 24 * 1024 * 1024) {
+		// Don't accept ELFs over 24MB, ones bigger than the buffer we allocated for them - nor ones
+		// with negative size, of course.
+		if (decryptedSize < 0 || decryptedSize > 24 * 1024 * 1024 || decryptedSize > (int)maxElfSize) {
 			*error_string = StringFromFormat("ELF/PRX corrupt, unreasonable decrypted size: %d", (u32)decryptedSize);
 			// TODO: Might be the wrong error code.
 			error = SCE_KERNEL_ERROR_FILEERR;