Files
ppsspp/Core/Util/PSARUnpack.h
T
Henrik RydgårdandClaude Opus 5.5 9ddfcca670 Firmware install: Fewer and parallel file operations into the staging directory
Through Android's storage framework, every exists check, create and open is a round trip to the
storage provider, and writing each file took four of them (a check of its directory, a check of
the file, the create, the open). That made a 6.61 install take around 30 s on a phone, against a
fraction of a second of decrypting and inflating.

The install unpacks into a staging directory it has just cleared, so nothing in it can exist yet.
With the new outputDirIsEmpty option, each directory is created once and each file is created and
opened without checking for it first (File::WriteDataToNewFile), on the IO threads while decoding
goes on. A name that turns up twice still goes the careful way, overwriting the first. The swap
into place at the end is unchanged.

Headless --unpack-updater uses the same path when the output directory doesn't exist yet. Its
output is identical either way. With SIMULATE_SLOW_IO, unpacking the fonts went from 17 s to 7.5 s,
most of what's left being the fixed cost of opening the updater.

The UI's install already runs on a thread pool IO thread, so writes queued on the same pool
could stall behind other long IO tasks, and the pool drops queued tasks unrun at teardown,
which would have left the install waiting forever on its in-flight count. Eight threads owned by
the writer (attached for JNI) avoid both, and headless no longer has to start the pool for it.

Also compare file names case-insensitively when looking for repeats, since the storage usually
is case-insensitive and a second create there makes a renamed copy.

Checked with ThreadSanitizer, and on a read-only destination (every file fails, no hang).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 16:06:47 +02:00

216 lines
11 KiB
C++

// Copyright (c) 2026- PPSSPP Project.
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, version 2.0 or later versions.
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License 2.0 for more details.
// A copy of the GPL 2.0 should have been included with the program.
// If not, see http://www.gnu.org/licenses/
// Official git repository and contact information can be found at
// https://github.com/hrydgard/ppsspp and http://www.ppsspp.org/.
#pragma once
#include <ctime>
#include <functional>
#include <string>
#include <string_view>
#include <vector>
#include "Common/CommonTypes.h"
class Path;
class IFileSystem;
// Unpacks the firmware image inside an official PSP updater (PSP/GAME/UPDATE/EBOOT.PBP).
//
// The updater's DATA.PSAR is a flat sequence of encrypted, compressed file records. The real
// updater installs them by executing on the PSP, but the archive can be walked directly - each
// record carries its own name, sizes and a standard PRX-style encryption header, so all it needs
// is the KIRK engine and the PRX decrypter we already have.
//
// The main use is pulling flash0:/font out of an updater the user supplies, which is why a
// prefix filter is part of the API rather than something the caller filters afterwards.
enum class PSARCompression {
None,
Zlib,
KL4E,
KL3E,
LZR,
Unknown,
};
const char *PSARCompressionToString(PSARCompression c);
// PSP hardware revisions, as the updater numbers them. An updater carries one file list per
// model, so which one you resolve names against decides both what a file is called and whether
// it's part of that model's firmware at all.
enum class PSPModelGeneration {
Any = 0, // Whichever list names a file first. Use this to extract everything.
PSP_1000 = 1,
PSP_2000 = 2,
PSP_3000 = 3,
PSP_4000 = 4,
PSP_N1000 = 5, // PSP Go
PSP_6000 = 6,
PSP_7000 = 7,
PSP_9000 = 9,
PSP_11000 = 11,
MAX = 12,
};
const char *PSPModelGenerationToString(PSPModelGeneration generation);
// The model we're claiming to be. An updater carries one file list per hardware revision, and
// anything the chosen model's list doesn't name isn't part of its firmware, so unpacking this
// one keeps flash0 consistent with what the emulator reports to games.
PSPModelGeneration EmulatedModelGeneration();
// Accepts "01g".."12g", a bare number, or "any". Returns false if it's none of those.
bool PSPModelGenerationFromString(std::string_view name, PSPModelGeneration *generation);
struct PSARUnpackOptions {
// Only unpack entries whose name starts with this, e.g. "flash0:/font/". Case insensitive.
// Empty means everything. Entries whose real name we can't recover are never matched by a
// non-empty filter.
std::string prefixFilter;
// Which model's file list to resolve names against. Anything that model's list doesn't name
// isn't part of its firmware, and is skipped.
PSPModelGeneration model = PSPModelGeneration::Any;
// Walk and report, but don't write any files.
bool listOnly = false;
// Log a line per entry. Off by default - an updater holds well over a thousand of them.
bool verbose = false;
// Called on the unpacking thread after every entry, with how far through the archive we are
// (0.0 to 1.0). Unpacking a full firmware takes a while, so a UI wants this.
std::function<void(float)> progress;
// outputDir was just created and is empty, so nothing in it can exist yet. Files are then
// created without checking for them first, and written on several threads. Through Android's
// storage framework every check and create is a slow round trip, so this saves most of the time.
bool outputDirIsEmpty = false;
};
struct PSARUnpackStats {
std::string firmwareVersion;
int entries = 0;
int directories = 0;
int written = 0;
int skippedByFilter = 0;
int nameTables = 0; // Entries that were file lists rather than files.
int unnamed = 0; // Short names no file list claimed.
int otherModel = 0; // Files that belong to a model other than the requested one.
int failed = 0;
// How many entries used each compression, indexed by PSARCompression.
int compressionCounts[6]{};
};
// psar points at the DATA.PSAR contents, starting with the "PSAR" magic.
bool UnpackPSAR(const u8 *psar, size_t psarSize, const Path &outputDir, const PSARUnpackOptions &options, PSARUnpackStats *stats, std::string *error);
// Finds the firmware image in whatever an updater arrives as and unpacks it:
// - a downloaded updater EBOOT.PBP, where the archive is its DATA.PSAR
// - a disc updater's DATA.BIN, which is the same archive without the PBP wrapper
// - a game ISO/CSO/CHD, which is searched for PSP_GAME/SYSDIR/UPDATE/DATA.BIN
// The last one is the interesting case: most UMDs carry a firmware updater, so the fonts can come
// from whatever game the user already has rather than a separate download.
bool UnpackUpdater(const Path &filename, const Path &outputDir, const PSARUnpackOptions &options, PSARUnpackStats *stats, std::string *error);
// What a game disc's bundled firmware updater says about itself. All of this comes from the
// PARAM.SFO and the directory entry next to the archive, so gathering it costs a couple of small
// reads - no decryption, and the archive itself is never touched.
struct BundledUpdateInfo {
bool present = false;
std::string version; // "6.61". Can be empty even when present, if the SFO is unreadable.
std::string title; // The updater's full SFO title, e.g. "PSP(tm) Update ver 6.61".
s64 archiveSize = 0; // Size of DATA.BIN, i.e. how much firmware is in there.
// When DATA.BIN was written, as Unix UTC seconds. 0 if the disc doesn't record one, which
// is normal for the shapes that aren't really an ISO.
s64 mtime = 0;
// "6.61 (2011-01-25)", or just the version if there's no date. Empty if there's no updater.
std::string Describe() const;
};
// Reads the above out of a disc that's already open, whether that's an ISOFileSystem the caller
// mounted or the running game's disc0:. pathPrefix is what to stick in front of "PSP_GAME/..." -
// "/" for a freshly mounted image, "disc0:/" for the meta file system.
bool ReadBundledUpdateInfo(IFileSystem *fs, std::string_view pathPrefix, BundledUpdateInfo *info);
// The version string an updater advertises ("6.61"), read from the PARAM.SFO next to it - no
// decryption needed, so it's cheap enough to check every disc with. Empty if there's no updater.
std::string ReadUpdaterVersion(const Path &filename);
// Pulls the version out of an updater's SFO title: "PSP(tm) Update ver 3.95" -> "3.95".
std::string VersionFromUpdaterTitle(std::string_view title);
// What's actually in the NAND directory right now. That can be anything from a handful of fonts
// we pulled off a game disc to a full firmware unpacked from an updater, so this reports what's
// there rather than assuming one or the other.
struct InstalledFirmwareInfo {
bool anythingInstalled = false; // flash0/flash1 exist and hold at least one file.
// From flash0:/vsh/etc/version.txt, which only a full firmware install has. Empty if all
// that's there is a partial install like the fonts.
std::string version; // "6.60"
std::string buildDate; // "2011-07-27". Empty if the file doesn't spell one out.
std::string target; // "WorldWide"
bool hasVsh = false; // flash0:/vsh/module/vshmain.prx - what launching the XMB needs.
int fontCount = 0; // Files in flash0:/font, which is all sceFont wants.
int kernelModuleCount = 0; // Files in flash0:/kd, which is what --disable-hle wants.
int fileCount = 0;
u64 totalSize = 0;
};
// Walks the NAND directory (the one holding flash0/flash1). Pass countFiles = false when
// fileCount and totalSize aren't wanted: those are the only fields that need the whole tree read,
// and a full firmware is a few hundred files, which isn't free on a phone's storage.
void ReadInstalledFirmwareInfo(const Path &nandRoot, InstalledFirmwareInfo *info, bool countFiles = true);
// Wipes what's in the NAND directory: flash0, flash1 and ipl. Two firmwares can't be merged -
// files a newer one dropped would linger and still get loaded - so an install starts from empty.
// Deliberately leaves an install's staging directory alone: InstallFirmware() calls this with a
// finished firmware sitting in there, waiting to be moved into the space this just cleared.
bool EraseInstalledFirmware(const Path &nandRoot, std::string *error);
// "6.61" -> 661. Sony writes the minor part with two digits, but a single-digit one is still a
// tens value, so "5.5" is 550 and not 505. Returns 0 if it isn't a version string at all, which
// makes the result safe to compare with: an unknown version is older than every real one.
int FirmwareVersionToInt(std::string_view version);
// The firmware versions we can actually boot the VSH (XMB) on. Every other version loads, but
// the module patches it needs are version-specific, so it won't get anywhere.
bool FirmwareVersionSupportsVSH(std::string_view version);
// Installs a firmware into the NAND directory. Two firmwares can't be merged - a file the new one
// doesn't have would linger and still get loaded - so this replaces rather than overlays.
//
// The unpack goes to a staging directory inside the NAND root first, and what's installed is only
// erased once the new firmware is complete on disk, so a failure of any kind leaves the existing
// one untouched. The cost is needing room for both at once. "Complete" is strict: a single entry
// that didn't unpack fails the whole install, because a firmware with holes in it still looks
// installed and would never be replaced.
//
// updater is an updater EBOOT.PBP, a disc image, or the folder holding one; pass an empty path to
// install from the disc mounted as disc0:, i.e. the running game's own disc.
bool InstallFirmware(const Path &updater, const Path &nandRoot, const PSARUnpackOptions &options, PSARUnpackStats *stats, std::string *error);
// Called while a game boots, with its disc mounted as disc0:. Most UMDs carry a firmware updater,
// and having a real firmware is what the LLE modules want - so if the disc's is newer than what's
// installed, or nothing identifiable is installed at all, unpack it. Shows progress on the OSD.
// Does nothing unless g_Config.bAutoUpgradeFirmware is set. Returns true if it installed one.
bool AutoInstallFirmwareFromDisc();
// The same three, for the disc mounted as disc0: - i.e. the game that's running. These read
// through the mounted filesystem instead of opening the image a second time, which also means
// they work for the shapes that aren't an image at all, like a folder-based "disc".
//
// This is the path the font extraction is meant to take: while a game is running, ask whether its
// disc carries an updater, and if so unpack just flash0:/font out of it.
bool MountedDiscHasUpdater();
std::string ReadMountedDiscUpdaterVersion();
bool UnpackUpdaterFromMountedDisc(const Path &outputDir, const PSARUnpackOptions &options, PSARUnpackStats *stats, std::string *error);