Savegame file format - XCOM:EU 2012

From Nexus Mods Wiki
Jump to: navigation, search


Overview

This information is derived from the Nexus Forum Save game file format? thread.

This article is concerned with what has been discovered about the format of the save game files. It provides the foundation for mods or tools designed to make changes to such files.

There is always considerable interest in being able to modify the files containing the current game state when saved to disk. Such files are referred to generally as save game files, and usually have either a player defined name or a game timestamp as part of the file name. Some games have multiple files associated with each save, but XCOM only has one. These files are saved in the user account's ...\Users\<UserName>\Documents\My Games\XCOM - Enemy Unknown\XComGame\Saves folder.

The format contains serialized data.

This type of data is currently being explored in the Nexus Forum threads:

MAINTAINERS: Once these explorations are reasonably completed, this article should be updated.

Programs and Tools

Details

The save file consists of:

  • Header - 1024 bytes
  • Compressed data - variable size

Compressed data are organized into numerous compressed chunks (N FCompressedChunk). There is no compressed chunks list (FCompressedChunksList) containing uncompressed/compressed size and offset of every FCompressedChunk. There is only one compressed block (FCompressedBlock) per chunk.

FCompressedChunk structure

The FCompressedChunk structure looks like this: 

FCompressedChunk
   - FCompressedChunkHeader     // 12 bytes
   - FCompressedBlock
       - FCompressedBlockHeader // 8 bytes
       - Raw compressed data    // variable size

Summarized, each FCompressedChunk looks like this: 

   FCompressedChunkHeader - 12 bytes
   FCompressedBlockHeader - 8 bytes
   Raw compressed data - variable size

An example: 

// FCompressedChunkHeader - 12 bytes
// Number of blocks = rounded up (uncompressed size / block size)
// Block size seems to be always 0x20000 (131072)
C1 83 2A 9E    4 bytes UE3 package signature (magic)
00 00 02 00    4 bytes Block size
3B 63 00 00    4 bytes Compressed size
00 00 02 00    4 bytes Uncompressed size

// FCompressedBlockHeader - 8 bytes
3B 63 00 00    4 bytes Compressed size
00 00 02 00    4 bytes Uncompressed size

// Raw compressed data - variable size

Compression types

The method of compression used depends upon the platform on which the game was "cooked" for optimization.

  • For PC/Xbox they used LZO1X_1 compression.
    (Guildor's UPK Decompressor supports this, but you have to append a parameter to the command line. Read Eliot's How to decompress packages article on how to set this up as a shortcut link.)
  • For Apple's iOS mobile platforms (derived from OS X), they used zlib.

Source: user carpi at 360haven.com - saves structure, compression and checksum.

Checksums

Saves are protected by CRC checksums, that must be updated when repacking a file. However, the PC version has an additional CRC, not present in the iOS file: one is calculated from the compressed data and the 2nd is generated from the header. The hash is generated using the CRC32b protocol. (See entries in #Separate Content.)

The save file CRC check is at 0xBE offset and it is the same customized CRC32b for iOS as for the Windows version. No discernable difference between PC and iOS versions; the savefile structure is exactly the same. Source: user carpi at 360haven.com - checksum.

Calculate CRC from all of the compressed data, which is from offset 1024 to EOF() and put it just after the game language code. In the example, it’s at 0xEA. To calculate the correct position it’s necessary to work through the save header variable. Sum of the first 3 [bytes] in red (it’s just the size of the string) plus 12 bytes. The 4th byte is for DLC: if you don’t have any it will be 0, but anything else it’s the string size. And finally add 8 to the results of the sum and you are in correct place for the 1st CRC location.

The 2nd CRC has a static offset, so it’s not needed to do a similar operation as for 1st one. The offset is 0x3FC. 4 bytes earlier, at 0x3F8 is the size of the block from beginning of the save, which will be used for generating the CRC32b checksum. Source: user obione at 360haven.com - confirming xcom-fix.exe works on PC.

Separate Content

CRCs

See Wikipedia article on Cyclic redundancy check on how CRCs are calculated. In particular, note the Commonly used and standardized CRCs table for particular polynomial protocols used by various algorithms in programs and utilities.

The CRC32b checksum appears to have been invented for the PHP preprocessor for HTML. It's a reversal of the official CRC-32 standard hash output. There is an web-based "text to CRC32b" app Online CRC32B calculator that can be used to test this by comparing hashes of the Perl test string "Hello world!" against the hash PHP/Web says it's CRC32b protocol should produce. See CRC32 vs. CRC32B for details.

CheckpointRecord

On the UPK side, it appears that the various structures that begin with "CheckpointRecord" control what goes into the save file. (See the article UPK File Format - XCOM:EU 2012 for this structure information.)

For an ammo mod it was necessary to alter which variables in XGWeapon were saved/loaded to/from the savefile. For various reasons it was necessary to use a new variable to store the ammo cost -- m_iTurnFired. Eventually it was recognized that this was not being preserved when quitting and reloading within a tactical mission. The vanilla CheckpointRecord for XGWeapon was : 

struct CheckpointRecord_XGWeapon extends CheckpointRecord_XGInventoryItem
{
   var int iAmmo;
   var int iOverheatChance;
};

The struct hex was altered so that now it's: 

struct CheckpointRecord_XGWeapon extends CheckpointRecord_XGInventoryItem
{
   var int iAmmo;
   var int m_iTurnFired;
};

and m_iTurnFired is now saved (allowing quitting and restarting during a tactical mission without weapons all automatically reloading.

Presumably these structures are being serialized into the savegame file -- see wghost's articles/documents to get a handle on serialization. (References in the UPK File Format - XCOM:EU 2012 article.) The easiest way to find all of the Checkpoint Records is via a search for "struct CheckpointRecord" in UE Explorer.

As an example, XGStrategySoldier has : 

struct CheckpointRecord
{
    var TCharacter m_kChar;
    var TSoldier m_kSoldier;
    var int m_aStatModifiers[ECharacterStat];
    var XGStrategySoldier.ESoldierStatus m_eStatus;
    var int m_iHQLocation;
    var int m_iEnergy;
    var int m_iTurnsOut;
    var int m_iNumMissions;
    var string m_strKIAReport;
    var string m_strKIADate;
    var string m_strCauseOfDeath;
    var bool m_bPsiTested;
    var bool bForcePsiGift;
    var bool m_bMIA;
    var bool m_bAllIn;
    var TInventory m_kBackedUpLoadout;
    var XGCustomizeUI.EEasterEggCharacter m_eEasterEggChar;
    var array<XComGame.XGTacticalGameCoreNativeBase.EPerkType> m_arrRandomPerks;
    var int m_arrMedals[EMedalType];
    var bool m_bBlueShirt;
};

(omitted the defaultProperties for the struct.)

Game loading appears (at least in part) to use the state LoadGameAsync in UILoadGame: 

simulated state LoadGameAsync
{
   J0x00:    // End:0x191 [Loop If]
   if(((XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).CheckpointIsSerializing || XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).CheckpointIsWritingToDisk) || XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).ProfileIsSerializing) || XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).StorageWriteCooldownTimer > float(0))
   {
       Sleep(0.10);
       // [Loop Continue]
       goto J0x00;
   }
   XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).LoadGame(m_iLoadGameAsync, ReadSaveGameComplete);
   GotoState('None');
   stop;            
}

Note the timer loop to ensure that the Checkpoint isn't still serializing or writing to disk before LoadGame can be called.

Unfortunately both LoadGame and SaveGame in XComOnlineEventMgr are native functions, so their details are not known.

CreateCheckpointRecord

Apparently used as a hook to be able to take class-specific actions when called just before the game saves your object. Needs investigation.

ApplyCheckpointRecord

Apparently used as a hook to be able to take class-specific actions when called just after the game restores your object. Needs investigation.

Data Structures

  • The following has information on investigations into Properties structures: Fog.Gene's XCOM EU/EW Save Structure
  • Klenor277 has created a program that extracts Roster data from an In Base game save (Does not work on a save generated while in a mission yet. I think know why it doesn't work but I'm not ready to tackle that problem yet.) Eventually the program will be a way to view your soldiers and sort, filter, etc. It is still WIP but others works with a Long War save file and others may find the source code helpful for their own purposed. XCOM 2012 Soldier Viewer

Adding New Checkpoint Records

New checkpoint structs can be added to new classes that derive from Actor that contain a CheckpointRecord struct.

The class XComGame.Checkpoint is the master data structure for determining which objects get recorded in and loaded from the save file through the following fields: 

var const array<class<Actor>> ActorClassesToRecord;
var const array<class<Actor>> ActorClassesNotToDestroy;
var const array<class<Actor>> ActorClassesToDestroy;

The field ActorClassesToRecord is a dynamic array of class names of all classes to store in the saved game. It is currently not well understood what the other fields represent, although several vanilla classes are listed in both ActorClassesToRecord and ActorClassesToDestroy.

The Checkpoint class defines no classes to save itself, and is a static class that has no instances. Several subclasses define the list of classes to save for the different game modes: Checkpoint_StrategyGame for the strategy layer, Checkpoint_TacticalGame for the tactical layer, and Checkpoint_StrategyTransport for the "transport" save that transfers data between the two.

New entries can be added to the default properties for these subclasses, assuming appropriate names are present in the import/export tables of the UPK files containing those subclasses. Alternatively, the ActorClassesToRecord array can be modded to be exposed to a config file and the list of classes can be specified in a .ini file. This is done by the CustomCheckpoint mod.


References

Referred to by this article:

That refer to this article:

 

Retrieved from "https://wiki.nexusmods.com/index.php?title=Savegame_file_format_-_XCOM:EU_2012&oldid=42956"