Modifying Textures without TexMod - XCOM:EU 2012
Overview
The following is based upon the work of wghost81 (aka Wasteland Ghost, developer of the UPKTools and a principle community investigator of XCOM code) in the thread Texture modding without texmod as the result of work previously started and documented in the Modifying Textures - XCOM:EU 2012 article. It combines research from that project and lessons learned in another unrelated project. Apparently, "5 years away gives you a broad enough perspective to realize what you've been doing wrong."
Programs and Tools
- UPKUtils: Github repository with updated tools required for this purpose.
- CustomTFC component
- ExportTexturesToDDS component
- ImportTexturesFromDDS component
- ExportTexturesToDDS: wghost81`s tutorial on how to mod the textures without TexMod, with links to the necessary tools.
Details
First off, Texture2D export data format.
UE3 Texture2D object inheritance looks like this:
Object <- Surface <- Texture <- Texture2D
Its serialized data starts with PrevObjectRef (same as everything else) followed by DefaultPropertiesList. (See Modding with the UDK - XCOM:EU 2012.) Then comes its own data as follows:
an empty bulk data chunk (12 zero bytes + 4 bytes of absolute file offset pointing to the MipMapCount) MipMapCount (4 bytes) serialized MipMaps array of MipMapCount elements unknown data
"Unknown data" seems to be just a bunch of memory variables someone had forgotten to exclude from serialization.
MipMapCount should correspond to MipTailBaseIdx property defined in DefaultPropertiesList:
MipTailBaseIdx = MipMapCount - 1
MipMaps are BulkData with 8 additional bytes of SizeX and SizeY:
4 bytes of Flags 4 bytes of ElementCount 4 bytes of BulkDataSizeOnDisk 4 bytes of BulkDataOffsetInFile BulkDataSizeOnDisk bytes of image data (if Flags do not have StoredInSeparateFile or StoredAsSeparateData bit set) 4 bytes of SizeX 4 bytes of SizeY
Possible Flags:
StoredInSeparateFile = 0x00000001 StoredAsSeparateData = 0x00000040 EmptyData = 0x00000020 CompressedZlib = 0x00000002 CompressedLzo = 0x00000010 CompressedLzx = 0x00000080
If StoredInSeparateFile bit is set, the TextureFileCacheName property defined in DefaultPropertiesList is used to obtain the name of the tfc file (texture cache file) - usually it's 'Textures'. Furthermore, externally stored mipmaps are compressed (usually with "lzo"), so expect CompressedLzo bit being set too. For such mipmaps ElementCount corresponds to uncompressed size, BulkDataSizeOnDisk corresponds to compressed size and BulkDataOffsetInFile corresponds to Textures.tfc absolute offset. SizeX and SizeY contain corresponding image dimensions.
If StoredAsSeparateData bit is set, the image data are stored in the same package, but outside of this object serialized data.
If EmptyData bit is set, both BulkDataSizeOnDisk and BulkDataOffsetInFile are set to 0xFFFFFFFF and ElementCount is set to zero.
There are three interesting parameters defined in DefaultPropertiesList: Format, LODBias, and NeverStream. Format is always present and defines texture block compression method. XCOM mostly uses BC1 and BC3 (DXT1/DXT5). LODBias is... well... LOD bias. It defines the number of textures to drop, effectively allowing to limit texture resolution. More info on textures and their properties can be found in Unreal Engine: Texture Support and Settings and Unreal Engine: Material Basics. For modding purposes it's important to know that for, say, 2k texture you should have 12 mips defined, but LODBias set to 2 will limit the resolution to 512 and cause the first two mips to be dropped (the corresponding BulkData will be empty). Adding those mips back won't do anything unless you also mod LODBias to zero. Be careful here, though, as the engine calculates the final LODBias as a sum of several parameters (see the links above), so do not assume the LODBias value based on Texture2D properties only! NeverStream boolean set to true forbids the engine from streaming the texture out of memory. Mipmaps for such textures are usually fully embedded in a startup package (all the mipmaps for all the LOD levels).
So, with this knowledge in mind, what's the problem of modding textures? Well, it's how the cooking is done.
The most frequently used textures are not streamable and embedded into just one startup package. Those are easy enough to mod by just replacing their mipmap data. But lots of textures (map tiles, for example) end up being cooked into a dozen of different packages with low-res mips being embedded directly (usually up to 512) and hi-res ones being kept in an external tfc (Textures.tfc). It's done to speed up the loading process. So if we want to alter such a texture, we need to alter all the packages it resides in, plus its Textures.tfc data. Which is... quite a task at a first glance.
And this is where we made a mistake 5 years ago: making a PatcherGUI patch file to mod just one texture is a pure pain and it becomes unrealistic if we want to change dozens of textures. And no, UPK lists won't help here. What we need is a completely new tool.
On the paper the process is simple enough: take an existing texture, export it to a DDS file with all the mipmaps, edit and re-import. We can move/resize export object data if needed, we can change default properties, we can even alter the TextureFileCacheName to point at something else (Texture2D is an obvious choice as the property is a NameProperty and thus has to be present in the name table). But we also need to be sure we do it for all the packages that use this texture.
ExportTexturesToDDS can export individual textures or the whole game into dds file(s) creating an inventory file along the way. The inventory file is a simple csv (comma-separated values) file that holds the full names of all the textures found with the list of packages they were found in. Using this list another tool, ImportTexturesFromDDS, can take all the textures you feed it and import them back into all the corresponding packages.
Since we can potentially go to higher resolutions, re-importing textures into the existing "Textures.tfc" doesn't seem to be the way here. Using a new custom tfc file is a better solution. But the problem is that we're working with cooked packages and while modding and testing stuff we're doing it one texture at a time, which complicates things due to "Textures.tfc" being just a bunch of "dumb" data with no additional information on what they belong to. To address the issue it was necessary to come up with a custom tfc format that stores metadata on embedded mipmaps in a separate field.
Vanilla "Textures.tfc" format:
C1 83 2A 9E "magic" followed by lzo compressed blocks alignment bytes (zero-padding, not used by xcom, used by batman) C1 83 2A 9E "magic" ... etc
Custom Texture2D.tfc format:
2B 42 91 53 "magic" 4 bytes of BlockOffset 4 bytes of BlockSize 4 bytes of NextBlockOffset a list of inventory entries C1 83 2A 9E "magic" followed by lzo compressed blocks ...
Each inventory entry has the following data:
4 bytes of BulkDataSizeOnDisk 4 bytes of BulkDataOffsetInFile 4 bytes of ObjectNameLength ObjectNameLength bytes of FullObjectName
Max block size is 0x8000. When the limit is reached, the tool makes another block and writes its offset to the NextBlockOffset field of the previous block.
The resulting macro-structure of a "custom tfc" looks like this:
[metadata block #1] [compressed mipmap #1] [compressed mipmap #2] ... [metadata block #2] [compressed mipmap #N]...
Metadata are ignored by the game itself, but are used by the importer to avoid hi-res texture duplication when importing a new texture on a per-texture per-package basis.
References
Referred to by this article:
- Texture modding without texmod tutorial.
- Modifying Textures - XCOM:EU 2012
- Modding with the UDK - XCOM:EU 2012
- Microsoft: texture block compression method
- UnrealEngine: Texture Support and Settings
- Unreal Engine: Material Basics
That refer to this article:
