How to make an NPC skin material support skin color customization
Original post on the Nexus Mods Forums.
Information about customizable skin and equip color
CMM tex defines what parts of a texture are color customizable. Usually red or green. This is usually for equipment. The game decides which color is customizable on a piece of equipment. I suspect this is defined in common/equip_scolor.esvc
For skin, skin materials have a shader switch in the mrl3 file material index, that tells the game they are a Player or NPC skin material. Skin materials usually refer to default or common player skin textures. There are also hand textures. In the case of NPC clothing, sometimes the skin is drawn onto the clothes texture. But even in the case of a single NPC texture having both clothing and skin, the skin is assigned its own material.
Let's say you have skin materials that aren't changing color. You need to figure out which these are, in the mrl3 file. "default_tex\skin_BM" seems to be used to change the skin color. The key to finding the skin materials is searching for the materials that use this texture. You can do this by counting the number order of textures and then doing a search for materials using that number.
For example, if "default_tex\skin_BM" is 13th in the texture list, its value is 0D. Search for 0D after the material index. You can figure out where materials start and how long they are, using the mrl3 image I uploaded. You can also figure out which material is which by what other textures are being used within the same material (they're in the same row). For example, hand textures being used means it's the material used for the hand.
Note that the bytes for location and size are in reverse. For example, in the top right of the mrl3 image, 4002 is the length. That's 0240 in hex. Write 0240 into Windows calculator (in hexadecimal mode) and then convert to decimal. The result is 576 bytes. Thus the material size is 576 bytes. Similarly, if location is 1280, that means it starts at 0x00001280.
Once you figure out which materials are skin materials, simply copy paste the green "skin ID" section from the confirmed skin material of a player mrl3.
Using the image I've uploaded, you should be also able to do things like append materials to the end of the mrl3 file and then edit address and size references for existing materials to point to the appended material. Note that you must keep the material IDs (I'm assuming that's what they are based on a video of a RE6 mod tool by MarioKart64n) so that the model knows which material you are referring to. The game will probably crash otherwise.
I hope my explanation makes sense (I'm tired).
Setting up material skins
1. Figure out which material is which. There are two ways to do this.
Method A: Look at the list of textures, figure out which numbers they are, then do a search for materials using those textures. For example, if you want to know which is the skin. Look for the textures skin_BM, skin_FM. They are usually the 9th, 10th or 11th materials. So, in Hexadecimal, that would be 09, 0A or 0B. Then you search for those bytes, and you should find a material referring to those bytes in a vertical column (center of each line).
Method B: Try changing the first few materials of each material texture entry to an obvious texture, like the black null textures. If you do this and reload the material in-game, that material should turn black. You can also edit the float value under the material texture refs, but that's more of a pain.
To reload quickly, either equip Layered Armor, then cancel, or go to workshop forge, preview and then unpreview.
2. Find a player model material that you want to copy paste over. Preferably one that matches the material you're replacing. For example, if it has emission, you will need to find a material that you can still plug in emission texture ref bytes.
You need both the index data and the material data for that material. Copy paste them to separate files (easier to refer to).
3. In the copy pasted material index, you will need to copy paste the 9th to 41th byte (32 bytes). Within those bytes are, in this order, the material ID ref (9th to 16th byte), material size byte (4 bytes, eg. 40 02 = 240 = 576 in decimal = 576 byte material), some unknown bytes, the material type identifier, which usually looks like "80 60 08".
The material type identifier is important in defining whether the color is customizable and can affect flickering (particularly the 3rd byte). The way the material is setup also affects flickering (which is why you need to copy paste the matierla). If you're copy pasting from a good player material, it should automatically have a good material type identifier that does not flicker.
Do not change the address references at the end of the material index data, unless you add new materials to the end of the file.
4. You need to figure out which textures are what numbers, for both the player materal, and the NPC material you're replacing. This is because the number order differs between materials. For example, skin_BM may be 0A in the player file, but 0D in the NPC file. Write a conversion table for these values, like this:
0A = 0D
05 = 0B
01 = 04
5. In the copy pasted player material data, change the texture bytes (middle of each line, it's the last byte before many 00 bytes, it's obvious) to match the target mrl3's texture numbers. For skin materials of matching size, it's usually the same order.
Material texture refs seem to often have similar order. First line is usually black or CMM, 2nd is usually the diffuse texture, and 3rd is usually normal map. Then 4th is usually black again. Note that even if you try to insert CMMs into materials that don't normally use CMM, it doesn't mean it'll treat it as a custom color map, presumably because of either the material values after the material texture refs, or the shader ID ref in the material index (I don't know).
6. Copy paste the modified player material over the NPC material. Skin materials are usually the same size as NPC ones, but even if a material's size is different, there are two ways to work around it.
Method A: Just paste it and change the leftover space in between materials to 00 bytes. Or you can edit the material index to define the size of the material.
Method B: Slap it at the end of the file, in which case you will need to edit the material index to refer to the appended material's address. Make sure you update material size in the material index.
7. You may need to restore the original material values under the material ref. Because those material values define stuff like, shininess, wetness, opacity of the texture, colors, etc. These are all in float. There are also some switches. One of them is for custom color. It'll be on a row of 00 bytes, and then suddenly an 01.
Note that once you activate custom color, it'll override most of the detail on that diffuse texture.
If you did all of this correctly, the material should appear correctly, be color customizable, and not flicker.