I've been rewriting my parsing code for all the asset types and figured I'd share what I've learned about the Textures files here.
SNOFileHeader
Each asset has a unique "SNO ID" and the TOC.dat/CoreTOC.dat files contain an index which maps SNO IDs to a file type + file name (the file type determines the directory and file extension used to find the file). Each file begins with a 16-byte SNOFileHeader. The first DWORD is always 0xDEADBEEF and the second DWORD seems to be a version number for the specific file type. All files of a given type have the same value for this field in any given version of the game and new versions either use the same number, or use a larger number with new copies of every file of that type. The last two DWORDs are always zero.
The Textures file format hasn't changed since the initial beta version as far as I can tell, but the version number has gone from 0x2B to 0x2C to 0x2D. I'm pretty sure the code in D3TexCov that defines these as "SNO_TEXTURE" and "SNO_SURFACE" is wrong (probably this field should just be read in case future versions require different handling of the format).
Type Descriptors
The Diablo III.exe contains type descriptors which include info about all the different types used in the asset files. Most of the descriptors are filled in at runtime by c++ static initializers, so its easier to pull this info from the running game (though, its possible to get the info through static analysis of the .exe). Each type contains a list of the fields the type contains, including the type of each field and the offset. Occasionally they also include a name for the field, but for the most part the names are blank. The fields don't always cover 100% of a struct's layout, but any data not covered seems to be zeros.
Each asset type has a corresponding type descriptor which may reference additional types. The first 12 bytes of an asset's type are not specified. The first DWORD is the SNO ID of that file and the next two DWORDs are always zero.
Serialization
The game uses a special "SerializeData" type in order to serialize pointer data. Each field that represents some type of pointer (typically a variable-length array of another type, but also things like a TagMap and raw buffers) references a SerializeData field. This struct is 8 bytes long and contains an Offset and Length of the serialized version of the data for that pointer. The offsets are relative to the end of the SNOFileHeader, so adding 16 to the offset will get the value relative to the start of the actual file. This system allows the entire file to be read into a buffer and then a pass could be made over the buffer to link all the pointer fields to their proper positions in the buffer, thus re-creating the original object graph.
Textures format
With that out of the way, here's the format info for Textures.
Code: Select all
struct Textures { // sizeof = 872
0 DT_SNO<Textures> Id
16 PixelFormat PixelFormat
20 DT_INT Width
24 DT_INT Height
28 DT_INT FaceCount // 1 for normal textures, 6 for cube maps
32 DT_INT _0032 // flags? (0, 1, 2, 4, 6)
36 DT_INT MipMapLevels // number of mip maps in addition to the main texture
40 SerializeData[60] PixelData // really 6 arrays (one per cube face) of 10 elements each (main texture plus 0-9 mip maps)
520 DT_INT FrameCount // number of sub-images
524 SerializeData serFrame s
532 TexFrame[] Frames
536 IVector2D _0536 // origin? almost always (0, 0), but a couple 32x32 textures use (16, 16)
544 DT_INT _0544 // flags? (0, 1, 2, 4, 8)
548 DT_INT _0548 // flags? (0, 1, 3, 5, 7, 8)
552 DT_INT _0552 // 0, 6, 7, 8, 9
560 DT_INT64 _0560 // ?
568 DT_INT _0568 // 0
572 DT_BYTE _0572 // 0-255
573 DT_BYTE _0573 // 0-255
574 DT_BYTE _0574 // 0-255
592 DT_CHARARRAY[256] SrcFile
848 ImageFileID[] ImageFileIDs
852 SerializeData serImageFileIDs
860 DT_INT _0860 // bool? (0 or 1)
864 DT_INT _0864 // flags? (0, 1, 2, 3, 4, 9, 16, 17, 18, 32, 33, 48, 96)
868 DT_INT _0868 // ?
}
I added the missing DT_SNO field at the start of the struct and converted the second field from a DT_INT to an enum, but otherwise field types and offsets are as specified by the game. The names and comments were added by me based on analysis of extracted texture files.
Two big differences jumped out at me compared to the D3TexConv source. First, "MipMapLevels" is the number of mip maps *in addition to* to base texture. This value ranges from 0 (no mip maps) to 9. The source code seems to be ignoring the final mip map level because of this.
Second, the "PixelData" has 60 elements rather than 31. Looking at the actual texture files, its pretty clear that this is actully 6 arrays of 10 elements. Mostly only the first 10 are used, but presumably there are 6 arrays of 10 to handle cube maps. Since there can be up to 9 mip maps plus the base texture, this will fill the 10 element array.
The other types referenced here are fairly simple and already understood. TexFrame is what the D3TexConv calls "TexAtlasEntry" and ImageFileID is what D3TexConv calls "TexEntry". IVector2D is just a pair of floats.
There are plenty of fields that I still don't really know what they mean, but I've added comments about the value ranges I've seen in those fields. In any case, they don't seem to be needed in order to extract the textures or the sub-images.
Code: Select all
enum PixelFormat {
'A8R8G8B8' = 0,
'A1R5G5B5' = 4,
'L8 (Grayscale)' = 7,
'DXT1?' = 9,
'DXT1' = 10,
'DXT3' = 11,
'DXT5' = 12,
}
The D3TexCov code lists a lot of other formats, but these are the only ones I've seen in the data files. I have absolutely no idea why there are two values which both seem to indicate DXT1 textures.
Data for the A8R8G8B8, A1R5G5B5, and L8 formats is just an array of Width * Height pixel values with 32, 16, or 8 bits of data per pixel as you would expect. The DXTN textures use the
S3 compression algorithms, but store the data in what appears to be a non-standard way. Each version takes a 4x4 block of pixels and compresses it into a few bits per pixel:
DXT1 uses 2bpp for color values (2 16-bit colors per block, which are interpolated to generate 2 additional colors) and 2bpp to indicate which of these 4 colors to use for each pixel.
DXT3 uses 4bpp for alpha values plus the same color information as DXT1.
DXT5 uses 1bpp for alpha values (2 8-bit values per block, which are interpolated to generate 6 additional values), and 3bpp to indicate which of these 8 alpha values to use for each pixel, plus the same color information as DXT1.
The DXTN format descriptions seem to indicate that all the data for a 4x4 block should be stored together in one chunk, but the D3 files contain all of the alpha values (if applicable) followed by all of the alpha indexes (if applicable) followed by all of the color values, followed by all of the color indexes. My best guess as to why the data is stored this way is that it probably compresses better in the MPQ archive this way.
).
Any info would be much appreciated 
