BIG FILE FORMAT SPECIFICATION -- RBF1.23 -- v0.8 (incomplete) 
01/04/2000
-----------------------------------------------------------------------------


by:
_!Lachesis_atatata!
pronounced:
shaka >yell< lachesis shaka atatata >yell<
(you may yell or scream at your capability. what you yell is irrelevant)


member of:
ICGOMA (International Cooperative Governance Organization of
Meaningless Acronyms)

in conjunction with:
UCFEDD (Uraguayan Consortium of Ferrets with Erectile Dysfunction
Disorder)
APDOFF (Agrarian Paramilitary Defense Organization of Feminist
Farmers)
MOOOOO (MOOOOO)

-----------------------------------------------------------------------------

A. INTRODUCTION
B. OVERALL LAYOUT
-B1. HEADER AREA
-B2. TABLE OF CONTENTS AREA
-B3. NAME/DATA BLOB AREA
C. >gratzi<

-----------------------------------------------------------------------------
A. INTRODUCTION
-----------------------------------------------------------------------------

this specification describes the format of .BIG archives used in
Homeworld by Relic Entertainment Inc. a BIG archive is a container for
other files. instead of distributing hundreds of files separately, files are
"packed" together into one larger file.

at this time, it is not a complete specification. enough information has
been learned to extract all files from a BIG archive, but not enough to be
able to create one.

if information from this document is used for implementations or in
other documents, please credit me. it's just the nice thing to do.

i'll be haunting the message boards, if you have a question, post it. if i
don't reply i:

1. don't know the answer
2. am busy with something at some point on the globe
3. am dead

if there are errors on this document ... oops :)

-----------------------------------------------------------------------------
B. OVERALL LAYOUT
-----------------------------------------------------------------------------

there are three main areas to the BIG file format arranged sequentially
one after the other:

1. header
2. table of contents
3. name/data blobs

each is described in the sections below.

-----------------------------------------------------------------------------
B1. HEADER AREA
-----------------------------------------------------------------------------

the header area comes at the beginning of the file. it has the normal
elements you would expect to be there. there is only one header.

-------------------------------------------------------------------------

typedef unsigned long ulong_t;
typedef unsigned char ubyte_t;

const char BIG_FILE_ID[] = { 0x52, 0x42, 0x46, 0x31, 0x2E, 0x32, 0x33 };
// note: "RBF1.23"

struct header {
char magic_cookie[7];
ulong_t toc_size;
ulong_t header_unknown;
};

-------------------------------------------------------------------------

magic_cookie
identifier for the file. The only currently known value for this is contained
in BIG_FILE_ID;

toc_size
number of toc_entry structures (*see B2) that are contained in the file.
(note: toc stands for Table of Contents)

header_unknown
this field's purpose is unknown. it's value is always 1. does not seem to
play a role in anything yet.

-----------------------------------------------------------------------------
B2. TABLE OF CONTENTS AREA
-----------------------------------------------------------------------------

the table of contents area holds a list of toc_entry structures which comes
directly after the header. header.toc_size tells you how many there are.
for each file that is in the BIG archive, there is a toc_entry structure.

-------------------------------------------------------------------------

struct toc_entry {
ulong_t crc_msb;
ulong_t crc_lsb;
ulong_t name_size;
ulong_t data_compressed_size;
ulong_t data_uncompressed_size;
ulong_t file_offset;
time_t timestamp;
ubyte_t toc_unknown[4];
};

-------------------------------------------------------------------------

crc_msb/crc_lsb
this is some sort of crc value. crc_msb is the most significant byte and
crc_lsb is the least significant byte of a larger 8-byte(64-bit) crc number.
exactly how it's generated and how to use it is unknown. (*see crc note
below)

name_size
the size(length) of the name in the name_data_blob (*see B3) that this
toc_entry refers to.

data_compressed_size
the size of the data in the name_data_blob (*see B3) in compressed
form.

data_uncompressed_size
the size of the data in the name_data_blob (*see B3) in uncompressed
form.

file_offset
the offset in bytes from the beginning of the BIG file to the
name_data_blob (*see B3) that this toc_entry refers to.

timestamp
the date/time of the entry.

toc_unknown
what part this element plays is unknown. what is known is that
toc_unknown[0] will be 1 if the data has been compressed and 0 if it has
not. (*see toc_unknown note below) toc_unknown[1..3] always seems to
equal {0xC9, 0xCA, 0xCB)

-------------------------------------------------------------------------

toc_unknown note:
sometimes the data is compressed, sometimes it's not. compression can
be determined by doing either:

1. data_compressed_size < data_uncompressed_size or ...
2. toc_unknown[0] == 1

I would personally suggest sticking to option 1 until toc_unknown is fully
understood.

-------------------------------------------------------------------------

crc note:
the entire table of contents seems to be sorted by the 8-byte crc value.
why is a complete mystery. the crc itself is most likely used as a data
validation mechanism. but to sort on it? it may be that the crc is also
being used as a mechanism to uniquely identify a file. i *think* that
there are no duplicate crc's.

-----------------------------------------------------------------------------
B3. NAME/DATA BLOB AREA
-----------------------------------------------------------------------------

the name/data blob area holds a list of name_data_blob structures which
comes directly after the table of contents area. a name_data_blob holds
the name and data of a file that exists in the BIG archive. each
name_data_blob is referred to by a toc_entry structure which gives the
sizes of the name and data fields. 
-------------------------------------------------------------------------

struct name_data_blob {
char name[];
char data[];
};

quick note:
the structure above is not legal C. the fields are not fixed length and i've
used this notation because it's useful. use the standard techniques of
dynamic memory when implementing.

-------------------------------------------------------------------------

name
this is the file name of the original file. it's length is determined from
toc_entry.name_size + 1 (note: there IS a terminating null byte) which
this name_data_blob is being referred by. it also happens to be
encrypted. (*see encryption note below)

data
this is the actual file data. it's length is determined from the
toc_entry.data_compressed_size field which this name_data_blob is being
referred by. if the data is compressed, the algorithm used is LZSS. (*see
compression note below)

-------------------------------------------------------------------------

encryption note:
as if the world isn't evil enough as it is, some sick sick sick bastard > my
kinda programmer actually :) < decided to encrypt the file names so that
those lurking hacker types would get all confused and bleary-eyed from
trying to hack the format. luckily, i knew this silly encryption trick ... i've
used it on others myself. ;)

what's used is an XOR run. it's really simple, really fast, and has no
redeeming value (ie. compression, secure encryption) other than to
screw with people. here's the code for doing it. 
void xor_run(char* buffer, ulong_t buffer_size)
{
char last_char;
ulong_t i;

last_char = (char)0xD5;

for (i = 0; i < buffer_size; i++) 
{
last_char ^= buffer[i];
buffer[i] = last_char;
}

}

for those of you who don't catch on, that's both the encryption AND
decryption routine. this particular version de/encrypts "in-place" and
writes over the buffer you pass in. 
as an implementation note, don't touch the terminating null when
decrypting. pass in toc_entry.name_size, not toc_entry.name_size + 1.

-------------------------------------------------------------------------

compression note:
compression (decompression) of data is done using the LZSS algorithm.
for this particular implementation:

> very basic LZSS (ie. no huffman a la ZIP)
> marker bit of 1 signals passthrough character
> marker bit of 0 signals dictionary entry
> dictionary entry is composed of 12-bit index and 4-bit length fields

don't know the LZSS algorithm? go to:

http://dogma.net/DataCompression/

if you look carefully, you will even find a clean LZSS implementation in C
up amongst the links that works perfectly. of course, since life isn't fair in
the least, i found it about an hour AFTER implementing it myself. ;(

note: i'm not associated with the link above, or any link off of it. so don't
waste your time speculating.

-----------------------------------------------------------------------------