Member Discussions

Anyone doing this (with a C/C++ engine) may want to look at the newly accepted Property Tree library in Boost (www.boost.org). It supports a number of hierarchical data formats via different parsers (including XML), which should negate the need for your own custom format.

Speed may suffer utilizing xml or some other format that must be parsed, unless you tweak it a bit. I utilize very basic binary file storage, hashing and a pointer to traverse for my data, and I have the most blazing-fast i/o one could ask for. Perhaps it can make the programming easier for someone, but if one is concerned with access times for data, custom format may be the way to go.

Well, instead of XML you could use some other standard structured format like YAML or JSON. JSON is supposed by Boost Property Trees. I'm not sure about specific formats, but generally reading ASCII flat files runs about 8x slower (parsing numbers, etc) than reading the numbers directly from a binary file. But there are still a number of advantages to ASCII.

When using binary files, how do you deal with changes in your structures/classes? I'm sure there's an answer, I just have no idea.

I've used flat binary files at work and had to deal with this problem. If you want to be able to handle format changes, one solution is to keep separate metadata files with a version history of the format, and stick a version number at the beginning of each data file. Then whenever you try to read a file that isn't the latest version, convert it to the latest version first using the version history. This gives you a one-time speed hit when you change versions, but otherwise it's as fast as flat binary files. It is, however, quite complicated to implement, especially if you try to include in the metadata enough info to do sensible defaulting of new fields in the structures.