Da’s Entropy

Simulation is an important research tool to reveal insight and for performance demonstration (but not proof). Sometimes, it is not uncommon that we obtain large amount of output from simulations and need to process them in different ways, which poses two questions: how to efficiently output the data, and how to store them.

The first question is quite important when you want to simulate really fast — writing data to the disk is very very slow so you don’t want that to slow down the entire program. A solution is to use a separate thread to do it, if your programming language supports it. For example, I use the Boost Thread library if I wrote my program in C++. The data sharing between the main program and the writer thread can, most of the time, be implemented in a producer-consumer queue.

The second question is equally important, because we want to automate this process as much as possible, when facing large volume of data. A good candidate would be sqlite, which can be accessed from many programming interfaces, and then you can manipulate the data via regular SQL statements.

In general, writing to a database may take longer as there is more work to do, so one may want to simply flush it to disk in files, and import it to a database later..

Above are some experience from myself, anyone has any other ideas on the tools or methods to deal with these kind of problems? That’ll be much appreciated!