mezamorphic mezamorphic - 1 year ago 116
C++ Question

Iterate over 10GB+ binary file

What is the simplest way to read a 10GB binary file and parse each byte without resorting to boost libraries?

Its so confusing with streams, files, memory mapped files etc.

I literally just want something like:

char* buffer = read(filename, binary);

while(buffer != EOF){
//Read byte

Performance does matter simply due to the file size.

Answer Source

If you want good performance for sequential access (reading from the beginning toward the end), use fread(). You can store the FILE* in a std::shared_ptr for RAII:

std::shared_ptr<FILE> file(fopen(...), fclose);

You can ignore C++ streams, memory mapped files, Boost, etc. None of that will be faster than fread().

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download