Mr_and_Mrs_D Mr_and_Mrs_D - 1 year ago 92
Python Question

Reading a binary file with memoryview

I read a large file in the code below which has a special structure - among others two blocks that need be processed at the same time. Instead of seeking back and forth in the file I load these two blocks wrapped in


with open(abs_path, 'rb') as bsa_file:
# ...
# load the file record block to parse later
file_records_block = memoryview(
# load the file names block
file_names_block = memoryview(
# close the file
file_records_index = names_record_index = 0
for folder_record in folder_records:
name_size = struct.unpack_from('B', file_records_block, file_records_index)[0]
# discard null terminator below
folder_path = struct.unpack_from('%ds' % (name_size - 1),
file_records_block, file_records_index + 1)[0]
file_records_index += name_size + 1
for __ in xrange(folder_record.files_count):
file_name_len = 0
for b in file_names_block[names_record_index:]:
if b != '\x00': file_name_len += 1
else: break
file_name = unicode(struct.unpack_from('%ds' % file_name_len,
names_record_index += file_name_len + 1

The file is correctly parsed, but as it's my first use of the mamoryview interface I am not sure I do it right. The file_names_block is composed as seen by null terminated c strings.

  1. Is my trick
    using the memoryview magic or do I create some n^2 slices ? Would I need to use
    here ?

  2. As seen I just look for the null byte manually and then proceed to
    . But I read in How to split a byte string into separate bytes in python that I can use
    (docs ?) on the memory view - any way to use that (or another trick) to split the view in bytes ? Could I just call
    ? Would this preserve the memory efficiency ?

I would appreciate insight on the one right way to do this (in python 2).

Answer Source

A memoryview is not going to give you any advantages when it comes to null-terminated strings as they have no facilities for anything but fixed-width data. You may as well use bytes.split() here instead:

file_names_block =
file_names = file_names_block.split(b'\00')

Slicing a memoryview doesn't use extra memory (other than the view parameters), but if using a cast you do produce new native objects for the parsed memory region the moment you try to access elements in the sequence.

You can still use the memoryview for the file_records_block parsing; those strings are prefixed by a length giving you the opportunity to use slicing. Just keep slicing bytes of the memory view as you process folder_path values, there's no need to keep an index:

for folder_record in folder_records:
    name_size = file_records_block[0]  # first byte is the length, indexing gives the integer
    folder_path = file_records_block[1:name_size].tobytes()
    file_records_block = file_records_block[name_size + 1:]  # skip the null

Because the memoryview was sourced from a bytes object, indexing will give you the integer value for a byte, .tobytes() on a given slice gives you a new bytes string for that section, and you can then continue to slice to leave the remainder for the next loop.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download