wsj wsj - 4 months ago 16
Python Question

How to unpack a binary file buffer into two variables?

I have a binary file that contains 4-byte binary values that represent a set of two

short int
each. I know that I can unpack a single 4-byte binary value into two short integers like this:

from struct import unpack

fval = b'\xba\x1e\x99\x01' #actualy read from some file
qualdip, azi = unpack('hh', fval)
print(type(qualdip), qualdip)
print(type(azi), azi)

>>> <class 'int'> 7866
>>> <class 'int'> 409


Now, I want to unpack the entire buffer. For the moment I am doing:

qualdips = []
azis = []
with open(bfile, 'rb') as buf:
fval = buf.read(4)
while fval:
qualdip, azi = unpack('hh', fval)
azis.append(azi)
qualdips.append(qualdip)
fval = buf.read(4)


Which takes over a minute for a 277MB file and seems to produce a huge memory overhead.

I would like to unpack the entire filebuffer directly into the two variables. How do I accomplish this?

I suspect that
struct.unpack_from
is my friend, but I am unsure how to formulate the format.

with open(bfile, 'rb') as buf:
qualdip, azi = unpack_from('hh', buf)


only extracts two values, and (i know the number of elements of my file)

with open(bfile, 'rb') as buf:
qualdip, azi = unpack_from('72457091h72457091h', buf)


expects this ridiculous amount of output variables. So:

How do I unpack the entire filebuffer directly into the two variables?

Answer

I don't know a way to unpack the values directly into two lists, but you can unpack the entire file into a tuple and then slice it in two:

fval = b'\xba\x1e\x99\x01' * 3
unpacked= unpack('3h3h', fval)
qualdip = unpacked[0::2]
azi = unpacked[1::2]

Alternatively, use islice to create a generator object to reduce memory consumption.

qualdip = islice(unpacked, 0, None, 2)
azi = islice(unpacked, 1, None, 2)