Miguel Miguel - 1 year ago 94
Perl Question

perl gunzip to buffer and gunzip to file have different byte orders

I'm using Perl v5.22.1,

2.53_01, and

I want to use Perl to gunzip a
file in memory, without using an intermediate file.

I have a variable
$zip_file = '/some/storable.gz'
that points to this zipped file.

If I gunzip directly to a file, this works fine, and
is correctly set to the

gunzip($zip_file, '/home/myusername/Programming/unzipped');
my %root = %{retrieve('/home/myusername/Programming/unzipped')};

However if I gunzip into memory like this:

my $file;
gunzip($zip_file, \$file);
my %root = %{thaw($file)};

I get the error

Storable binary image v56.115 more recent than I am (v2.10)`

so the Storable's magic number has been butchered: it should never be that high.

However, the strings in the unzipped buffer are still correct; the buffer starts with
which is the correct
header. It only seems to be multi-byte variables like integers which are being broken.

Does this have something to do with byte ordering, such that writing to a file works one way while writing to a file buffer works in another? How can I gunzip to a buffer without it ruining my integers?

Answer Source

That's not related to unzip but to using retrieve vs. thaw. They both expect different input, i.e. thaw expect the output from freeze while retrieve expects the output from store. This can be verified with a simple test:

$ perl -MStorable -e 'my $x = {}; store($x,q[file.store])'
$ perl -MStorable=freeze -e 'my $x = {}; print freeze($x)' > file.freeze

On my machine this gives 24 bytes for the file created by store and 20 bytes for freeze. If I remove the leading 4 bytes from file.store the file is equivalent to file.freeze, i.e. store just added a 4 byte header. Thus you might try to uncompress the file in memory, remove the leading 4 bytes and run thaw on the rest.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download