Miguel Miguel - 3 months ago 12
Perl Question

perl gunzip to buffer and gunzip to file have different byte orders



I'm using Perl v5.22.1,

2.53_01, and
IO::Uncompress::Gunzip
2.068.

I want to use Perl to gunzip a
Storable
file in memory, without using an intermediate file.

I have a variable
$zip_file = '/some/storable.gz'
that points to this zipped file.

If I gunzip directly to a file, this works fine, and
%root
is correctly set to the
Storable
hash.

gunzip($zip_file, '/home/myusername/Programming/unzipped');
my %root = %{retrieve('/home/myusername/Programming/unzipped')};


However if I gunzip into memory like this:

my $file;
gunzip($zip_file, \$file);
my %root = %{thaw($file)};


I get the error


Storable binary image v56.115 more recent than I am (v2.10)`



so the Storable's magic number has been butchered: it should never be that high.

However, the strings in the unzipped buffer are still correct; the buffer starts with
pst
which is the correct
Storable
header. It only seems to be multi-byte variables like integers which are being broken.

Does this have something to do with byte ordering, such that writing to a file works one way while writing to a file buffer works in another? How can I gunzip to a buffer without it ruining my integers?

Answer

That's not related to unzip but to using retrieve vs. thaw. They both expect different input, i.e. thaw expect the output from freeze while retrieve expects the output from store. This can be verified with a simple test:

$ perl -MStorable -e 'my $x = {}; store($x,q[file.store])'
$ perl -MStorable=freeze -e 'my $x = {}; print freeze($x)' > file.freeze

On my machine this gives 24 bytes for the file created by store and 20 bytes for freeze. If I remove the leading 4 bytes from file.store the file is equivalent to file.freeze, i.e. store just added a 4 byte header. Thus you might try to uncompress the file in memory, remove the leading 4 bytes and run thaw on the rest.