Bananach Bananach - 1 month ago 7
Python Question

Why is dill much faster and more disk-efficient than pickle for numpy arrays

I'm using Python 2.7 and NumPy 1.11.2, as well as the latest versions of dill ( I just did the

pip install dill
) , on Ubuntu 16.04.

When storing a NumPy array using pickle, I find that pickle is very slow, and stores arrays at almost three times the 'necessary' size.

For example, in the following code, pickle is approximately 50 times slower (1s versus 50s), and creates a file that is 2.2GB instead of 800MB.

import numpy
import pickle
import dill
B=numpy.random.rand(10000,10000)
with open('dill','wb') as fp:
dill.dump(B,fp)
with open('pickle','wb') as fp:
pickle.dump(B,fp)


I thought dill was just a wrapper around pickle. If this is true, is there a way that I can improve the performance of pickle myself? Is it generally not advisable to use pickle for NumPy arrays?

EDIT: Using Python3, I get the same performance for
pickle
and
dill


PS: I know about
numpy.save
, but I am working in a framework where I store lots of different objects, all residing in a dictionary, to a file.

Answer Source

This ought to be a comment, but I have not enough reputation... My guess is that this is due to the pickle protocol used.

On Python 2, the default protocol is 0 and highest supported protocol is 2. On Python 3, the default protocol is 3 and highest supported protocol is 4 (as of Python 3.6).

Each protocol version improves on the previous one, but protocol 0 is especially slow for largish objects. It should be avoided in most cases, except if you need to be able to read your pickles using extremely old versions of Python. Protocol 2 is already much better.

Now, I suppose dill uses pickle.HIGHEST_PROTOCOL by default, and if that is indeed the case, it would probably be the cause of a good deal of the speed difference. You could try using pickle.HIGHEST_PROTOCOL to see if you get similar performance using dill and standard pickle.

with open('dill', 'wb') as fp:
    dill.dump(B, fp, protocol=pickle.HIGHEST_PROTOCOL)
with open('pickle', 'wb') as fp:
    pickle.dump(B, fp, protocol=pickle.HIGHEST_PROTOCOL)