bin381 bin381 - 1 month ago 5
Python Question

Why does my class cost so much memory?

from guppy import hpy

hp = hpy()


class Demo(object):
__slots__ = ('v0', 'v1')

def __init__(self, v0, v1):
self.v0 = v0
self.v1 = v1


from array import array

value = 1.01
ar = array('f')
ar2 = array('f')
for i in range(5000000):
ar.append(value + i)
ar2.append(value + i * 0.1 + i * 0.01 + i * 0.001 + i * 0.0001 + i * 0.000001)
a = []
for i in range(5000000):
vex = Demo(ar[i], ar[2])
a.append(vex)
print "Heap at the end of the functionn", hp.heap()


Here is the output:

Heap at the end of the functionn Partition of a set of 15063247 objects. Total size = 650251664 bytes.

Index Count % Size % Cumulative % Kind (class / dict of class)
0 5000000 33 320000000 49 320000000 49 __main__.Demo
1 10000108 66 240002592 37 560002592 86 float
2 368 0 42008896 6 602011488 93 list
3 2 0 40000112 6 642011600 99 array.array
4 28182 0 2214784 0 644226384 99 str
5 12741 0 1058448 0 645284832 99 tuple
6 189 0 669624 0 645954456 99 dict of module
7 371 0 588104 0 646542560 99 dict (no owner)
8 258 0 509232 0 647051792 100 dict of sip.wrappertype
9 3176 0 406528 0 647458320 100 types.CodeType


I am wondering why the Demo class cost so much memory. Because Demo class just keeps a reference for the float, it doesn't copy the float value.

getSizeOf(Demo) # 984


50W of Demo class maybe just cost memory:
984*50W=40215176
but, now costs
320000000
.
It is unbelievable, why?

Answer

sys.getsizeof() doesn't recurse into sub-objects, and you only took the size of the class, not of an instance. Each instance takes up 64 bytes, plus 24 bytes per float object (on OS X, using Python 2.7.12):

>>> d = Demo(1.0, 2.0)
>>> sys.getsizeof(d)
64
>>> sys.getsizeof(d.v0)
24
>>> sys.getsizeof(d) + sys.getsizeof(d.v0) + sys.getsizeof(d.v1)
112

Each slot only reserves memory for a pointer in the instance object; on my machine that's 8 bytes per pointer.

There are several differences between your Demo() instances and the array:

  • Instances have a minimal overhead to support reference counting and weak references, as well as contain a pointer to their class. The arrays store the values directly, without any of that overhead.
  • The instance stores Python floats. These are full-fledged objects, including reference counting and weak reference support. The array stores single precision floats as C values, while the Python float object models double precision floats. So the instance uses 2 * 24 bytes (on my Mac) just for those floats, vs. just 4 bytes per single-precision 'f' value in an array.
  • To track 5 million Demo instances, you also needed to create a list object, which is sized to handle at least 5 million object references. The array stores the C single-precision floats directly.

The hp.heap() output only counts the instance footprint, not the referenced float values on each line, but the totals match up:

  • 5 million times 64 bytes is 320.000.000 bytes of memory for the Demo instances.
  • 10 million times 24 bytes is 240.000.000 bytes of memory for the float instances, plus a further 108 floats referenced elsewhere.

Together, these two groups make up the majority of the 15 million Python objects on the heap.

  • The list object you created to hold the instances contains 5 million pointers, that's 40.000.000 bytes just to point to all the Demo instances, plus the accounting overhead for that object. There are a further 367 lists on the heap, referenced by other Python code.
  • 2 array instances with each 5 million 4-byte floats is 40.000.000 bytes, plus 56 bytes per array overhead.

So array objects are vastly more efficient to store a large number of numeric values, because it stores these as primitive C values. However, the disadvantage is that Python has to box each value you try to access; so accessing ar[10] returns a Python float object.

Comments