Demaunt Demaunt - 4 months ago 12
Python Question

python memory of list in "for" loop

Using some function I create a list of million items within and check memory consumption

print ('Memory at start: {0} Mb'.format(memory_usage_psutil()))
my_list = people_list(1000000)
print ('Memory at end: {0} Mb'.format(memory_usage_psutil()))


What I get is

Memory at start: 100.90625 Mb
Memory at end: 403.8125 Mb


So creating a list and keeping it in memory takes ~300mb

For example I want to count amount of items in PART of my_list WITHOUT using
len()


count = 0
additional_list = my_list[0:500000]
for item in additional_list:
count += 1
if count == 499999:
print ('Memory in loop: {0} Mb'.format(memory_usage_psutil()))


and results confused me.

Memory at start: 100.90625 Mb
Memory in loop: 403.6953125 Mb
Memory at end: 403.6953125 Mb


I expected that creating additional list will increase memory consumption by 150Mb but it did not happen. Can u explain why? I mean id's are different

In[30]id(my_list)
Out[30]: 2672641621704
In[31]id(additional_list)
Out[31]: 2672642905864


and same results come from

for item in my_list[0:500000]:
count += 1
if count == 499999:
print ('Memory in loop: {0} Mb'.format(memory_usage_psutil()))

Answer

It's true that your two lists are different objects and have different addresses in memory. However, the individual objects in the list are the same. Using a slice basically gave you a view on some elements in your list.

You can confirm that with::

>>> map(id, my_list)[0:500000] == map(id, additional_list)