Kanika Rawat Kanika Rawat - 1 month ago 13
Python Question

Why getting Memory Error? Python

I have a 5gb text file and i am trying to read it line by line.
My file is in format-: Reviewerid<\t>pid<\t>date<\t>title<\t>body<\n>
This is my code

o = open('mproducts.txt','w')
with open('reviewsNew.txt','rb') as f1:
for line in f1:
line = line.strip()
line2 = line.split('\t')
o.write(str(line))
o.write("\n")


But i get Memory error when i try to run it. I have an 8gb ram and 1Tb space then why am i getting this error? I tried to read it in blocks but then also i get that error.

MemoryError

Answer

Update:

Installing 64 bit Python solves the issue.

OP was using 32 bit Python that's why getting into memory limitation.


Reading whole comments I think this can help you.

  • You can't read file in chunk (as 1024) since you want to process data.
  • Instead, read file in chunk of lines i.e N lines at a time.
  • You can use yield keyword and itertools in Python to achieve above.

Summary : Get N lines at time, process it and then write it.

Sample Code :

from itertools import islice
#You can change num_of_lines
def get_lines(file_handle,num_of_lines = 10):
    while True:
        next_n_lines = list(islice(file_handle, num_of_lines))
        if not next_n_lines:
            break
        yield next_n_lines


o = open('mproducts.txt','w')

with open('reviewsNew.txt','r') as f1:
    for data_lines in get_lines(f1):
        for line in data_lines:
            line = line.strip()
            line2 = line.split('\t')
            o.write(str(line))
            o.write("\n")
o.close()