Gloria Bazargan Gloria Bazargan - 5 months ago 28
Python Question

Looping over N lines of a file and storing in multipel lists

I have a large file (about 80,000 lines) and I want to store each 10 line block into a separate list. For the first three 10-line blocks I have:

N=10 #Number of lines per block

with open("file", "r") as myfile:
profile1 = list(islice(myfile, 0,N))
profile2 = list(islice(myfile, 0,N))
profile3 = list(islice(myfile, 0,N))

I expect to have several hundred of these 10-line blocks so this is clearly not a very good way to do this.

How can I incorporate the list generation and
function into a loop?

Thank you in advance!


Use the following:

with open('file', 'r') as f:
    lines = f.readlines()
chunks = [lines[item:item+10] for item in range(0, len(lines), 10)]  # with Python 2 you can use xrange instead of range for large lists

To convert each chunk to array, try the following:

import numpy as np

my_arrays = [np.asarray(chunk) for chunk in chunks]