Gerenuk Gerenuk - 4 months ago 36
Python Question

Iterate an iterator by chunks (of n) in Python?

Can you think of a nice way (maybe with itertools) to split an iterator into chunks of given size?

Therefore

l=[1,2,3,4,5,6,7]
with
chunks(l,3)
becomes an iterator
[1,2,3], [4,5,6], [7]


I can think of a small program to do that but not a nice way with maybe itertools.

Answer

The grouper() recipe from the itertools documentation's recipes comes close to what you want:

def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

It will fill up the last chunk with a fill value, though.

A less general solution that only works on sequences but does handle the last chunk as desired is

[my_list[i:i + chunk_size] for i in range(0, len(my_list), chunk_size)]

Finally, a solution that works on general iterators an behaves as desired is

def grouper(n, iterable):
    it = iter(iterable)
    while True:
       chunk = tuple(itertools.islice(it, n))
       if not chunk:
           return
       yield chunk