Gerenuk Gerenuk - 6 months ago 66
Python Question

Iterate an iterator by chunks (of n) in Python?

Can you think of a nice way (maybe with itertools) to split an iterator into chunks of given size?


becomes an iterator
[1,2,3], [4,5,6], [7]

I can think of a small program to do that but not a nice way with maybe itertools.


The grouper() recipe from the itertools documentation's recipes comes close to what you want:

def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

It will fill up the last chunk with a fill value, though.

A less general solution that only works on sequences but does handle the last chunk as desired is

[my_list[i:i + chunk_size] for i in range(0, len(my_list), chunk_size)]

Finally, a solution that works on general iterators an behaves as desired is

def grouper(n, iterable):
    it = iter(iterable)
    while True:
       chunk = tuple(itertools.islice(it, n))
       if not chunk:
       yield chunk