xuanyue xuanyue - 2 months ago 5x
Python Question

Pythonic way to chain python generator function to form a pipeline

I'm doing a pipeline code refactoring using python.

Assuming we have a series of generator function and we want to chain those to form a data process pipeline.


import itertools

def foo1(g):
for i in g:
yield i + 1

def foo2(g):
for i in g:
yield 10 + i
def foo3(g):
for i in g:
yield 'foo3:' + str(i)

res = foo3(foo2(foo1(range(0, 5))))

for i in res:
print i



I do not think
foo3(foo2(foo1(range(0, 5))))
is a pythonic way to achieve my pipeline goal. Expecially when the number of stages in the pipeline is large.

I wish I could rewrite it like chain in jquery. Something similar to :

range(0, 5).foo1().foo2().foo3()

Or may be

l = [range(0, 5), foo1, foo2, foo3]
res = runner.run(l)

But I'm new to generator topic and couldn't find a way to achieve this.

Any help will be welcome.


I sometimes like to use a left fold (called reduce in Python) for this type of situation:

from functools import reduce
def pipeline(pipeline_list):
    return reduce(lambda x, y : y(x), pipeline_list)

res = pipeline([range(0,5), foo1, foo2, foo3])

Or even better:

def compose(func_list):
    return lambda arg : reduce(lambda f, g : g(f), [arg] + func_list)

p = compose([foo1, foo2, foo3])
res = p(range(0,5))