xis_one xis_one - 3 months ago 35
Python Question

Python itertools with multiprocessing - huge list vs inefficient CPUs usage with iterator

I work on n elements (named "pair" below) variations with repetition used as my function's argument. Obviously everything works fine as long as the "r" list is not big enough to consume all the memory. The issue is I have to make more then 16 repetitions for 6 elements eventually. I use 40 cores system in cloud for this.

The code looks looks like the following:

if __name__ == '__main__':
pool = Pool(39)
r = itertools.product(pairs,repeat=16)
pool.map(f, r)


I believe i should use iterator instead of creating the huge list upfront and here the problem starts..

I tried to solve the issue with the following code:

if __name__ == '__main__':
pool = Pool(39)
for r in itertools.product(pairs,repeat=14):
pool.map(f, r)


The memory problem goes away but the CPUs usage is like 5% per core. Now the single core version of the code is faster then this.

I'd really appreciate if you could guide me a bit..

Thanks.

Answer

Your original code isn't creating a list upfront in your own code (itertools.product returns a generator), but pool.map is realizing the whole generator (because it assumes if you can store all outputs, you can store all inputs too).

Don't use pool.map here. If you need ordered results, using pool.imap, or if result order is unimportant, use pool.imap_unordered. Iterate the result of either call (don't wrap in list), and process the results as they come, and memory should not be an issue:

if __name__ == '__main__':
    pool = Pool(39)
    for result in pool.imap(f, itertools.product(pairs, repeat=16)):
        print(result)

If you're using pool.map for side-effects, so you just need to run it to completion but the results and ordering don't matter, you could dramatically improve performance by using imap_unordered and using collections.deque to efficiently drain the "results" without actually storing anything (a deque with maxlen of 0 is the fastest, lowest memory way to force an iterator to run to completion without storing the results):

from collections import deque

if __name__ == '__main__':
    pool = Pool(39)
    deque(pool.imap_unordered(f, itertools.product(pairs, repeat=16)), 0)