Martin Legry Martin Legry - 5 months ago 45
Python Question

Multiprocessing within infinite while loop

I wanted to use multiprocessing package to read data with JSON from multiple sources. I also wanted to do it indefinitely.

Here is my code text :

while True:
time_start =
f = open("data_logging_pv.csv", "ab")
c = csv.writer(f)
if __name__ == '__main__':
output =, [2, 4, 5, 6, 7, 9])
j = 0
for i in [2, 4, 5, 6, 7, 9]:
c.writerow([time_start, i, output[j][0], output[j][1], output[j][2], output[j][3], output[j][4]])
j = j + 1;
print("\nTemps d'execution:" +str(

Where the function getData is the JSON request.

The execution does not realize the, but the processes are wel created.
I have no clue how to debug this.

I'm a bit rusty with python and have enver used multiprocessing, then there may be some 'basic' errors.


Answer Source
  1. Added KeyboardInterrupt handler, so it wont spam exceptions into stdout
  2. Little optimized csv writing
  3. Pool initialized only once

Not sure, that kind of errors did you have, but this snippet works propely on python2.7:

import csv
import datetime
import signal
from multiprocessing.pool import Pool

def getData(item):
    return [item, item + 1, item ** item, item * 2, 0]

def worker():
    signal.signal(signal.SIGINT, signal.SIG_IGN)

if __name__ == '__main__':

    time_start =
    file = open("data_logging_pv.csv", "ab")
    c = csv.writer(file)
    pool = Pool(6, initializer=worker)


        while True:
            output =, [2, 4, 5, 6, 7, 9])

            for i, res in zip([2, 4, 5, 6, 7, 9], output):
                c.writerow([str(, i] + res)

    except KeyboardInterrupt:
        print("\nTemps d'execution:" + str( - time_start))
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download