witrin witrin - 1 year ago 170
Python Question

Process synchronization in Python

I try to synchronize multiple processes by using semaphores. I thought instead of creating a function for each process, it might be possible in a more generic way, with a single function and some arguments for the dependencies:

import multiprocessing
from multiprocessing import *

class Worker:
def __init__(self, size):
self._semaphores = [Semaphore(0)]*(size + 1)

def run(self, name, acquire, release):
for i in acquire:

print('Running', name)

for i in release:

In this case I've got five processes. The first starts first, the second and third after the first, the fourth after the second and third, and the fifth after the fourth.

if __name__ == '__main__':
worker = Worker(5)
Process(target=worker.run, args=('5', [5], [])).start()
Process(target=worker.run, args=('4', [4,4], [5])).start()
Process(target=worker.run, args=('3', [3], [4])).start()
Process(target=worker.run, args=('2', [2], [4])).start()
Process(target=worker.run, args=('1', [], [2,3])).start()

The expected output would be:

Running 1
Running 2
Running 3
Running 4
Running 5

But the synchronization doesn't work as expected. The execution is random including deadlocks. Why is that?

Answer Source

[Semaphore(0)]*(size + 1) creates a list with size + 1 references to a single Semaphore object. You need to make different instances of that class:

self._semaphores = [Semaphore(0) for _ in range(size + 1)]
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download