Mahin Ra Mahin Ra - 2 months ago 14
Python Question

MultiProcessing Lock() doesn't work

I have a program that reads some input text files and write all of them into a list called

ListOutput
which is a shared memory between two processes used in my program (I used two processes so my program runs faster!) I also have a shared memory variable called
processedFiles
which stores the names of the already read input files by any of the processes so the current process does not read them again!


  1. To make sure that two processes do not check the "processedFiles" at the same time (For example in the beginning, it is possible that at the same time they both may come to the conclusion that "processedFiles" is empty so they read the same file), therefore, I added a
    Lock
    around the checking part for
    processedFiles
    so one process should complete and release the locked part before the other process checking in to that part!

    My problem is that the
    Lock
    function seems not to work and when I print the current
    ProcessName
    inside the lock part, it shows that both processes are inside the lock part. I cannot figure out what is wrong with my code? (See the output below.)

  2. Since my main program is not only about reading the input text files and printing them in the list, but it has to do a very complicate operation on the input files before printing them into a list, should I use
    Pool
    instead of
    Process
    and why?



Code:

import glob
from multiprocessing import Process, Manager
from threading import *
import timeit
import os

os.chdir("files")

def print_content(ProcessName,processedFiles,ListOutput,lock):

for file in glob.glob("*.txt"):
newfile=0

lock.acquire()

print "\n Current Process:",ProcessName

if file not in processedFiles:
print "\n", file, " not in ", processedFiles," for ",ProcessName
processedFiles.append(file)
newfile=1#it is a new file

lock.release()
#if it is a new file
if newfile==1:
f = open(file,"r")
lines = f.readlines()
ListOutput.append(lines)
f.close()

# Create two processes as follows
try:
manager = Manager()

processedFiles = manager.list()
ListOutput = manager.list()

lock=Lock()
p1 = Process(target=print_content, args=("Procees-1",processedFiles,ListOutput,lock))
p2 = Process(target=print_content, args=("Process-2",processedFiles,ListOutput,lock))

p1.start()
p2.start()

p1.join()
p2.join()

print "ListOutput",ListOutput

except:
print "Error: unable to start process"


I have 4 input files called
1.txt
(contains "my car"),
2.txt
(contains "your car"),
3.txt
(contains "my book"),
4.txt
(contains "your book").
The output that it shows me changes in different runs. This is the output in one of the runs:

Current Process: Procees-1

Current Process: Process-2

1.txt not in [] for Procees-1

Current Process: Procees-1

2.txt not in
Current Process: Process-2
['1.txt'] for Procees-1

2.txt not in ['1.txt', '2.txt'] for Process-2

Current Process: Procees-1

3.txt not in ['1.txt', '2.txt', '2.txt'] for Procees-1

Current Process: Process-2

Current Process: Process-2

4.txt not in
Current Process: Procees-1
['1.txt', '2.txt', '2.txt', '3.txt'] for Process-2

4.txt not in ['1.txt', '2.txt', '2.txt', '3.txt', '4.txt'] for Procees-1
ListOutput [['my car'], ['your car'], ['your car'], ['my book'], ['your book'], ['your book']]

Answer

Bingo! Thanks for including the imports. Now the problem is obvious ;-)

You need to use a multiprocessing.Lock to get a lock that works across processes. The Lock you're actually using is implicitly obtained along with a mountain of other stuff via your

from threading import *

A threading.Lock is useless for your purpose: it has no effect whatsoever across processes; it only provides exclusion among threads within a single process.

It's usually a Bad Idea to use import *, and this is one reason why. Even if you changed your second import to

from multiprocessing import Process, Manager, Lock
                                              ^^^^

it wouldn't do you any good, because from threading import * would overwrite the Lock you really want.