戴志廉 戴志廉 - 3 months ago 20
Python Question

python multiprocessing.Process's join can not end

I'm going to write a program which has multiple process(CPU-crowded) and multiple threading(IO-crowded).(the code below just a sample, not the program)

But when the code meet the

join()
,it make the program become a deadlock.

My code is post below

import requests
import time
from multiprocessing import Process, Queue
from multiprocessing.dummy import Pool


start = time.time()
queue = Queue()
rQueue = Queue()
url = 'http://www.bilibili.com/video/av'
for i in xrange(10):
queue.put(url+str(i))


def goURLsCrawl(queue, rQueue):
threadPool = Pool(7)
while not queue.empty():
threadPool.apply_async(urlsCrawl, args=(queue.get(), rQueue))
threadPool.close()
threadPool.join()
print 'end'


def urlsCrawl(url, rQueue):
response = requests.get(url)
rQueue.put(response)


p = Process(target=goURLsCrawl, args=(queue, rQueue))
p.start()
p.join() # join() is here
end = time.time()
print 'totle time %0.4f' % (end-start,)


Thanks in advance.

Answer

I finally find the reason. As you can see, I import the Queue from the multiprocessing, so the Queue should only used for Process, but I make the Thread access the Queue on my code, so it must something unknown occur behind the program.

To correct it, just import Queue instead of multiprocessing.Queue