戴志廉 戴志廉 - 1 year ago 83
Python Question

python multiprocessing.Process's join can not end

I'm going to write a program which has multiple process(CPU-crowded) and multiple threading(IO-crowded).(the code below just a sample, not the program)

But when the code meet the

join()
,it make the program become a deadlock.

My code is post below

import requests
import time
from multiprocessing import Process, Queue
from multiprocessing.dummy import Pool


start = time.time()
queue = Queue()
rQueue = Queue()
url = 'http://www.bilibili.com/video/av'
for i in xrange(10):
queue.put(url+str(i))


def goURLsCrawl(queue, rQueue):
threadPool = Pool(7)
while not queue.empty():
threadPool.apply_async(urlsCrawl, args=(queue.get(), rQueue))
threadPool.close()
threadPool.join()
print 'end'


def urlsCrawl(url, rQueue):
response = requests.get(url)
rQueue.put(response)


p = Process(target=goURLsCrawl, args=(queue, rQueue))
p.start()
p.join() # join() is here
end = time.time()
print 'totle time %0.4f' % (end-start,)


Thanks in advance.

Answer Source

I finally find the reason. As you can see, I import the Queue from the multiprocessing, so the Queue should only used for Process, but I make the Thread access the Queue on my code, so it must something unknown occur behind the program.

To correct it, just import Queue instead of multiprocessing.Queue

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download