Bharel Bharel - 7 months ago 61
Python Question

Interrupting a Queue.get

How can I interrupt a blocking

Queue.get()
in Python 3.X?

In Python 2.X setting a long timeout seems to work but the same cannot be said for Python 3.5.

Running on Windows 7, CPython 3.5.1, 64 bit both machine and Python.
Seems like it does not behave the same on Ubuntu.

Answer

The reason it works on Python 2 is that Queue.get with a timeout on Python 2 is implemented incredibly poorly, as a polling loop with increasing sleeps between non-blocking attempts to acquire the underlying lock; Python 2 doesn't actually feature a lock primitive that supports a timed blocking acquire (which is what a Queue internal Condition variable needs, but lacks, so it uses the busy loop). When you're trying this on Python 2, all you're checking is whether the Ctrl-C is processed after one of the (short) time.sleep calls finishes, and the longest sleep in Condition is only 0.05 seconds, which is so short you probably wouldn't notice even if you hit Ctrl-C the instant a new sleep started.

Python 3 has true timed lock acquire support (thanks to narrowing the number of target OSes to those which feature a native timed mutex or semaphore of some sort). As such, you're actually blocking on the lock acquisition for the whole timeout period, not blocking for 0.05s at a time between polling attempts.

It looks like Windows allows for registering handlers for Ctrl-C that mean that Ctrl-C doesn't necessarily generate a true signal, so the lock acquisition isn't interrupted to handle it. Python is informed of the Ctrl-C when the timed lock acquisition eventually fails, so if the timeout is short, you'll eventually see the KeyboardInterrupt, but it won't be seen until the timeout lapses. Since Python 2 Condition is only sleeping 0.05 seconds at a time (or less) the Ctrl-C is always processed quickly, but Python 3 will sleep until the lock is acquired.

Ctrl-Break is guaranteed to behave as a signal, but it also can't be handled by Python properly (it just kills the process) which probably isn't what you want either.

If you want Ctrl-C to work, you're stuck polling to some extent, but at least (unlike Python 2) you can effectively poll for Ctrl-C while live blocking on the queue the rest of the time (so you're alerted to an item becoming free immediately, which is the common case).

import time
import queue

def get_timed_interruptable(q, timeout):
    stoploop = time.monotonic() + timeout - 1
    while time.monotonic() < stoploop:
        try:
            return q.get(timeout=1)  # Allow check for Ctrl-C every second
        except queue.Empty:
            pass
    # Final wait for last fraction of a second
    return q.get(timeout=max(0, stoploop + 1 - time.monotonic()))                

This blocks for a second at a time until:

  1. The time remaining is less than a second (it blocks for the remaining time, then allows the Empty to propagate normally)
  2. Ctrl-C was pressed during the one second interval (after the remainder of that second elapses, KeyboardInterrupt is raised)
  3. An item is acquired (if Ctrl-C was pressed, it will raise at this point too)
Comments