Stefano Borini Stefano Borini - 5 months ago 51
Python Question

Using a context manager with a coroutine

This code does not work

from contextlib import contextmanager
import tornado.ioloop
import tornado.web
from tornado import gen
from tornado.httpclient import AsyncHTTPClient


@contextmanager
def hello():
print("hello in")
yield
print("hello out")


class MainHandler(tornado.web.RequestHandler):
@gen.coroutine
def get(self):
client = AsyncHTTPClient()
with hello():
result = yield client.fetch("http://localhost")
return "Hello "+str(result)

app = tornado.web.Application([('/', MainHandler)])
app.listen(12345)
tornado.ioloop.IOLoop.current().start()


And the reason why it doesn't work is that the context manager yielding and the coroutine yielding are incompatible in their behavior.

Do you confirm that the only way to achieve this is to use a
try finally
(particularly annoying if the context manager code must be used in many places). Maybe there's a subtle trick I don't know about? Googling did not help.

edit

This is the output I get

(venv) sborini@Mac-34363bd19f52:tornado$ python test.py
hello in
ERROR:tornado.application:Uncaught exception GET / (::1)
HTTPServerRequest(protocol='http', host='localhost:12345', method='GET', uri='/', version='HTTP/1.1', remote_ip='::1', headers={'Upgrade-Insecure-Requests': '1', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36', 'Accept-Language': 'en-US,en;q=0.8,it;q=0.6', 'Connection': 'keep-alive', 'Host': 'localhost:12345', 'Accept-Encoding': 'gzip, deflate, sdch'})
Traceback (most recent call last):
File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/web.py", line 1445, in _execute
result = yield result
File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/gen.py", line 1014, in run
yielded = self.gen.throw(*exc_info)
File "test.py", line 20, in get
result = yield client.fetch("http://localhost")
File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
ConnectionRefusedError: [Errno 61] Connection refused
ERROR:tornado.access:500 GET / (::1) 5.04ms


The point is that I never get the
hello out
message. I would expect that, once
fetch
spawns the future and the future errors, I return back to the yield point, get the exception, and leave the context, triggering the
print('hello out')
.

Note that I do get hello out if I just do a
try: finally:
around the
yield

Answer

The structure of the code is correct, and it's fine to mix context managers and coroutines this way. The @contextmanager and @coroutine decorators each assign their own meanings to yield within their decorated functions, but they remain independent.

As written, this code will print "hello in" and "hello out" if the fetch to http://localhost succeeds (or if you change it to point to a server that works), but it won't print "hello out" if the fetch raises an exception. To do that, you need to use a try/finally in your decorator:

@contextmanager
def hello():                                                                              
    print("hello in")                                                                     
    try:
        yield                                                                                 
    finally:
        print("hello out")   

One other error in this code is that you're returning a value from get(). The return value of get() is ignored; in Tornado to produce output you must call self.write() (or finish() or render()).