Dave Hirschfeld Dave Hirschfeld - 1 year ago 113
Python Question

How to get the result of a future in a callback?

The

add_done_callback
method was recently added to the distributed
Future
object which allows you to take some action after the future finishes, irrespective of whether it succeeded or not.

http://distributed.readthedocs.io/en/latest/api.html?highlight=add_done_callback#distributed.client.Future.add_done_callback

The callback function will hang if you try to directly call any of the methods
result
,
exception
or
traceback
on the passed future object.

The exception and traceback can however be accessed in the callback as follows:
fut._exception().result()

fut._traceback().result()


Trying the same pattern with the result - i.e.
fut._result().result()
raises an exception:

File "C:\Python\lib\site-packages\tornado\concurrent.py", line 316, in _check_done
raise Exception("DummyFuture does not support blocking for results")
Exception: DummyFuture does not support blocking for results


Without being able to access the result of the future in the callback, being able to add a callback is of limited use to me.

Am I missing something - is there a way to get the result of the future in the callback?

In the asyncio documentation it seems to give an example where accessing the
result
method directly is possible:

https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-forever

...I'm not sure how this related to tornado/distributed, but it would be very useful to be able to do the same.

from distributed import Client


client = Client("127.0.0.1:8786")

def f(delay):
from time import sleep
from numpy.random import randn
sleep(delay)
if randn() > 1:
1/0
return delay

def callback(fut):
import logging
logger = logging.getLogger('distributed')
if fut.status == 'finished':
res = future._result().result() # <-------------- Doesn't work!
logger.info("{!r} - {!s}".format(fut, res))
else:
logger.info("{!r} - {!s}".format(fut, fut.status))


args = rand(10)
futs = client.map(f, args)
for fut in futs:
fut.add_done_callback(callback)

Answer Source

Currently your callback gets called within the Tornado Event loop. If you want to get the result of the future you'll have to use the Tornado API.

Here is a minimal example:

In [1]: from distributed import Client
In [2]: client = Client()
In [3]: def inc(x):
   ...:     return x + 1
   ...: 
In [4]: from tornado import gen

In [5]: @gen.coroutine
   ...: def callback(future):
   ...:     result = yield future._result()
   ...:     print(result * 10)
   ...:     
In [6]: future = client.submit(inc, 1)

In [7]: future.add_done_callback(callback)

20

However, your question highlights that perhaps this is not the most intuitive way for users to interact with add_done_callback, so I wouldn't be surprised if we introduced a breaking change for later versions.

In [8]: import distributed

In [8]: distributed.__version__
Out[8]: '1.14.0'