Dave Hirschfeld Dave Hirschfeld - 28 days ago 6
Python Question

How to get the result of a future in a callback?

The

add_done_callback
method was recently added to the distributed
Future
object which allows you to take some action after the future finishes, irrespective of whether it succeeded or not.

http://distributed.readthedocs.io/en/latest/api.html?highlight=add_done_callback#distributed.client.Future.add_done_callback

The callback function will hang if you try to directly call any of the methods
result
,
exception
or
traceback
on the passed future object.

The exception and traceback can however be accessed in the callback as follows:
fut._exception().result()

fut._traceback().result()


Trying the same pattern with the result - i.e.
fut._result().result()
raises an exception:

File "C:\Python\lib\site-packages\tornado\concurrent.py", line 316, in _check_done
raise Exception("DummyFuture does not support blocking for results")
Exception: DummyFuture does not support blocking for results


Without being able to access the result of the future in the callback, being able to add a callback is of limited use to me.

Am I missing something - is there a way to get the result of the future in the callback?

In the asyncio documentation it seems to give an example where accessing the
result
method directly is possible:

https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-forever

...I'm not sure how this related to tornado/distributed, but it would be very useful to be able to do the same.

from distributed import Client


client = Client("127.0.0.1:8786")

def f(delay):
from time import sleep
from numpy.random import randn
sleep(delay)
if randn() > 1:
1/0
return delay

def callback(fut):
import logging
logger = logging.getLogger('distributed')
if fut.status == 'finished':
res = future._result().result() # <-------------- Doesn't work!
logger.info("{!r} - {!s}".format(fut, res))
else:
logger.info("{!r} - {!s}".format(fut, fut.status))


args = rand(10)
futs = client.map(f, args)
for fut in futs:
fut.add_done_callback(callback)

Answer

Currently your callback gets called within the Tornado Event loop. If you want to get the result of the future you'll have to use the Tornado API.

Here is a minimal example:

In [1]: from distributed import Client
In [2]: client = Client()
In [3]: def inc(x):
   ...:     return x + 1
   ...: 
In [4]: from tornado import gen

In [5]: @gen.coroutine
   ...: def callback(future):
   ...:     result = yield future._result()
   ...:     print(result * 10)
   ...:     
In [6]: future = client.submit(inc, 1)

In [7]: future.add_done_callback(callback)

20

However, your question highlights that perhaps this is not the most intuitive way for users to interact with add_done_callback, so I wouldn't be surprised if we introduced a breaking change for later versions.

In [8]: import distributed

In [8]: distributed.__version__
Out[8]: '1.14.0'
Comments