user270700 user270700 - 9 days ago 7
Python Question

What is the alternative of tf.Variable.ref() in Tensorflow version 0.12?

I'm trying to run open code of A3C reinforcement learning algorithm to learn A3C in A3C code

However,I got several errors and I could fix except one.
In the code,

ref()
which is a member function of tf.Variable is used (1,2), but in recent tensorflow version 0.12rc, that function seems to be deprecated.
So I don't know what is the best way to replace it (I don't understand exactly why the author used
ref()
). When I just changed it to the variable itself (for example
v.ref()
to
v
), there was no error, but reward is not changed. It seems it cannot learn and I guess it is because the variables are not properly updated.

Please advise me what is the proper way to modify the code to work.

Answer

The new method tf.Variable.read_value() is the replacement for tf.Variable.ref() in TensorFlow 0.12 and later.

The use case for this method is slightly tricky to explain, and is motivated by some caching behavior that causes multiple uses of a remote variable on a different device to use a cached value. Let's say you have the following code:

with tf.device("/cpu:0")
  v = tf.Variable([[1.]])

with tf.device("/gpu:0")
  # The value of `v` will be captured at this point and cached until `m2`
  # is computed.
  m1 = tf.matmul(v, ...)

with tf.control_dependencies([m1])
  # The assign happens (on the GPU) after `m1`, but before `m2` is computed.
  assign_op = v.assign([[2.]])

with tf.control_dependencies([assign_op]):
  with tf.device("/gpu:0"):
    # The initially read value of `v` (i.e. [[1.]]) will be used here,
    # even though `m2` is computed after the assign.
    m2 = tf.matmul(v, ...)

sess.run(m2)

You can use tf.Variable.read_value() to force TensorFlow to read the variable again later, and it will be subject to whatever control dependencies are in place. So if you wanted to see the result of the assign when computing m2, you'd modify the last block of the program as follows:

with tf.control_dependencies([assign_op]):
  with tf.device("/gpu:0"):
    # The `read_value()` call will cause TensorFlow to transfer the
    # new value of `v` from the CPU to the GPU before computing `m2`.
    m2 = tf.matmul(v.read_value(), ...)

(Note that, currently, if all of the ops were on the same device, you wouldn't need to use read_value(), because TensorFlow doesn't make a copy of the variable when it is used as the input to an op on the same device. This can cause a lot of confusion—for example when you enqueue a variable to a queue!—and it's one of the reasons that we're working on enhancing the memory model for variables.)