lhlmgr lhlmgr - 3 months ago 20
Python Question

Assertion error when comparing two float tensors

I'm using tensorflow-10.0 with gpu support, and I want to compare if 2 equations return the the same thing (which they actually should). I simplified the second equation up to the point where both equations are equal, but the assert returns that they are not.

In the following code snippet b,c,d are float32 tensors and calculated above.

a1 = tf.reduce_sum(-1/2 - 1/2*b + 1/2*tf.exp(c) + 1/2*tf.square(d), 1)
a2 = tf.reduce_sum(-1/2 - 1/2*b + 1/2*tf.exp(c) + 1/2*tf.square(d), 1)

a1 = control_flow_ops.with_dependencies([tf.assert_equal(a1, a2)], a1)



tensorflow.python.framework.errors.InvalidArgumentError: assertion
failed: [Condition x == y did not hold element-wise: x = ] [../Sum:0]
[0.038984224 0.047407709 0.043144785...] [y = ] [../Sum_1:0]
[0.038984239 0.047407985 0.043144524...]


Is there a reason, why comparing two float32 vectors lead to an assertion error (even the equation is deterministic, and handling the same values)? Rounding errors should be occur for both equations the same way, or am I wrong?

Thanks in advance!

Answer

The tf.reduce_sum() operator—in common with most of TensorFlow's reduction operators—is implemented using multiple threads to perform the reduction in parallel. However, since floating-point addition is not associative, and the threads may complete in a non-deterministic order, the results of aggregating the same tensor multiple times can be different. The error can be particularly large when the numbers being aggregated have a large dynamic range.