I'm using tensorflow-10.0 with gpu support, and I want to compare if 2 equations return the the same thing (which they actually should). I simplified the second equation up to the point where both equations are equal, but the assert returns that they are not.
In the following code snippet b,c,d are float32 tensors and calculated above.
a1 = tf.reduce_sum(-1/2 - 1/2*b + 1/2*tf.exp(c) + 1/2*tf.square(d), 1)
a2 = tf.reduce_sum(-1/2 - 1/2*b + 1/2*tf.exp(c) + 1/2*tf.square(d), 1)
a1 = control_flow_ops.with_dependencies([tf.assert_equal(a1, a2)], a1)
failed: [Condition x == y did not hold element-wise: x = ] [../Sum:0]
[0.038984224 0.047407709 0.043144785...] [y = ] [../Sum_1:0]
[0.038984239 0.047407985 0.043144524...]
tf.reduce_sum() operator—in common with most of TensorFlow's reduction operators—is implemented using multiple threads to perform the reduction in parallel. However, since floating-point addition is not associative, and the threads may complete in a non-deterministic order, the results of aggregating the same tensor multiple times can be different. The error can be particularly large when the numbers being aggregated have a large dynamic range.