user1021819 user1021819 - 4 months ago 163
Python Question

tensorflow - memory leak?

I'm running tensorflow 0.10.0rc0 on OSX 10.9.5 Mavericks.

There are approximately 25k training examples, 250 features (x), 15 classes (y_) and the predict (y) is a single-hidden-layer NN perceptron.

The following snippet of a simple training loop seems to have a massive memory leak (of order 10s of GBs over =~ 200 iterations - brings down my MBP :( ) :

import tensorflow as tf

# Initialize placeholders and variables etc...
...

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y,y_))
train_step = tf.train.GradientDescentOptimizer(lrate).minimize(cost)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(niter):
# Train
_,c=sess.run([train_step,cost])
correct_prediction = tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
sess.run(correct_prediction)
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
print sess.run(accuracy)

# EDIT: Calculate test error
ytest=sess.run(y[itrain:itrain+itest,:])
ytest_=sess.run(y_[itrain:itrain+itest,:])
test_prediction = tf.equal(tf.argmax(ytest,1), tf.argmax(ytest_,1))
test_accuracy=tf.reduce_mean(tf.cast(test_prediction,tf.float32))
print sess.run(test_accuracy)

sess.close()


Am I doing something obviously wrong, or is this per chance a bug? Thanks!

PS: If this is fixed in a later tensorflow build, note that bazel requires Yosemite or higher, so I can't generate my own .whl file (AFAIK) from source; is a nightly whl available? I would rather not be forced into an OS upgrade right now.

Answer
  1. Its unnecessary to run sess.run(correct_prediction) -- it's a tensorflow graph variable on which the accuracy variable is dependant. This implies that it will be evaluated during the call to sess.run(accuracy) in any case.
  2. You're probably modifying your graph by creating new correct_prediction and accuracy variables on each iteration. This is also unnecessary -- they can be moved outside the loop and simply evaluated each time with calls to sess.run. So your inner loop will be something like
for i in range(niter):
    # Train
    _, c = sess.run([train_step, cost])
    print sess.run(accuracy)