user1021819 user1021819 - 1 year ago 307
Python Question

tensorflow - memory leak?

I'm running tensorflow 0.10.0rc0 on OSX 10.9.5 Mavericks.

There are approximately 25k training examples, 250 features (x), 15 classes (y_) and the predict (y) is a single-hidden-layer NN perceptron.

The following snippet of a simple training loop seems to have a massive memory leak (of order 10s of GBs over =~ 200 iterations - brings down my MBP :( ) :

import tensorflow as tf

# Initialize placeholders and variables etc...

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y,y_))
train_step = tf.train.GradientDescentOptimizer(lrate).minimize(cost)

init = tf.initialize_all_variables()
sess = tf.Session()

for i in range(niter):
# Train
correct_prediction = tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))

# EDIT: Calculate test error[itrain:itrain+itest,:])[itrain:itrain+itest,:])
test_prediction = tf.equal(tf.argmax(ytest,1), tf.argmax(ytest_,1))


Am I doing something obviously wrong, or is this per chance a bug? Thanks!

PS: If this is fixed in a later tensorflow build, note that bazel requires Yosemite or higher, so I can't generate my own .whl file (AFAIK) from source; is a nightly whl available? I would rather not be forced into an OS upgrade right now.

Answer Source
  1. Its unnecessary to run -- it's a tensorflow graph variable on which the accuracy variable is dependant. This implies that it will be evaluated during the call to in any case.
  2. You're probably modifying your graph by creating new correct_prediction and accuracy variables on each iteration. This is also unnecessary -- they can be moved outside the loop and simply evaluated each time with calls to So your inner loop will be something like
for i in range(niter):
    # Train
    _, c =[train_step, cost])