romeasy romeasy -4 years ago 96
Python Question

tensorflow: Strange result from convolution compared to theano (not flipping, though)

I am trying to implement some deep neural network with tensorflow. But I have already a problem at the first steps.

When I type the following using theano.tensor.nnet.conv2d, I get the expected result:

import theano.tensor as T
import theano
import numpy as np
# Theano expects input of shape (batch_size, channels, height, width)
# and filters of shape (out_channel, in_channel, height, width)
x = T.tensor4()
w = T.tensor4()
c = T.nnet.conv2d(x, w, filter_flip=False)
f = theano.function([x, w], [c], allow_input_downcast=True)
base = np.array([[1, 0, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]).T
i = base[np.newaxis, np.newaxis, :, :]
print f(i, i) # -> results in 3 as expected because np.sum(i*i) = 3


However, when I do the presumingly same thing in tf.nn.conv2d, my result is different:

import tensorflow as tf
import numpy as np
# TF expects input of (batch_size, height, width, channels)
# and filters of shape (height, width, in_channel, out_channel)
x = tf.placeholder(tf.float32, shape=(1, 4, 3, 1), name="input")
w = tf.placeholder(tf.float32, shape=(4, 3, 1, 1), name="weights")
c = tf.nn.conv2d(x, w, strides=[1, 1, 1, 1], padding='VALID')
with tf.Session() as sess:
base = np.array([[1, 0, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]).T
i = base[np.newaxis, :, :, np.newaxis]
weights = base[:, :, np.newaxis, np.newaxis]
res = sess.run(c, feed_dict={x: i, w: weights})
print res # -> results in -5.31794233e+37


The layout of the convolution operation in tensorflow is a little different from theano, which is why the input looks slightly different.
However, since strides in Theano default to (1,1,1,1) and a valid convolution is the default, too, this should be the exact same input.

Furthermore, tensorflow does not flip the kernel (implements cross-correlation).

Do you have any idea why this is not giving the same result?

Thanks in advance,

Roman

Answer Source

Okay, I found a solution, even though it is not really one because I do not understand it myself. First, it seems that for the task that I was trying to solve, Theano and Tensorflow use different convolutions. The task at hand is a "1.5 D convolution" which means sliding a kernel in only one direction over the input (here DNA sequences).

In Theano, I solved this with the conv2d operation that had the same amount of rows as the kernels and it was working fine.

However, Tensorflow (probably correctly) wants me to use conv1d for that, interpreting the rows as channels.

So, the following should work but didn't in the beginning:

import tensorflow as tf
import numpy as np

# TF expects input of (batch_size, height, width, channels)
# and filters of shape (height, width, in_channel, out_channel)
x = tf.placeholder(tf.float32, shape=(1, 4, 3, 1), name="input")
w = tf.placeholder(tf.float32, shape=(4, 3, 1, 1), name="weights")

x_star = tf.reshape(x, [1, 4, 3])
w_star = tf.reshape(w, [4, 3, 1])
c = tf.nn.conv1d(x_star, w_star, stride=1, padding='VALID')
with tf.Session() as sess:
    base = np.array([[1, 0, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]).T
    i = base[np.newaxis, :, :, np.newaxis]
    weights = base[:, :, np.newaxis, np.newaxis]
    res = sess.run(c, feed_dict={x: i, w: weights})
    print res # -> produces 3 after updating tensorflow

This code produced NaN until I updated Tensorflow to version 1.0.1 and since then, it produces the expected output.

To summarize, my problem was partly solved by using 1D convolution instead of 2D convolution but still required the update of the framework. For the second part, I have no idea at all what might have caused wrong behavior in the first place.

EDIT: The code I posted in my original question is now working fine, too. So the different behavior came only from an old (maybe corrupt) version of TF.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download