CliffordVienna - 6 months ago 22

Python Question

Given is a grayscale image **I** as 2D Tensor (Dimension W,H) and a Tensor of coordinates **C** (Dim. None,2). I want to interpret the rows of **C** as coordinates in **I**, sample **I** at those coordinates using some kind of interpolation (bilinear would probably be fine for my use case), and store the resulting values in a new Tensor **P** (of dimension None, i.e. 1-dimensional with as many entries as **C** has rows).

Is this possible (efficiently) with TensorFlow? All I can find are functions for resizing (equidistant resampling if you like) of images. But I can't find anything out-of-the-box to sample at a list of coordinates.

I.e. I would have expected to find something like a tf.interpolate() function:

`I = tf.placeholder("float", shape=[128, 128])`

C = tf.placeholder("float", shape=[None, 2])

P = tf.interpolate(I, C, axis=[0, 1], method="linear")

Ideally I would be looking for a solution that would allow me to interpolate in an N dimensional tensor

(The "image" in my application isn't a picture btw., it's sampled data from a physical model (when used as placeholder) or an alternative learned model (when used as variable). Right now this physical model has 2 degrees of freedom, thus interpolating in an "image" is sufficient for now, but I might look into higher dimensional models in the future.)

In case something like that is not possible with existing TensorFlow features: Where should I start when I'd like to implement something like this tf.interpolate() operator? (documentation and/or simple example code)

Answer

There is no built-in op that performs this kind of interpolation, but you should be able to do it using a composition of existing TensorFlow ops. I'd suggest the following strategy for the bilinear case:

From your tensor

`C`

of indices, compute integer tensors corresponding to the four corner points. For example (with names assuming that the origin is at the top left):`top_left = tf.cast(tf.floor(C), tf.int32) top_right = tf.cast( tf.concat(1, [tf.floor(C[:, 0:1]), tf.ceil(C[:, 1:2])]), tf.int32) bottom_left = tf.cast( tf.concat(1, [tf.ceil(C[:, 0:1]), tf.floor(C[:, 1:2])]), tf.int32) bottom_right = tf.cast(tf.ceil(C), tf.int32)`

From each tensor representing a particular corner point, extract a vector of values from

`I`

at those points. For example, for the following function does this for the 2-D case:`def get_values_at_coordinates(input, coordinates): input_as_vector = tf.reshape(input, [-1]) coordinates_as_indices = (coordinates[:, 0] * tf.shape(input)[1]) + coordinates[:, 1] return tf.gather(input_as_vector, coordinates_as_indices) values_at_top_left = get_values_at_coordinates(I, top_left) values_at_top_right = get_values_at_coordinates(I, top_right) values_at_bottom_left = get_values_at_coordinates(I, bottom_left) values_at_bottom_right = get_values_at_coordinates(I, bottom_right)`

Compute the interpolation in the horizontal direction first:

`# Varies between 0.0 and 1.0. horizontal_offset = C[:, 0] - tf.cast(top_left[:, 0], tf.float32) horizontal_interpolated_top = ( ((1.0 - horizontal_offset) * values_at_top_left) + (horizontal_offset * values_at_top_right)) horizontal_interpolated_bottom = ( ((1.0 - horizontal_offset) * values_at_bottom_left) + (horizontal_offset * values_at_bottom_right))`

Now compute the interpolation in the vertical direction:

`vertical_offset = C[:, 1] - tf.cast(top_left[:, 1], tf.float32) interpolated_result = ( ((1.0 - vertical_offset) * horizontal_interpolated_top) + (vertical_offset * horizontal_interpolated_bottom))`