ckorzhik ckorzhik - 5 months ago 190
Python Question

TensorFlow strings: what they are and how to work with them

When I read file with

I get something with type
. Documentation says only that it is "Variable length byte arrays. Each element of a Tensor is a byte array." ( I have no idea how to interpret this.

I can do nothing with this type. In usual python you can get elements by index like
, but when I run following code I get an error.

import tensorflow as tf
import numpy as np

x = tf.constant("This is string")
y = x[:4]

init = tf.initialize_all_variables()
sess = tf.Session()
result =
print result

It says

  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/", line 621, in assert_has_rank
raise ValueError("Shape %s must have rank %d" % (self, rank))
ValueError: Shape () must have rank 1

Also I cannot convert my string to
tensor. It is
file and it has magic header "PIEH". This numpy code successfuly convert such header into number (see example here but I can't do that with tensorflow. I tried
tf.string_to_number(string, out_type=tf.float32)
but it says

tensorflow.python.framework.errors.InvalidArgumentError: StringToNumberOp could not correctly convert string: PIEH

So, what string is? What it's shape is? How can I at least get part of the string? I suppose that if I can get part of it I can just skip "PIEH" part.

UPD: I forgot to say that
tf.slice(string, [0], [4])
also doesn't work with same error.


Unlike Python, where a string can be treated as a list of characters for the purposes of slicing and such, TensorFlow's tf.strings are indivisible values. For instance, x below is a Tensor with shape (2,) whose each element is a variable length string.

x = tf.constant(["This is a string", "This is another string"])

However, to achieve what you want, TensorFlow provides the tf.decode_raw operator. It takes a tf.string tensor as input, but can decode the string into any other primitive data type. For instance, to interpret the string as a tensor of characters, you can do the following :

x = tf.constant("This is string")
x = tf.decode_raw(x, tf.uint8)
y = x[:4]
sess = tf.InteractiveSession()
# prints [ 84 104 105 115]