How to input a list of lists with different sizes in tf.data.Dataset
You can use tf.data.Dataset.from_generator()
to convert any iterable Python object (like a list of lists) into a Dataset
:
t = [[4, 2], [3, 4, 5]]
dataset = tf.data.Dataset.from_generator(lambda: t, tf.int32, output_shapes=[None])
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
print(sess.run(next_element)) # ==> '[4, 2]'
print(sess.run(next_element)) # ==> '[3, 4, 5]'
For those working with TensorFlow 2 and looking for an answer I found the following to work directly with ragged tensors. which should be much faster than generator, as long as the entire dataset fits in memory.
t = [[[4,2]],
[[3,4,5]]]
rt=tf.ragged.constant(t)
dataset = tf.data.Dataset.from_tensor_slices(rt)
for x in dataset:
print(x)
produces
<tf.RaggedTensor [[4, 2]]> <tf.RaggedTensor [[3, 4, 5]]>
For some reason, it's very particular about having at least 2 dimensions on the individual arrays.
In addition to @mrry's answer, the following code is also possible if you would like to create (images, labels) pair:
import itertools
data = tf.data.Dataset.from_generator(lambda: itertools.izip_longest(images, labels),
output_types=(tf.float32, tf.float32),
output_shapes=(tf.TensorShape([None, None, 3]),
tf.TensorShape([None])))
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
image, label = sess.run(next_element) # ==> shape: [320, 420, 3], [20]
image, label = sess.run(next_element) # ==> shape: [1280, 720, 3], [40]