sum over a list of tensors in tensorflow
The tf.nn.l2_loss()
function returns a tensor with 0 dimensions.
But it's nice to not need to manually apply that to each weight tensor, so storing the weight tensors in a list is one way to solve the problem (as @mrry noted).
But rather than needing to write that out every time, what you could do is use the following function
def l2_loss_sum(list_o_tensors):
return tf.add_n([tf.nn.l2_loss(t) for t in list_o_tensors])
In your case this would look like:
total_loss = l2_loss_sum([layers[j].weights for j in range(self.n_layers)])
Also, tf.nn.l2_loss()
implicitly applies the squaring operation to the values as well as multiplying all the squared values by 1/2 , so were you use something like tf.nn.l2_loss(layers[j].weights**2 for j in range(self.n_layers))
you would actually be raising the weights to the 4th power. As a result your derivative of this loss term would be strange: it wouldn't cancel the 1/2 to 1 (but would implicitly be doubling your β) and the weights would be cubed.
The standard way to sum a list of tensors is to use the tf.add_n()
operation, which takes a list of tensors (each having the same size and shape) and produces a single tensor containing the sum.
For the particular problem that you have, I am assuming that each layers[j].weights
could have a different size. Therefore you will need reduce each element down to a scalar before summing, e.g. using the tf.nn.l2_loss()
function itself:
weights = [layers[j].weights for j in range(self.n_layers)]
losses = [tf.nn.l2_loss(w) for w in weights]
total_loss = tf.add_n(losses)
(Note however that when the values to be added are large, you may find it more efficient to calculate a sequence of tf.add()
operations, since TensorFlow keeps the values of each of the add_n
arguments in memory until all of them have been computed. A chain of add
ops allows some of the computation to happen earlier.)