How to average summaries over multiple batches?

I would avoid calculating the average outside the graph.

You can use tf.train.ExponentialMovingAverage:

ema = tf.train.ExponentialMovingAverage(decay=my_decay_value, zero_debias=True)
maintain_ema_op = ema.apply(your_losses_list)

# Create an op that will update the moving averages after each training step.
with tf.control_dependencies([your_original_train_op]):
    train_op = tf.group(maintain_ema_op)

Then, use:

sess.run(train_op)

That will call maintain_ema_op because it is defined as a control dependency.

In order to get your exponential moving averages, use:

moving_average = ema.average(an_item_from_your_losses_list_above)

And retrieve its value using:

value = sess.run(moving_average)

This calculates the moving average within your calculation graph.

I think it's always better to let tensorflow do the calculations.

Have a look at the streaming metrics. They have an update function to feed the information of your current batch and a function to get the averaged summary. It's going to look somewhat like this:

accuracy = ... 
streaming_accuracy, streaming_accuracy_update = tf.contrib.metrics.streaming_mean(accuracy)
streaming_accuracy_scalar = tf.summary.scalar('streaming_accuracy', streaming_accuracy)

# set up your session etc. 

for i in iterations:
      for b in batches:
               sess.run([streaming_accuracy_update], feed_dict={...})

     streaming_summ = sess.run(streaming_accuracy_scalar)
     writer.add_summary(streaming_summary, i)

Also see the tensorflow documentation: https://www.tensorflow.org/versions/master/api_guides/python/contrib.metrics

and this question: How to accumulate summary statistics in tensorflow

Do the averaging of your measure in Python and create a new Summary object for each mean. Here is what I do:

accuracies = []

# Calculate your measure over as many batches as you need
for batch in validation_set:
  accuracies.append(sess.run([training_op]))

# Take the mean of you measure
accuracy = np.mean(accuracies)

# Create a new Summary object with your measure
summary = tf.Summary()
summary.value.add(tag="%sAccuracy" % prefix, simple_value=accuracy)

# Add it to the Tensorboard summary writer
# Make sure to specify a step parameter to get nice graphs over time
summary_writer.add_summary(summary, global_step)