What does tf.train.ExponentialMovingAverage do?

Maintains moving averages of variables by employing an exponential decay.

When training a model, it is often beneficial to maintain moving averages of the trained parameters. Evaluations that use averaged parameters sometimes produce significantly better results than the final trained values.

doc: https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage?version=stable