Data augmentation in test/validation set?

Data augmentation is done only on training set as it helps the model become more generalize and robust. So there's no point of augmenting the test set.

This answer on stats.SE makes the case for applying crops on the validation / test sets so as to make that input similar the the input in the training set that the network was trained on.

I would argue that, in some cases, using data augmentation for the validation set can be helpful.

For example, I train a lot of CNNs for medical image segmentation. Many of the augmentation transforms that I use are meant to reduce the image quality so that the network is trained to be robust against such data. If the training set looks bad and the validation set looks nice, it will be hard to compare the losses during training and therefore assessing overfit will be complicated.

I would never use augmentation for the test set unless I'm using test-time augmentation to improve results or estimate aleatoric uncertainty.

Only on training. Data augmentation is used to increase the size of the training set and to get more different images. Technically, you could use data augmentation on the test set to see how the model behaves on such images, but usually, people don't do it.