Pytorch Operation to detect NaNs

You can always leverage the fact that nan != nan:

>>> x = torch.tensor([1, 2, np.nan])
tensor([  1.,   2., nan.])
>>> x != x
tensor([ 0,  0,  1], dtype=torch.uint8)

With pytorch 0.4 there is also torch.isnan:

>>> torch.isnan(x)
tensor([ 0,  0,  1], dtype=torch.uint8)

Starting with PyTorch 0.4.1 there is the detect_anomaly context manager, which automatically inserts assertions equivalent to assert not torch.isnan(grad).any() between all steps of backward propagation. It's very useful when issues arise during backward pass.