Why my one-filter convolutional neural network is unable to learn a simple gaussian kernel?

By examining the photos it seems like the network is learning OK, as the predicted image is not so far off the true label - for better results you can tweak some hyperparams but that is not the case.

I think what you are missing is the fact that different kernels can get quite similar results since it is a convolution. Think about it, you are multiplying some matrix with another, and then summing all the results to create a new pixel. Now if the true label sum is 10, it could be a results of 2.5 + 2.5 + 2.5 + 2.5 and -10 + 10 + 10 + 0. What I am trying to say, is that your network could be learning just fine, but you will get a different values in the conv kernel than the filter.