How to find out why Windows Server feature installation failed?

Try install it via powershell:

Add-WindowsFeature [name_of_feature]

This case: Add-WindowsFeature SMTP-Server

https://technet.microsoft.com/en-us/library/cc732263.aspx

List of available you can get by Get-WindowsFeature