IIS SSL error "ssl_error_rx_record_too_long"
"This is a typical sign that the SSL port on your IIS server is actually serving plain HTTP."
Exactly. I had to change Skype to not use ports 80 & 443, that did the trick. Based on solution in this thread: Applying SSL certificate in IIS manager and "The process cannot access the file because it's being used by another process" error
This is a typical sign that the SSL port on your IIS server is actually serving plain HTTP.
Have you set the SSL port directive in IIS to 443? Perhaps if you can show your IIS config?
Also this MS support article may help you: http://support.microsoft.com/kb/324069.
NOTE - I've voted to migrate this to serverfault.com as it's system-admin related as opposed to programming-related.
I had this same exact problem. Beat my head on the wall for a while, restarting IIS, checking all the settings, Googling for possible answers, etc. etc. Then I did the magical cure-all for every Windows system: I restarted the system. Everything worked perfectly after that.