Black-box fuzzing a TCP Port running an unknown applicaiton

You are correct: technically, fuzzing is usually regarded as sending invalid or random requests/data, it's implied that you know what you're testing in order to "break" the input. In some terminology (PDF) white-box fuzzing is the close to former (generated input) and black-box fuzzing (random input) is the latter.

What you're attempting is better described as just "black box testing". The general problem here is that while some protocols (SMTP, IMAP) freely offer details with banners, or some (HTTP) are overly chatty about protocol transgressions, there are many that need a magic protocol handshake (LDAP, RPC, and many more).

Try nmap again, but with the version detection turned up to 11 (actually only to 9, but no matter):

nmap -sV --version-all --all-ports -p $port $host

nmap -vvv -A --reason --script="+(safe or default) and not broadcast" -p $port $host

Note in the second example the script prefix of + -- this means run scripts even though they would not ordinarily run. Many plugins will fail to run anyway, you'll need to read the output carefully. Hopefully this will give you some extra info (make sure to use a recent nmap, scripts often hang in old versions).

You haven't given the port or the nmap reason, so I cannot explain why it concluded it was SIP, my best guess is it either responded to a GET or OPTIONS request or it is port 5060 or 5061.

More general advice:

• it should be easy to confirm or deny the existence a public web server, tune down the nmap rate with -T1 or -T0 in case an IPS is blocking you.
• make sure to scan SSL with a recent tool in order to properly support contemporary TLS versions and options

Finally, there are other application scanners out there, amap isn't nearly as comprehensive an nmap, but it's worth a shot.

Maybe fire up Wireshark and see if one of the more popular Wireshark protocol dissectors successfully identifies the traffic while you interrogate/probe the service (i.e. use the “decode as” feature to force Wireshark to decode the packets as a particular protocol).

Current list of dissectors: http://anonsvn.wireshark.org/viewvc/trunk/epan/dissectors/

If it is some private protocol then reverse engineering it may be possible -> https://reverseengineering.stackexchange.com/a/2494.