HTTP status code for unsupported browser

400 (bad/malformed request) is the only one that fits... somewhat

403 Forbidden is the most appropriate.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4 says:

The server understood the request, but is refusing to fulfill it.

400 Bad Request is not a good fit because it implies the request itself is malformed, which is probably not true.

If you are building a website, however, it is not a good practice to forbid specific web browsers in this way. Try to build a website that is compliant with all the browsers used by your audience. If you are building an HTTP API, then that's a different story, and you should use a secure mechanism for authorizing clients. If that's what you need, consider OAuth.

EDIT July 2015: The newer RFC 7231 elaborates its explanation of 403, making it more clear that it is okay to use even when it is not a credential authorization issue.

https://www.rfc-editor.org/rfc/rfc7231#section-6.5.3 (bold added by me):

The 403 (Forbidden) status code indicates that the server understood the request but refuses to authorize it. A server that wishes to make public why the request has been forbidden can describe that reason in the response payload (if any).

If authentication credentials were provided in the request, the server considers them insufficient to grant access. The client SHOULD NOT automatically repeat the request with the same credentials. The client MAY repeat the request with new or different credentials. However, a request might be forbidden for reasons unrelated to the credentials.