Is there a secure way to have a publicly facing terminal server?

RDP is a complex protocol which requires complex implementations, and thus likely to contain bugs. The initial versions of the protocol did not include much encryption. Ulterior versions are better, and can use either a homemade encryption system (which may or may not use a certificate to embed the server public key), or SSL/TLS (which necessarily uses a server certificate). Note that, in the latter case, the TLS records are encapsulated into RDP-specific packets, so you cannot content yourself by saying "this is just SSL".

There is no conceptual reason which would make RDP inherently insecure; but the whole tower of inner protocols and the general lack of decent documentation is a problem. It is already hard to know what features will be supported by a given client. Microsoft's implementation had its lot of remote exploits, e.g. this one and that one.

Microsoft's own response is that if you want security, you need Remote Desktop Gateway, which adds yet another layer, but on the outside: a standard SSL/TLS, with user authentication, and RDP in it (so you end up with a SSL-RDP-SSL-RDP sandwich). The idea is that a public-facing SSL server is a known situation with code which has been thoroughly debugged, through years of exposition (in IIS, for HTTPS Web servers). RD Gateway won't prevent RDP holes, but exploits will be limited to people who can get through the outer SSL, i.e. people who can open a session on the server itself and already have a lot of power. Of course, RD Gateway is not free, so it is Microsoft's best interest to sell licenses for it.

Note that client support for RD Gateway is not a given, when the client software is not Microsoft's implementation.

To sum up, there is only one authoritative source for RDP (Microsoft) and they themselves have about given up the idea of claiming that their product is sufficiently bug free to be public-facing. This does not bode well. Some of the alarmist discourse on the subject might be prompted by the urge of selling RD Gateway licenses, but it makes sense that making Microsoft's RDP implementation secure is hard, because of the complexity and long history of the code.