What are the disadvantages of Tor?
Performance sucks
It's better than a couple of years ago, but still not great.
Low latency anonymizers are prone to traffic analysis.
In particular if somebody can observe your traffic and your target's traffic, they can correlate that.
Anonymous remailers avoid this problem by adding longer delays, but you can't use them for interactive applications, such as browsing the web.
Exit nodes see your traffic in plain
If you don't use a secure protocol on top of Tor, the exit node can sniff your passwords etc. Using SSL or SSH you should be fine, but be sure to validate the certificate/fingerprints.
This doesn't apply to using hidden services, since there is no exit node, and MitM attacks are prevented by the fingerprint in the domain. But be careful that you're accessing the correct domain, they're pretty hard to memorize/verify.
One interesting variant is connecting to a VPN over Tor. That helps with 3), provided you trust the VPN, but you need to figure out a way to buy VPN access anonymously.
SSH is very sensitive to latency, since every single key stroke implies a round-trip to the server. Doing SSH over a high-latency link is a recipe for high levels of frustrations. I already find more than 200 ms of latency unbearable after a few minutes of SSH -- with Tor, you'll get much more. That point alone will make you regret considering Tor several times per day.
As for the security benefits... well, you used the expressions "misdirection" and "fewer would be aware of it". That's security through obscurity, which is not very popular in these parts. Such measures do not bring much security overall (and, worse, do not bring quantifiable security). They will deter amateur attackers who are not knowledgeable or not very motivated; but these small fry attackers would not have been a big threat anyway. The real danger lies in powerful, competent attackers who will make the effort to do a bit of research on their target, and, for instance, find this very security.SE post where you argue, in plain words, about restricting access to Tor users. In any case, competent attackers will use Tor, if only to hide their tracks. Therefore, a case could be made about not restricting SSH access to Tor users, but quite the opposite: block SSH access from Tor.
If you want "misdirection" just to keep your log files smaller, set the SSH server to a port distinct from the standard 22. It would be a grave error to believe that it increases security, but it will avoid the thousands of daily connection attempts from botnet which try to extend their territory in a purely automated fashion; thus, you will keep smaller log files and correspondingly increase your chances of spotting fishy business in the said log files.
For security, use SSH key-based authentication and/or good, very random passwords.
If I understand you correctly, your objective is to use Tor to increase the security of the communications channel between yourself and your web server?
Tor's main purpose is anonymity. It does not secure the communications channel between the exit node and the destination. Proper use of SSH (use a passphrase) should be more than adequate.
The only reason you would want to use Tor in this scenario is if you need to hide the fact that you're admin-ing a particular server. Based on your description, this isn't a consideration, so just use SSH.