Can a hacked torrent client be used to prevent others from torrenting?

You can poison a particular torrent with false IPs quite easily. The tracker is just a script that takes an announcement request over HTTP(S) with a torrent's infohash, the IP address and port of the peer, and some details about how much of the torrent has downloaded for that peer. The IP given is not required to match the source IP of the announcement.

There are a number of attacks beyond this, e.g. having peers at these address which return garbage data.

In terms of defense against this, modern BitTorrent clients are quite robust. For example, DHT allows the user to reach additional clients as long as it has contacted at least one legitimate peer. Clients also detect many common tricks to falsify availability and ban those peers for the remainder of the download. This means that for a torrent with any significant number of legitimate seeds and peers, the false peers would have to vastly (1000-to-1 or more) outweigh them in order to have any real effect. Of course the side-effect here is that the torrent jumps to the top of the list on search engines, because it appears to be very popular, which is the opposite of what most intellectual property holders want.

Most interestingly about the torrent system is that it is a real world application of game theory. Each client typically uses a tit-for-tat strategy.

BitTorrent peers have a limited number of upload slots to allocate to other peers. Consequently, when a peer's upload bandwidth is saturated, it will use a tit-for-tat strategy. Cooperation is achieved when upload bandwidth is exchanged for download bandwidth. Therefore, when a peer is not uploading in return to our own peer uploading, the BitTorrent program will choke the connection with the uncooperative peer and allocate this upload slot to a hopefully more cooperating peer.

Therefore, it should be secure against the attack vector in question by virtue of it's nature.

There are a variety of torrent poisoning techniques; however, BT has been developed to be resistant to these types of attack (up to a certain extent). I'm not sure about the counter-measures currently existing, but it's not difficult to identify uncooperative peers and block them. You might have missed a couple of chunks by then, but nothing dramatic. Unless you can fake a lot of peers.

See also How does torrent poisoning work?