How do I enter a literal tab character in a bash shell?

Don't use double quotes.

sort -t $'\t'

Or I think Ctrl V inserts a Tab??

Edit:

http://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoting.html#ANSI_002dC-Quoting

Try Control-v, then Tab. If you see the cursor tab over to the right, it worked.

According to the comment by Mark you can also try Control-v and then Control-i.