Why is the AC coupling capacitor always 0.1uF or 0.01uF in high speed transceivers?

It's a HIGH-pass filter, not a low-pass filter. That cutoff frequency you calculated is the LOWEST that can get through, not the highest. Frequencies lower than that are blocked, and frequencies higher than that are passed.

A binary data signal isn't a single frequency. If it's truly random, it will have frequency content from near DC all the way up to about half the data rate (i.e. a 1 Gbps signal has content from DC to ~500 MHz).

Using a smaller capacitor value would block some of the low-frequency content of the signal, resulting in excessive wander of the binary signal. For example, consider if the data stream happened to have 16 ones in a row at some point. With a too-low blocking capacitor value, the signal would tend to drift back towards zero during this long run of ones.

To alleviate this, it's possible to use encoding to limit the low-frequency content of the signal. For example, Gigabit Ethernet uses 8b/10b code to limit the maximum run length and ensure that the signal is balanced between ones and zeros. Given a maximum run length of zeros or ones, it's now possible to choose a minimum capacitor value that allows you pass the high frequency content in the signal without allowing excessive wander during the (limited length) runs of similar values.