What is the purpose of 8b/10b encoding?

The simple answer for an encoding like 8b/10b is that it ensures that the encoded data stream has at least a certain amount of data transitions. Without such encoding a long stream of 0's or 1's would tend toward looking like trying to send DC through the channel.

The same philosophy applies to Manchester encoding where the net bandwidth to transmit through the channel is two times the actual data bandwidth. The advantages touted for 8b/10b is that the transmission bandwidth used is much less than the 2X imposed by Manchester.

One wants to avoid DC through the channel for a number of reasons. Whilst each transmission channel has different types of requirements the main consideration is that DC does not go through capacitor or transformer coupled circuits very well. The other big consideration, even for direct coupled circuits, is that transitions are required to be able to sync a PLL circuit on the receiver end so that clocking can be recovered to allow strobing the data bits at the correct time.

Apart from the nice properties mentioned by others, the other good things that 8b10b gives you include: 1. Easy discrimination at the receiver between link control and data symbols 2. Easy detection of ~75% of errors.

It's also surprisingly easy to build 8b10b transmitters and receivers in programmable logic; the original IBM patent specifies all of the logic operations required (and if you're too lazy for that, Chuck Benz has done it for you in Verilog).

Wikipedia has a very useful page on 8b10b as well.