Why is digital serial transmission used everywhere? i.e. SATA, PCIe, USB

There are two main reasons for the rise of serial

1) It's possible. Low cost transistors have been able to manage GHz switching for a decade now, long enough for the capability to be used and become standard.

2) It's necessary. If you want to shift very high speed data more than a few inches. This distance starts to rule out mobo to PCI card links, and definitely rules out mobo to hard disk, or mobo/settopbox to display connections.

The reason for this is skew. If you transmit several parallel signals along a cable, then they have to arrive within a small fraction of the same clock period. This keeps the clock rate down, so the cable width has to increase. As data rates climb, that becomes more and more unweildy. The prospect for increasing the rate into the future is non-existent, double or quadruple width ATA anybody?

The way to slay the skew demon is to go serial. One line is always synchronised with itself, there is nothing for it to be skewed with. The line carries data that is self-clocked. That is, uses a data coding scheme (often 8b/10b, sometimes much higher) that provides a minimum guaranteed transition density which allows clock extraction.

The prospect for increasing data rate or distance into the future is excellent. Each generation brings faster transistors, and more experience crafting the mediium. We saw how that played out with SATA, which started at 1.5Gb/s, then moved through 3 and is now 6Gb/s. Even cheap cables can provide sufficiently consistent impedance and reasonable loss, and equalisers are built into the interface silicon to handle frequency dependent loss. Optical fibre is available for very long runs.

For higher data rates, several serial links can be operated in parallel. This is not the same as putting conductors in parallel, which have to be matched in time to less than a clock cycle. These serial lanes have only need to be matched to within a high level data frame, which can be µs or even ms long.

Of course the advantage in data width doesn't just apply to the cables and connectors. Serial also benefits PCB board area between connectors and chip, chip pinout, and chip silicon area.

I do have a personal angle on this. As a designer working on software defined radio (SDR) from the 90's onwards, I used to rail at people like Analog Devices and Xilinx (and all the other ADC and FPGA companies) (they would visit and ask us from time to time) for making me run so many parallel differential connections between multi-100MHz ADCs and FPGAs, when we were just starting to see SATA emerge to displace ATA. We finally got JESD204x, so now we can hook up converters and FPGAs with only a few serial lines.

Why there are no widespread system communication protocols that heavily employ some advanced modulation methods for a better symbol rate?

If the basic copper connection between two points supports a digital bit rate that is in excess of the data rate needed to be transmitted by the "application", then why bother with anything else other than standard differential high-speed signalling?

Employing an advanced modulation scheme is usually done when the "channel" has a bandwidth that is much more limited than copper or fibre.

If you want an example of something that is widely used, but different, look at 1000BASE-T gigabit Ethernet. That uses parallel cables and non-trivial signal encoding.

Mostly, people use serial buses because they are simple. Parallel buses use more cable, and suffer from signal skew at high data rates over long cables.