Is there a standard for running audio/video over CAT5/CAT5e UTP?

The thing you're looking for are called Baluns:

For Distributing Audio/Video Via Cat5 Cable

Today's A/V industry has created the need to send high resolution audio and video signals over increasingly long distances of cable. However, the size and cost of the video cabling, especially when long lengths are involved, had created a challenge. As needs and technology change, future cable replacement can be quite costly.

A solution has been developed to tackle this problem. By placing "balun" adaptors at each end, signals can be adapted to be effectively sent over long distances of Cat5 cable.

i found a company that has a whole product page of them.

One of the product pics has this schematic on the actual device:
_{(source: mcmelectronics.com)}

So you can steal their pinouts and use that.