Good video quality over VNC or RDP?

So why don't the videos play well when I am using RDP or VNC, and is there any solution?

Easy. VNC and RDP are very practical/pragmatic protocols designed mainly for remote systems access and administration. Not video playback. If you remotely connect to a system to administer it, the fact the mouse lags a little is not that big of a deal because the goal is not the visual smoothness of mouse movement, but rather just making sure the pointer can be positioned correctly when you interact with the system.

The problem is that VNC and RDP are not really optimized to stream video at high quality; they are designed to stream a desktop interface in a way that allows the interface to be just usable enough to allow for the remote administration/access a system. This is done by only redrawing specific areas of the screen when needed during a remote connection session; basically a type of on-the-fly video compression optimized not for smoothness but for practical use.

To see what I mean, remotely access your desktop on the iPad and then do something like open a dozen desktop Windows; I think you will notice a visible delay and some stuttering when drawing even on a rock-solid, high-speed connection. And that is only with bare-bones desktop imagery/visuals being streamed to you.

In contrast, video playback often demands the video area be redrawn at 24 or even 30 frames per second to achieve smoothness of movement. Factor in that the video area can be something large—such as a basic HD 720x480 playback area perhaps—and VNC and RDP will just utterly choke on that since it’s not designed to compress visual data smoothly at that rate.

Unlike a desktop video playback—where the video data is compressed and a small portion of the video will be updated at a time to effect a playback stream—VNC and RDP will just attempt to redraw the whole video area at every moment something changes. And while VNC and RDP do compress the data it sends, again the compression is not optimized for video playback. The protocol VNC and RDP currently uses is just not designed or intended for smooth and high-quality video frame-rates.

That said, Microsoft seems to be taking strides to improve RDP performance in RDP 8.1 and RDP 10. But that is an exception to the rule: In general you just can’t rely on VNC or RDP to be reliable for smooth video streaming. This is why stand-alone streaming apps exist and work: They are designed to stream video streams to a playback device that knows how to properly decode—and display—that stream.