Apple - Are there any MKV video players that feature Picture in Picture?
IINA (https://iina.io/)
I found that IINA can use macOS's native Picture in Picture mode.
For and only for modern macOS.
IINA is born to be a modern macOS application, from its framework to the user interface. It adopts the post-Yosemite design language of macOS and keeps up the pace of new technologies like Force Touch, Touch Bar, and Picture-in-Picture.
Source: GitHub
Picture-in-picture is a specific Safari browser and iTunes feature, and Safari is incompatible with MKV container file formats. At present there is no way to add macOS support for non-Apple approved container formats or codecs (the old Perian tool has not been updated for a very long time).
The way to achieve what you want would be to use an app like Adapter to convert the MKV file to MP4, which is natively understood by macOS apps, and then play it either in iTunes or QuickTime Player. In QuickTime Player you will need to select View -> Float on Top. This is not strictly picture-in-picture but has the exact same effect.