When installing Mint Linux in VMware player which OS version should I select?

This box is used only to determine minimum hardware required to run given OS, best compatibility options and installation automation method.

Speaking of requirements and compatibility, Mint should work fine with Ubuntu presets. I'm not sure about setup automation, but you can give it a try. In case of setup failing take notes on what hardware Player had assigned for the Ubuntu preset, then create new VM but this time choosing Other Linux 3.x kernel and configure identical hardware. Installation automation will be skipped for Other.

Mint is a spin-off of Ubuntu which is a spin-off of Debian.
So first preference is Ubuntu 64bit, then Debian 64bit.
As a last resort try "Other Linux" with the highest 64bit kernel available and then work you way downwards to earlier kernel presets if needed.