High memory usage but no process is using it

I understand you're using Hyper-V, but the concepts are similar. Maybe this will set you on the right track.

Your issue is likely due to virtual memory ballooning, a technique the hypervisor uses to optimize memory. See this link for a description

I observed your exact same symptoms with my VMs in vSphere. A 4G machine with nothing running on it would report 30M used by cache, but over 3G "used" in the "-/+ buffers" line.

Here's sample output from VMWare's statistics command. This shows how close to 3G is being tacked on to my "used" amount:

vmware-toolbox-cmd stat balloon
3264 MB

In my case, somewhat obviously, my balloon driver was using ~3G

I'm not sure what the similar command in Hyper-V is to get your balloon stats, but I'm sure you'll get similar results