Enable mouse for scrolling only in Vim in iTerm (MacOSX)

You could use

:set mouse=nicr

It works only with mouse scroll as I've tested.

If you set mouse=a, you can still use Option-click to make iTerm do selection. Not ideal, but it's the best option I've found.