What determines color -- wavelength or frequency?

Colour is defined by the eye, and only indirectly from physical properties like wavelength and frequency. Since this interaction happens in a medium of fixed index of refraction (the vitreous humour of your eye), the frequency/wavelength relation inside your eye is fixed.

Outside your eye, the frequency stays constant, and the wavelength changes according to the medium, so I would say the frequency is what counts more. This explains why objects' colour don't change when we look at them under (transparent) water ($n=1.33$) or in air ($n=1$).

For almost all detectors, it is actually the energy of the photon that is the attribute that is detected and the energy is not changed by a refractive medium. So the "color" is unchanged by the medium...

As FrankH said, it's actually energy that determines color. The reason, in summary, is that color is a psychological phenomenon that the brain constructs based on the signals it receives from cone cells on the eye's retina. Those signals, in turn, are generated when photons interact with proteins called photopsins. The proteins have different energy levels corresponding to different configurations, and when a photon interacts with a photopsin, it is the photon's energy that determines what transition between energy levels takes place, and thus the strength of the electrical signal gets sent to the brain.

Side note: I posted a pretty detailed but underappreciated (at least, I thought so) answer to a very similar question on reddit a few days ago. I could edit it in here if you find it useful.