If we feel it's hotter when humidity increases, then why do we feel it's colder when inside water?

You feel cold when heat is flowing from you to the surroundings, your body tries to burn more energy to keep up your temperature, so you shiver.

Water conducts heat much more effectively than air (more than 100x as well) so even with water at the same temperature as air you will lose a lot more heat and feel cold.

When your body is too hot it losses energy most efficiently by sweating. It releases water which evaporates, the energy needed for the water to go from liquid to gas comes from your skin which is then cooled.

In humid conditions it is harder for the water to evaporate (because there is already a lot of gaseous water in the air) so you can't cool as efficiently and so feel hotter.

When you feel hot, you perspire so as to benefit by evaporative cooling. As the relative humidity gets closer to 100%, the sweat cannot evaporate and evaporative cooling becomes less effective.

Liquid water is a much better conductor of heat than air (even humid air) is, so if the water is even a few degrees cooler than your body, you feel cold because the water is efficiently conducting your body heat away. A room at 60°F (15°C) feels slightly chilly; in 60° water, you will die in a few hours.

The same is true of metal. On a very hot or very cold day, touch a metal pole and compare it to the feeling of a wood post: it will feel much, much hotter or colder respectively, because metal is a much better conductor of heat than wood.