Why do distinct temperature layers form in bodies of water?

The gradient is not linear because the thermocline is produced by the absorption of solar radiation. When radiation is absorbed by a material, its intensity decreases exponentially with penetration depth, so exponentially more energy is deposited in the top layers than the bottom. Therefore, if there were no mixing at all, you would expect an exponential gradient in temperature, such that most of the solar energy would be absorbed by the top few cm of water. But waves driven by the wind, tides, and currents mix the top few cm of water with lower layers, down to roughly a few hundred meters, so that the warmth is somewhat more uniformly distributed and the temperature gradient is somewhat less than exponential. In very, very deep water, the pressure at lower depths is high enough to affect the heat capacity of seawater, which might change the temperature gradient (plus, there may be deep ocean currents responsible for mixing in lower layers).

So what you should see should basically be a distorted exponential curve, and that's essentially what most thermocline curves show.

The three layers are depicted here (from Wikipedia):

The top and bottom layers are distinguished by mostly constant temperature across their depth.

The top layer mixes due to external factors like wind.

The bottom layers mixes because all the water is close to its highest density and the density is approximately the same across the entire layer. Therefore, very little energy is required to mix it.

The remaining middle layer is where the water shows the expected temperature drop.

An important point is that the three layers all exist for different reasons and are not caused by the same physical phenomenon and are not homogeneous.