Why does my kettle only make a noise when it is turned on

Your kettle needs incoming energy from the heating element to turn water in to steam. Steam bubbles forming and collapsing make the familiar sound. Early on many of the steam bubbles don't make it to the top because they cool off when they rise away from the heating element. This is why the familiar rumbling sound starts way before the water boils. The water is actually boiling around the surface of the heating element but is cooler away from it.

The reason is stops right away when you turn off the power is because water needs energy to transition from liquid to steam. Even though boiling water and steam are actually both at 100 degrees Cecilius steam has more kinetic energy. This is why steam burns can be worse than the boiling water. For water to turn to steam it must acquire enough energy to jump the energy gap. This is why the kettle needs a constant input of new energy to create new steam even though the steam and water are at the same temperature. Turn it off and the process stops and it stops quickly because the water not balanced precariously on the edge between liquid and steam. There's an energy gap to overcome.

The noise is either from the AC electricity, which would be a 60Hz buzzing, or from small bubbles forming on the heating element itself. When the electricity stops, both the buzzing and the bubble formation will stop as well.

Bubbles create sound due to quickly expanding from a small nucleus.

Here's a book I found with a section on noise from bubble creation