Loud pop noise while plugging to a headphones jack
That's a "DC pop". A sound engineer dies somewhere every time you make one!
simulate this circuit – Schematic created using CircuitLab
Figure 1. Amplifier output and disconnected speaker.
Amplifiers fed from a single-rail supply hold their output at 1/2 supply when quiet. If the speaker was connected directly to the amplifier output there would be a DC current flowing through it continuously, heating it up and biasing the cone off centre. It would have limited travel in that direction and, so, would clip and distort much earlier than it should.
By adding a de-coupling capacitor the DC is blocked but the AC can pass through.
If the amplifier is switched on without the speaker connected as shown in Figure 1 the left side of C1 will be pulled to V+/2. The right side will follow suit. When the speaker is plugged in a current will flow to ground until the right side reaches zero volts.
A similar result will occur when plugging an audio source into a device with a DC blocking capacitor on the input.
Note also that if the devices are disconnected while powered up the capacitor may discharge back to power-up conditions due to capacitor leakage or PCB leakage.
simulate this circuit
Figure 2. Adding a discharge resistor.
As noted in the comments, we can add a discharge resistor to the circuit so that after some delay the right side of C1 has fallen to 0 V. Since this resistor will provide additional loading to the circuit we don't want to make it any lower than necessary.
One approach to finding a solution would be to decide how long we can wait for discharge.
Figure 3. Capacitor discharge curve. The capacitor discharges by 63% per RC time period.
A handy rule of thumb is that the "time constant" of an RC circuit is given by multiplying R and C. \$ \tau = RC \$. After \$ \tau \$ the voltage will have decayed by 63%. After \$ 3\tau \$ it will have decayed by 95% and \$ 5\tau \$, 99%.
Let's say we want to incorporate a resistor that will discharge by 99% in 1s: \$ 5 \tau = 1\;s \$ so \$ \tau = 0.2 \;s\$. In our example C1 is 470 µF so \$ R = \frac {\tau}{C} = \frac {0.2}{470\mu} = 420 \; \Omega \$. This is more than ten times the speaker impedance so it won't load the circuit too much.
"Exactly like when a charged object is discharged and it needs some time to charge up again."
That is exactly what you are hearing. The output is AC-coupled through a capacitor to the headphones to block any DC component of the signal. With the headphones first unplugged the capacitor had one side open circuit so it stays discharged.
When you plug in the headphones, the capacitor needs to charge to whatever the DC voltage is before it... and POP.. you hear that current as it energizes the speakers in the headphones.
Unplug and plug again, the capacitor retains that charge a long time.
Note the same effect can be heard through the speakers when first turning on an amp if special measures are not taken to limit it.
This can be significantly corrected by adding a largish resistor to ground across the outputs of the amp on the right of the cap.
ADD By the way, the reason you only hear it on one speaker is likely to do with the way jack plugs work. The right speaker is likely attached to the tip of the plug, as such, as you plug it in it first makes contact with the left channel, charging that cap, then makes contact with its own channel charging that, too. By the time the left speaker contact makes its connection, its output capacitor is already charged.
@Transistor is right, however there is a tiny subtlety here...
Jack connectors short everything to ground when you plug them in. When you slide it in, the various metal bits of the jack and the plug make contact in every possible combination before the jack finally settles into position.
There's a little buzz when the tip of the jack is inserted
I would guess that in this position, both channels are shorted together, and the opamps on both outputs go in and out of short circuit protection, or misbehave in other weird ways that cause this buzz.
When they go into protection, most likely the output goes to ground, so the cap ends up discharged... or maybe they clip to the rail, who knows.
And when the jack slides into place, both opamps come out of protection and snap their outputs to Vcc/2, therefore you hear something loud.
The weird thing is that your pop is the wrong polarity. It should be positive. Either your soundcard inverts polarity, or what I said above is true, and there is some weird clipping/instability going on as the outputs are shorted during insertion.