What is the significance of negative frequency in Fourier transform?

If you have a real function of time $x(t)$, which may represent, for example, a modulated carrier, you can, if you wish, avoid negative frequencies by using the Fourier Sine Transform $$X_s(\omega)=\int_{-\infty}^{\infty}{x(t)\sin\omega t dt} $$ and Fourier Cosine Transform $$X_c(\omega)=\int_{-\infty}^{\infty}{x(t)\cos\omega t dt} $$ because then, the inverse transforms can be written (ignoring factors of 2s and $\pi$s): $$x(t)=\int_{0}^{\infty}X_c(\omega)\cos \omega t + \int_{0}^{\infty}X_s(\omega)\sin \omega t $$ i.e no negative frequencies are present.

However, it's much neater to work with the complex exponential representation of modulated signals. A generic modulated signal, in the time domain will have its amplitude and phase varying as a function of time $$x(t)=A(t)\cos(\omega_c t+\phi(t)) $$ We can write this as the real part of a complex signal $$ x(t)=\Re\{A(t)\cos(\omega_c t+\phi(t))+jA(t)\sin(\omega_c t+\phi(t))\}$$ Now I could have written any old rubbish in the imaginary part, and the formula would still have been correct, but the choice I made makes it extremely convenient - mainly because I can write it in exponential form $$ x(t)=\Re\{A(t)\exp(j(\omega_c t+\phi(t)))\}$$ and this makes it really easy for me to "factor out" the carrier frequency $$ x(t)=\Re\{X_B(t)\exp(j(\omega_c t))\} $$ where $X_B(t)$ is the "baseband equivalent" signal. "Baseband" meaning "effectively at 0Hz carrier frequency". The baseband equivalent signal is just $$X_B(t)=A(t)\exp(j(\phi (t))) $$ and is complex and contains the "interesting" information contained in the modulation (the carrier itself being not so interesting!).

Working with these exponential forms, it now seems much more natural to define the Fourier transform in terms of exponentials in the normal way: $$X(\omega) = \int_{-\infty}^{\infty}{x(t)\exp(-j\omega t)}dt $$ together with the inverse transform $$x(t) = \int_{-\infty}^{\infty}{X(\omega)\exp(j\omega t)}d\omega $$ The Fourier transform of the baseband equivalent signal $X_B(t)$ is not symmetric about zero - you can't specify it by giving its values for positive frequencies and invoking symmetry. So if you want to work with this formalism then negative frequencies are fundamental.

The actual spectrum of the signal centred at $+\omega_c$ is the one you'd get if you swept a very narrowband filter over the frequency range. It's not necessarily symmetric about $+\omega_c$ and this gets translated to the complex baseband one (shown dotted) which is now not symmetric about zero, so negative frequencies are needed.

Choose a square root of negative one.

And now throw it out because statisticians hate complex numbers and negative frequencies, and never use them if we can help it. For us, the power spectrum really measures the variance of the data, we use the word «power» because these concepts were started by physicists and engineers who were studying light or signals. The power spectrum is a kind of analysis of variance, which attributes different parts of the variance of the data to the different frequencies. There is no such thing as imaginary data, ummm, well it should be stamped out of course, it's fraud... and no such thing as non-commutative data...

If we look at the power spectrum (even of light, even in optics) the phase makes no difference, so the distinction between polarised one way or another way is irrelevant. And the actual lab analysers measure the power, ignoring the phase. (But if one is measuring cross-correlations between two different light beams, then the relative phases do make a difference since they affect the coherency of the two beams with each other.) So there is no need to use complex numbers and Shuster did not do so.

The only reason negative frequencies would be needed is if you use complex numbers instead of sines and cosines. Now many scientists prefer to use complex numbers because it makes the proofs simpler, it makes the Fourier inversion formulae more elegant, and more manageable too, and it encodes the phase information in the angle of the complex number. But then this has the disadvantage that when you want to translate it back into the actual physical terms your lab bench instruments put out in the print out, you always have to combine the contributions due to the positive frequency $\lambda$ with those due to the negative one $-\lambda$ simply because $\cos (\lambda t)$, which (along with its phase shifted sine term) is made up out of two complex exponenetials, one with positive frequency and one with negative frequency, viz., $e^{i\lambda} + e^{-i\lambda}$.

Your initial choice of a square root of negative one, which was arbitrary, is reflected in the choice of a direction of motion in time. (The same formula for the propagator will propagate you backwards in time if you choose the other square root, but then you can call backwards «forwards» and then that fixes everything...if you are a theorist...)