How do moving charges produce magnetic fields?

If you are not well-acquainted with special relativity, there is no way to truly explain this phenomenon. The best one could do is give you rules steeped in esoteric ideas like "electromagnetic field" and "Lorentz invariance." Of course, this is not what you're after, and rightly so, since physics should never be about accepting rules handed down from on high without justification.

The fact is, magnetism is nothing more than electrostatics combined with special relativity. Unfortunately, you won't find many books explaining this - either the authors mistakenly believe Maxwell's equations have no justification and must be accepted on faith, or they are too mired in their own esoteric notation to pause to consider what it is they are saying. The only book I know of that treats the topic correctly is Purcell's Electricity and Magnetism, which was recently re-released in a third edition. (The second edition works just fine if you can find a copy.)

A brief, heuristic outline of the idea is as follows. Suppose there is a line of positive charges moving along the $z$-axis in the positive direction - a current. Consider a positive charge $q$ located at $(x,y,z) = (1,0,0)$, moving in the negative $z$-direction. We can see that there will be some electrostatic force on $q$ due to all those charges.

But let's try something crazy - let's slip into $q$'s frame of reference. After all, the laws of physics had better hold for all points of view. Clearly the charges constituting the current will be moving faster in this frame. But that doesn't do much, since after all the Coulomb force clearly doesn't care about the velocity of the charges, only on their separation. But special relativity tells us something else. It says the current charges will appear closer together. If they were spaced apart by intervals $\Delta z$ in the original frame, then in this new frame they will have a spacing $\Delta z \sqrt{1-v^2/c^2}$, where $v$ is $q$'s speed in the original frame. This is the famous length contraction predicted by special relativity.

If the current charges appear closer together, then clearly $q$ will feel a larger electrostatic force from the $z$-axis as a whole. It will experience an additional force in the positive $x$-direction, away from the axis, over and above what we would have predicted from just sitting in the lab frame. Basically, Coulomb's law is the only force law acting on a charge, but only the charge's rest frame is valid for using this law to determine what force the charge feels.

Rather than constantly transforming back and forth between frames, we invent the magnetic field as a mathematical device that accomplishes the same thing. If defined properly, it will entirely account for this anomalous force seemingly experienced by the charge when we are observing it not in its own rest frame. In the example I just went through, the right-hand rule tells you we should ascribe a magnetic field to the current circling around the $z$-axis such that it is pointing in the positive $y$-direction at the location of $q$. The velocity of the charge is in the negative $z$-direction, and so $q \vec{v} \times \vec{B}$ points in the positive $x$-direction, just as we learned from changing reference frames.

Electric and magnetic fields are what the electromagnetic field 'looks like' from a particular (inertial) frame of reference.

Take a charged particle: In its rest frame, it appears to generate an electric field only and no magnetic field at all. From a different frame of reference (in particular one in relative motion), we'll see the charge moving, thus a current which generates a magnetic field as well.

This does not mean that setting the particle in motion somehow flipped a switch within the particle - rather, it's an artifact of our choice of frame of reference: Observers in relative motion will measure different strengths of electric and magnetic fields the same way they measure different velocities and momenta.

There are however invariants of the electromagnetic field, i.e. things all observers can agree upon, and in particular $$ \begin{align*} P &= \mathbf {B}^2 - \mathbf E^2 \\ Q &= \mathbf E\,\cdot\mathbf B \end{align*} $$

Let's take a nonzero em field with $P,Q=0$, i.e. $\mathbf E^2=\mathbf B^2$ and $\mathbf E\perp\mathbf B\;.$ An example would be a plane electromagnetic wave, which will look like a plane wave for everyone.

Now, let $P\not=0$ but $Q=0\;.$ Then, we can find frames of reference where either the electric (in case of $P>0$) or the magnetic field (in case of $P<0$) vanishes. The rest frame of our charged particle would be such a one.

For more details, you'll need to look into the literature on special relativity.

Although Chris White’s answer to the question “Why Moving Charges Produce a Magnetic Field?” posted by a High School teacher (Claws) last year, was selected as the best answer, I think it contains several pitfalls. Chris White imagines a stream of positive charges flowing in the $+z$ axis direction, while a test charge $+q$ initially located at $(1,0,0)$ is moving in the opposite $(-z)$ direction with speed $v$. Next he intends to prove that when the observer locates himself in the frame of the moving test charge, he will see, in addition to the regular electrostatic Coulomb (repulsion) force acting on the test charge, an additional repulsion in the $+x$ direction whose origin is entirely relativistic. This happens, he says, because the original separation $Δz_0$ between the charges (when seen from the Lab rest frame) is now contracted to $Δz = Δz_0\sqrt{(1-v^2/c^2)}$ (The “famous” Lorentz contraction).

Consequently all the distances of the flowing charges to the test charge become smaller (as if the charge density increased) and, hence, Coulomb repulsions also increase. This excess of repulsion is the “illusory” magnetic force that the Lab observer sees when the test charge moves in the $–z$ direction with speed $v$.

In short: there is no intrinsic magnetic force. All is Coulomb force, seen from the Lab frame (pure electrostatic force), or seen from the moving charge frame (electrostatic plus more Coulomb repulsion). We can bypass here all the quantitative details which White also omits, but we cannot overlook the pitfalls:

1. First there is a verbal contradiction: to notice the contracted $Δz$, smaller than $Δz_0$, the observer must locate himself at rest with the charge $q$ (i.e., moving with the charge). But then at the end, White says that the new “anomalous force seemingly experienced by the charge” (i.e., the defined magnetic field), occurs “when we are observing it not in its own rest frame” (emphasis mine). So, what’s the deal? To predict the extra Coulomb (magnetic) force we have to adopt the frame of the moving charge. But to observe it we have to remain in the Lab frame, which is NOT the moving charge frame.
2. In the same vein there is a numerical pitfall: the new (contracted) charge separation Δz observed from the frame of the moving charge is calculated as $Δz=Δz_0\sqrt{(1-v^2/c^2)}$ where $v$, says White, is “$q$’s speed in the original frame”. He should have put not $v$ but $2v$, since the relative velocity between the charge stream going up, $v$, and the test charge going down, $-v$, is $v-(-v) = 2v$. So the contraction factor should be $\sqrt{1-4v^2/c^2}$.
3. Furthermore, if we use the heuristic strategy used by White, we reach a contradiction: Start with all charges at rest: the $z$ axis full of charges and the test charge at $(1,0,0)$. Call $Δz_0$ the separation between all charges at rest. Now allow the $z$ axis charges to move as before, with a speed $+v$. Already the Lab observer AND THE TEST CHARGE $q$, will see a contraction of the separation according to $Δz = Δz_0\sqrt{(1-v^2/c^2)}$. Hence by the same maneuvers as before, special relative must predict an additional “Coulomb” repulsion due to the compacted charge density. So the “magnetic” force, thus predicted, must act on the RESTING charge at $(1,0,0)$. And this is not observed. To the best of my knowledge, no current along the $z$ axis can ever produce a magnetic force on a resting charge at the origin.

In conclusion: contrary to what White says, magnetism is NOT JUST electrostatics plus special relativity. Such reductionistic view converts magnetism into a superficial play-game between frames of reference.