Which mathematical operation does the right hand rule for current come from?

It comes from the cross product. Every situation in which you have to use the right-hand rule corresponds to some mathematical equation that involves a cross product.

In this case, the relevant equation is the Biot-Savart law,

$$\vec{B} = \frac{\mu_0}{4\pi}\int \frac{I\,\mathrm{d}\vec{l}\times\vec{r}}{r^3}$$

If you use the right-hand rule - the version you know to use for cross products - to compute the cross product of $\mathrm{d}\vec{l}$ and $\vec{r}$, like this (apologies for the crude drawing):

you will get the same result as from the curling-fingers form of the right-hand rule that you've shown in your question. That's by design, and in fact the curling-fingers form of the rule is just a shortcut for an infinite number of applications of the cross-product form.

In case you're curious, there is a deeper reason the right hand rule is needed for cross products. When you take the cross product of a vector and another vector, you get a slightly different mathematical object called a pseudovector or axial vector. The magnetic field at a point is the best-known example of a pseudovector. Despite looking just like a vector, pseudovectors actually represent a magnitude and an oriented plane, whereas an ordinary vector represents a magnitude and direction. Now, if you have a plane, and you want to represent it with an arrow, in a sense you can do that by picking the arrow to be perpendicular to the plane, and then your convention is that an arrow represents the plane that is perpendicular to it. But there are two arrows perpendicular to the plane; how do you choose which one to use? That's where the right-hand rule comes in. It plays a role in mapping a pseudovector (the thing that e.g. magnetic field really is) to a vector (the thing we use to represent e.g. magnetic field).

The interesting thing about right-hand-rules is that they always manifest in pairs.

To find the direction of precession for a gyroscope you use a RHR to get the direction of the angular momentum and then use a RHR find the direction of the torque. Finally, you apply that torque as the time-rate-of-change of the angular momentum.

If you adopted the LHR for both applications you'd get the same resulting precession. In other words there is a arbitrary convention at work here.

There is a similar situation with magnetic fields, but it is less obvious that it is a pure convention. The direction of the magnetic field and the direction of the Lorentz force both involve a handed-rule and the physical observable is the force.

You could replace the RHR everywhere with a LHR and all out experiments would still work. Well, until you start describing weak decays.