The "inverse" of $\nabla\times$ operator

First, note that we expect such an inverse will be nonlocal, since the inverse of the standard differential operator $d/dx$ is the integral $\int_a^x \, dx'$.

Suppose in the simplest case we have the simply connected domain $\mathbb{R}^3$ (with enough decay at infinity that whatever integrals we write down will converge), and we are trying to solve $$ \nabla \times A = B $$ for $A$. Taking another curl gives $$ \nabla \times (\nabla \times A) = \nabla \times B, $$ and it looks like I've made things worse. But we have $$ \nabla \times (\nabla \times A) = \nabla(\nabla \cdot A)-\nabla^2 A, $$ where the last term is the vector Laplacian. Now, if we can say that $\nabla \cdot A=0$ (which it's admittedly not clear is possible; let's come back to that), then we need to solve the equation $$ -\nabla^2 A = \nabla \times B. $$ But if we use Cartesian coordinates, the vector Laplacian acts like the ordinary Laplacian on each component of $A$; therefore, we can invert it using the Green's function for the Laplacian (if you haven't met this, it's given by the solution to $-\nabla_x^2 G(x-y) = \delta(x-y)$ satisfying the right boundary conditions), which in this case is $-1/(4\pi |x-y|)$. Then we define $$ A_B(x) = \int_{\mathbb{R}^3} G(x-y) (\nabla \times B)(y) \, dy=\int B(y) \times [\nabla G(x-y)] \, dy, $$ integrating by parts. Does this work? Well, $$ \nabla \times (X \times a) = (a\cdot \nabla)X-(\nabla \cdot X)a, $$ so we have $$ \nabla \times A_B = \int (B(y) \cdot \nabla)\nabla G(x-y) \, dy + \int B(y) (-\nabla^2 G(x-y)) \, dy; $$ the former term is zero because if we integrate it by parts, we get a $\nabla \cdot B$, which is zero since $B$ is supposed to be the curl of something. The second term is just $B(x)$ by the definition of the Green's function!

(Some more care is needed in the above: deciding how to actually turn the $\nabla_x$ into a $\nabla_y$ and so on, but that's the right idea.)

Okay, that works. Now let's tidy up. We firstly want to show that we can take $\nabla \cdot A=0$. Suppose we define $\Lambda$ so that $-\nabla^2\Lambda=\nabla \cdot A$ (easy enough, using the Green's function). But then $A_{\Lambda}=A+\nabla \Lambda$ also solves $\nabla \times A_{\Lambda} = B$, and has zero divergence. This also tells us how to get from our $A_B$ to a more general $A$ that does not have $\nabla \cdot A=0$: add on a gradient of something.

You have the Helmholtz decomposition in physics:

$$ {\bf F} = -\nabla \Phi + \nabla \times {\bf A}$$

which say that the differential parts of a vector fields can be decomposed as the sum of a rotation-free (scalar potential) part and a rotational part ( the curly one ). Therefore it should be impossible to "invert", as the curl only captures part of the vectors which is not part of the scalar potential. While an inverse therefore is impossible we can probably find a suitable generalized inverse or pseudoinverse, usually assuming that the missing components are 0.