What is an "Instanton" in classical gauge theory? (to a mathematician)
By itself, a (Yang-Mills) instanton is a classical concept. It is a solution of the classical Yang-Mills equations (considered on a manifold with a Riemannian, rather than a Lorentzian, metric), such that the classical Yang-Mills action functional evaluated on this solution is finite (not divergent). Also, the concept of instanton is not restricted to Yang-Mills type gauge theories and applies to other kinds of field theories as well.
Though classical, instantons have applications in quantum theory, at least heuristically. When the path integral (still in Riemannian/Euclidean signature) is formally considered in the saddle point approximation, instantons are precisely the saddle points which determine the asymptotics of the approximation.
A linguistic remark: "Instantons" are the same mathematically to "solitons", particle-like solutions of classical field theories (explaining the suffix "on"). Unlike solitons, instantons are structures in time (explaining the prefix "instant").
A mathematical remark (using Donaldson's book on Yang-Mills Floer homology, appendix C of section 2.8), which supplements Igor Khavkine's answer:
Consider the Yang-Mills equations over the (3+1)-dimensional spacetime $Y\times\mathbb{R}$, using the Lorentzian metric $dy^2-dt^2$ (this is the "real life" physical picture). Yang-Mills solutions are solutions to the Euler-Lagrange equations $d_A^\ast F_A=0$ of the Yang-Mills functional $\int_{Y\times\mathbb{R}}(|E|^2-|B|^2)$, where we have decomposed $F_A=\ast B+E\wedge dt$. These solutions can be viewed as paths $[A_t]$ in the configuration space $\mathcal{B}_P$ of (gauge equivalence classes of) connections on the principal bundle $P\to Y$. In this viewpoint, $B$ is the curvature of $A_t$ (on $Y$), and $E$ is the velocity vector of the path $[A_t]\subset\mathcal{B}_P$, and the Yang-Mills functional is thus $\int(||\nabla_tA_t||^2-V(A_t))dt$ with $V(A_t)=\int_Y|F_{A_t}|^2$. That means the 4-dimensional Lorentzian Yang-Mills solutions can be regarded as the motions of a particle moving on $\mathcal{B}_P$ in the potential $\int_Y|F_A|^2$.
However, instantons are Yang-Mills solutions for the Euclidean metric. In the above picture, that means we need to reverse the sign of the potential, and we lose our physical description of particles. By the way, so far we have been describing the first paragraph of Igor Khavkine's answer. Let's move on to his second paragraph:
If we are to relate instantons with a physical description of particles, then we need pass to quantum mechanics on $\mathcal{B}_P$. We look for wavefunctions that are energy eigenstates for the potential $V=\int_Y|F_A|^2$, i.e. solutions to Schrodinger's equation on $\mathcal{B}_P$. Instantons will approximate these solutions. If the energies are greater than $V$, we have our usual classical picture of a ball rolling over a hill, but if the energies are less than $V$ ($E_0<V$) then we have "quantum tunnelling". Clarifying, the "leading order" approximation of Schrodinger's equation (the instantons) in these classically inaccessible regions will be given by trajectories of particle motions on $\mathcal{B}_P$ with energy $-E_0$ in the potential $-\int_Y|F_A|^2$.
This is really cool... In the toy model of a double-well potential (see Wikipedia's article on instantons), instantons are the solutions which tunnel from well to well. Mathematically that means we have a path $[A_t]\subset\mathcal{B}_P$ of connections which are asymptotic to flat connections on both ends (as $t\to\pm\infty$), and this is the setup for Yang-Mills Floer homology!
Generally speaking, you could say they are a special type of solution to the field equations of gauge theories. More specifically, an instanton is a classical solution in a classical Euclidean field theory with finite non-zero action.
The name is due to the fact that they happen for an 'instant' (a point) of Euclidean time and so they are important in the path-integral formulation of a theory which uses Euclidean signature and as critical points. As other commentators have mentioned, in QFT we are generally talking about the Yang-Mills instanton (where a Yang-Mills theory is a QFT with a non-abelian gauge group). For a mathematical interpretation, the instanton solution of the Euclidean Yang-Mills equation leads an $SU(2)$ fibre bundle over $S^{4}$ but it can also be proved that any finite action solution of the Euclidean Yang-Mills equations leads to a fibre bundle over the four-sphere (see Uhlenbeck, 1979).
If we consider the best-known example, we take a pure Yang-Mills theory with symmetry group $SU(2)$ in $R^{4}$ with Euclidean signature. The equations of motion (ie. the Yang-Mills equations) are $D*F=0$ and $DF=0$. When we introduce the condition called the anti-self-duality equation, these equations reduce to ODEs for the gauge potential $A$. If we make an ansatz for the solution to the anti-self-duality equation which only differs from a pure gauge by a function of $r$ at infinite radius, we guarantee that our solutions have finite, non-zero action. Our ansatz for the gauge field is such that it becomes a pure gauge as $r$ tends to infinity and the associated field strength disappears, meaning that the action is finite.
If we choose the appropriate gauge transformation and use this to evaluate the field strength, we then obtain an equation whose full-solution is the 'instanton potential' which is regular on all of $R^{4}$. The action is equal to $-8 \pi^{2}/g^{2}$, so it is obviously finite. See Gockeler and Schucker (1987) for more on the fibre bundle interpretation: this looks at other related fibre bundle structures such as the Dirac monopole. The fibre bundle structure of the Dirac monopole is extremely similar (basically the same) as the Yang-Mills instanton fibre bundle. (I only add this as seems like you were after a more interesting mathematical interpretation).
Following this logic we can attempt to construct a 'gravitational instanton' ie. we look for a metric with Euclidean signature described locally by an orthonormal frame and solve the Einstein field equations without matter. We end up with a solution such that $g$ and $f$ tend to $1$ as $r$ tends to infinity: this is similar to the Yang-Mills case where the Yang-Mills instanton potential becomes a pure gauge as $r$ tends to infinity. This is similar to the way that gravity ends up being 'analogous' to other gauge theories, rather than directly comparable, since gravity does not quantize well.