Why don't we define potential due to a magnetic field?

Actually, we do!

It's just that it's not the same "kind" of potential - and the reason for this is that magnetic forces work differently than electric forces.

Magnetic fields, if you know, do not directly exert a force on charged particles, simply for being charged. (They would exert such on hypothetical "magnetically charged particles", but we've never found any to exist.) Rather, the force they exert does one thing only: to change the direction of motion of moving charged particles.

The usual thing we call "potential" is what you can think of as a kind of "specific potential energy": it is the potential energy that a unit quantity of charge has from sitting at a certain place in an electric field, and if the particle moves between two areas of different potential, it gains or loses energy as a result of the shifting - but ever-present - pull of the electric force upon it.

Magnetic fields, though, cause no changes in energy - changing something's direction of motion takes no energy, only speeding or slowing its motion does. Think about how that a bullet fired from a gun doesn't hurt more or less depending on the direction from which it comes, only from how powerful the gun is. In theory, to deflect the bullet from one to the other direction in flight, likewise, would not take any energy (though you would need a rather strong source of deflecting force).

But nonetheless, that doesn't mean you can't still describe them using something like a potential, but it doesn't have quite the same meaning any more. As I just see you mentioned you've tried some vector calculus, I'll give this a shot. You see, there's a kind of "duality", if one will, between two operations one can do with at least three-dimensional vectors: the dot product and the cross product, which gives rise to related differential notions of the divergence and gradient, vs. curl, respectively.

The usual ideal of a "potential", that is, for a field like the electric field (and also the Newtonian gravitational field), is based on the following result. "Under certain reasonable conditions", the following implication holds true. If $\mathbf{F}$ is some sort of force field, and

$$\oint_C \mathbf{F} \cdot d\mathbf{l} = 0$$

for all closed paths $C$, then there exists a scalar function $V$ (i.e. with 3 real spatial coordinate arguments and outputting one real number) such that

$$\mathbf{F} = -\nabla V$$

Intuitively, the first equation is a kind of "conservation of energy": the left hand integral is, in effect, a work integral if $\mathbf{F}$ is serving as a force field, describing the amount of energy gained or lost (positive is gain, negative is loss) by a particle moving in a closed circuit through that field as it is pushed and pulled by the exerted force. The above implication, then, says that "if the force field conserves energy, we can describe it by a potential energy". This is how you get the usual electric potential, which is the "specific" potential energy: energy per unit of charge, which in SI units comes to be joules per coulomb, which we call as "volts". Moreover, the first equation, "under certain reasonable conditions", corresponds to the one

$$\mathbf{\nabla} \times \mathbf{F} = \mathbf{0}$$

where the left-hand side is a differential operation called "curl", and intuitively represents the amount by which a vector field, thought of as force, locally *fails* to conserve energy.

Now, it turns out that, however, there is another, analogous form but involving this integral: if

$$\mathop{\vcenter{\huge\unicode{x222F}}}_S \mathbf{F} \cdot d\mathbf{S} = 0,$$

a surface integral over a closed surface $S$, then it follows that another, vector field $\mathbf{A}$ exists such that

$$\nabla \times \mathbf{A} = \mathbf{F}$$

which is very much like the relationship to potential and, indeed, we call this $\mathbf{A}$ a vector potential.

Again, we should think about the intuitive meaning of the first integral: this integral now is a flux integral - in effect, if you imagine the field as representing the stream lines of some fluid, i.e. if the vectors returned by $\mathbf{F}$ are mass-flow, i.e. mass per time, with direction of flow, vectors, the flux integral would represent the net amount of fluid flowing into or out of that space - and to set it to zero says that, in effect the field "conserves fluid": no new fluid is destroyed or created at any point. Analogously, this corresponds to a similar "local" statement per the divergence:

$$\nabla \cdot \mathbf{F} = 0$$

which, you may recognize, is exactly the equation satisfied by the magnetic field, $\mathbf{B}$:

$$\nabla \cdot \mathbf{B} = 0$$

and says there are "no magnetic sources", i.e. no magnetic charges. In a sense, magnetic "flux", which can be thought of as a kind of "fluid", swirls around magnetic objects but none is created or destroyed, and this conservation of flux gives rise to a magnetic vector potential, also typically denoted $\mathbf{A}$. This "potential" is a vector, not scalar, quantity - and this is the answer to your question. It doesn't represent energy, but more like "specific flux", I'd suppose, though it's hard to answer and, moreover, interestingly, is much less unique.

The potential is a kind of primitive function of a vector field, primitive in the sense of being the reverse of a differentiation, ie., an integral with a variable upper limit. The derivatives of the potential in all directions represent the vector field; not all vector fields can be represented that way but some do, for example the electrostatic field. Some other vector fields do not have such representation but one may define another type of differentiation with a corresponding primitive function but it is not a scalar potential and the magnetic field is such an entity. The field usually denoted by B has a primitive function called the vector potential. The mathematical difference between the electric and magnetic fields is that the electric field is an "along a line thing" while the magnetic fields is a "surface thing".

In the electrostatic case the total work done in any loop is zero from which it follows the existence of a potential function. In the case of the B field the total flux passing through any closed surface being zero is the reason for the existence of a vector potential such that the flux through any simple surface spanned by an arbitrary loop is the same as the loop integral of the primitive function, here vector potential integrated around the loop.

There is a "magnetic potential" that appears in more advanced books and is defined as ${\bf B}= \nabla \phi$, just as ${\bf E}= -\nabla V$. (please excuse my equations if you are not familar with the $\nabla$ symbol) It's useful in regions where there is no current or no time-changing electric field, but it is "multivalued." Because of Ampere's law, if you encircle a current carrying wire, the potential decreases all the way round so when you get back to where you are started you have a different potential -- rather like the Escher prints of water-driven perpetual motion machine in https://en.wikipedia.org/wiki/Waterfall_(M._C._Escher). In many cases we can live with this multivaluedness because $\phi$ is not a potential energy (as is $V$), but is just a mathematical convenience.

This scalar $\phi$ is a simpler thing than the vector potential ${\bf A}$ (defined so that ${\bf B}=\nabla \times {\bf A}$) mentioned by hyportnex, but it is less useful.