Inverse of a Function exists iff Function is bijective
Here's a crazy answer, in the sense that it's very formal. First, for a pair of sets $A$ and $B$, define $\pi_1: A \times B \to A: (a, b) \mapsto A$, and similarly define the map $\pi_2$ to $B$.
Now a function is (at least according to one formal definition) a triple $$ f = (X, Y, R) $$ where $R$ is a subset of $X \times Y$ with two properties:
$\pi_1 ( R) = X$, i.e., every element of $X$ appears as the first element of some ordered pair in $R$, and
$\pi_1(x, y) = \pi_1(x, y') \Rightarrow y = y'$, i.e., each element of $X$ corresponds to at most one element of $Y$.
In this context, the definition of "surjective" is $\pi_2(R) = Y$, and the definition of "injective" is $\pi_2(x, y) = \pi_2(x', y) \Rightarrow x = x'$.
If we have a surjective and injective function, $f = (X, Y, R)$ we can build a new function $g = (Y, X, R')$, where $R' = \{ (y, x) | (x, y) \in R \}$.
Clearly the properties of "surjective" and "injective" for $f$ turn into properties $1$ and $2$ for $g$; also pretty evident is that $g$ is the thing we usually call $f^{-1}$.