Why do reciprocal functions work for undefined values?

The reciprocal of $5/(x-3)$ is $(x-3)/5$ except at $x=3$ where it is undefined.
So you look near $x=3$, and you find it approaches $0$ both when $x<3$ and when $x>3$.
That is called a 'removable discontinuity'.
You might as well define $g(x)$ to be $(x-3)/5$ except at $x=3$, and $0$ at $x=3$. But that is just $g(x)=(x-3)/5$ for all $x$.

If you want to be technical, you can think of only defining the reciprocal on the domain of the original function, which means that at any point $f(x)$ is undefined, $(f(x))^{-1}$ is undefined... but what's more commonly done is to try to define a function everywhere that function makes sense, and in the two functions you have mentioned, since they are continuous functions around these "bad points" and the limits exist, it seems reasonable to extend the definition of $(f(x))^{-1}$ to include these "bad points," setting the value to be the limit at the previously undefined point.

The first idea makes sense if you want to define the reciprocal as "the function $g(x)$ such that $g(x)f(x)=1$ for all $x$ such that $f(x)$ is defined." The latter idea makes more sense if you want to think of the reciprocal in its own right. For an example of the latter, there's no reason to hold back from defining $\cot(\pi/2)=\cos(\pi/2)/\sin(\pi/2)=0,$ since the middle expression is perfectly well-defined there, and $\cot(x)$ agrees with this function everywhere in a neighborhood of $\pi/2.$

If $f(x)$ is undefined for some specific value $x^*$, so is $(f(x))^{-1}$. However, if $x^*$ is a value for which we have $\lim_{x\to x^*} f(x)=\pm\infty$ (undefined in your terms), then one usually uses this other (but not actually correct) reciprocal function:

$$g(x):=\begin{cases} (f(x))^{-1} &\text{for $x\ne x^*$}\\ 0 &\text{for $x=x^*$} \end{cases}.$$

This is because this function $g(x)$ equals $(f(x))^{-1}$ whereever it is defined, and otherwise extends it continuously. It is in some sense a natural extension of $(f(x))^{-1}$. It is easier to deal with this $g(x)$ than always having to keep in mind to exclude $x^*$.