Formal basis for variable substitution in limits

The complete story is as follows:

If the functions $g:\ A\to B$ and $f:\ B\to C$ have limits $$\lim_{x\to\xi}g(x)=:\eta\ ,\qquad \lim_{y\to\eta}f(y)=:\zeta\ ,$$ and if $f$ is continuous at $\eta$ in case $\eta$ occurs as value of $g$, then $$\lim_{x\to\xi}f\bigl(g(x)\bigr)=\lim_{y\to\eta} f(y)\ .$$

This holds also if any one of $\xi$, $\eta$, $\zeta$ is $\ =\infty$.

The extra condition "and if $f$ is continuous $\ldots$" is usually fulfilled, but one cannot do without it: Consider the example $g(x):\equiv 1$ and $f(y):=2$ $\ (y=1)$, $\ f(y):=3$ $\ (y\ne1)$. Then $\lim_{x\to1}f\bigl(g(x)\bigr)=2$, but $\lim_{x\to1}g(x)=1$, $\ \lim_{y\to1}f(y)=3$.

Yes, that's essentially the idea.

But this follows from the definition of continuity: we just need to show that if $f$ is continuous at $g(a)$ and $g$ is continuous at $a$, then $f\circ g$ is continuous at $a$. Because then both sides evaluate to $f(g(a))$, by definition of continuity (which requires that the limit exist and be equal to evaluating the function at the point.

To prove that if $g$ is continuous at $a$ and $f$ is continuous at $g(a)$, then $f\circ g$ is continuous at $a$, note first that $f\circ g$ is defined at $a$. Now, let $\epsilon\gt0$. then we know that there exists $\delta_1\gt 0$ such that if $|y-g(a)|\lt\delta_1$, then $|f(y)-f(g(a))|\lt\epsilon$; this holds, because $f$ is continuous at $g(a)$.

Now, since $g$ is continuous at $a$, and $\delta_1\gt 0$, this means that there exists $\delta\gt 0$ such that if $|x-a|\lt\delta$, then $|g(x)-g(a)|\lt\delta_1$.

Thus, if $|x-a|\lt\delta$, then $|g(x)-g(a)|\lt\delta_1$, and therefore $|f(g(x))-f(g(a))|=|f\circ g(x) - f\circ g(a)|\lt\epsilon$.

Thus, for all $\epsilon\gt 0$ there exists $\delta\gt 0$ such that if $|x-a|\lt\delta$, then $|f\circ g(x) - f\circ g(a)|\lt\epsilon$. Therefore, $f\circ g$ is continuous at $a$.

Therefore, we have that $\lim\limits_{y\to g(a)} f(y) = f(g(a))$, since $f$ is continuous at $g(a)$; and $\lim\limits_{x\to a}f\circ g(x) = f\circ g(a) = f(g(a))$, since $f\circ g$ is continuous at $a$.

You don't quite need $g$ to be continuous: if $\lim\limits_{x\to a}g(x)=L$ and $f$ is continuous at $L$, then we have $$\lim_{x\to a}f\circ g(x) = \lim_{y\to L}f(y) = f(L).$$ To verify this, let $\epsilon\gt 0$. Then there exists $\delta_1\gt 0$ such that for all $x$, $|y-L|\lt\delta_1$ implies $|f(y)-f(L)|\lt\epsilon$. Since $\lim\limits_{x\to a}g(x)=L$, there exists $\delta\gt 0$ such that if $0\lt |x-a|\lt \delta$ then $|g(x)-L|\lt\delta_1$. So, suppose that $0\lt |x-a|\lt\delta$. Then $|g(x)-L|\lt\delta_1$, and therefore $|f(g(x))-f(L)|\lt\epsilon$. Therefore, for every $\epsilon\gt 0$ there exists $\delta\gt 0$ such that if $0\lt |x-a|\lt\delta$, then $|f(x)-f(L)|\lt\epsilon$. This proves that if $\lim\limits_{x\to a}g(x) = L$ and $f$ is continuous at $L$, then $$\lim\limits_{x\to a}f\circ g(x) = \lim\limits_{y\to L}f(y) = f(L).$$ In particular, if $g$ is continuous at $a$, then we replace $L$ with $g(a)$.

We cannot omit the continuity of $f$ at $L$, though: take $g(x) = 0$ for all $x$, and let $$f(x) = \left\{\begin{array}{ll} 1 &\text{if }x\neq 0\\ 0 &\text{if }x=0. \end{array}\right.$$ Then $\lim\limits_{x\to a}f(g(x)) = 0$, because $f(g(x))=f(0)=0$. But $$\lim\limits_{y\to 0}f(y) = 1,$$ because we never take the value $y=0$ in evaluating the limit.

One can replace continuity of $f$ with other conditions; for example, we may ask that $g$ have a limit $L$ at $a$, and moreover, that for every $\delta\gt0$ there exist an $\eta\gt 0$ such that $g$ takes all values on $(L-\eta,L+\eta)$, except perhaps $L$ itself, on $(a-\delta,a+\delta)-\{a\}$.

Added. The situation with limits as $x\to\infty$ is essentially the same if $\lim\limits_{x\to\infty}g(x)$ exists and is real. It is more complicated when the limit of $g$ is $\pm\infty$. See this answer for some discussion on that.