When can you interchange composition of a limit?

Continuity turns out to be the right condition. As long as $f$ is continuous and the limit of $g$ exists at the point in question, then the limit will commute with composition. That is, for a given $x$ in the domain of $g$, $$ \lim_{x\to t} f(g(x)) = f (\lim_{x\to t} g(x))$$ When dealing with values at infinity, technically you want the functions to be continuous on the extended real line ( either two point or one point compactifications). In practice, that amounts to proving $$ \lim_{x\to \infty} g(x)$$ exists.

For direct substitution like that to be correct, the function you are substituting into has to be continuous at that value. For example, $\lim \limits_{x \to a} f(x) = f(\lim \limits_{x \to a} x) = f(a) $ only if $f(x)$ is continuous at $x = \lim \limits_{x \to a} x=a$.

So your generalization would be correct if the outer function $f(x)$ is continuous at the value of the inner limit $x=\lim \limits_{t \to a} g(x)$. This could be for any value of $a$, as long as continuity is ensured.