Is Spivak Cheating?
Suppose we know that $\displaystyle \lim_{h \to 0} F(h) = 0$. This means that for every $\epsilon > 0$, there exists $\delta > 0$ such that $$|h| < \delta \implies |F(h)| < \epsilon.$$
Fix $x \in \mathbb{R}^n$. If $|t| < \delta/|x|$, then $|tx| = |t||x| < \delta$, so $|F(tx)| < \epsilon$. This says exactly that $\displaystyle \lim_{t \to 0} F(tx) = 0$.
In your situation, take $$F(h) := \frac{|\lambda(h) - \mu(h)|}{|h|}.$$
Edit: If $\displaystyle \lim_{h \to h_0} F(h) = L$ and $\displaystyle \lim_{t \to t_0} g(t) = h_0$, then $\displaystyle \lim_{t \to t_0} F(g(t)) = L$.
Proof: Fix $\epsilon > 0$. By the first limit hypothesis, there exists $\delta_1 > 0$ such that $$|h - h_0| < \delta_1 \implies |F(h) - L| < \epsilon$$ By the second limit hypothesis, there exists $\delta_2 > 0$ such that $$|t - t_0| < \delta_2 \implies |g(t) - h_0| < \delta_1.$$ Therefore, if $|t - t_0| < \delta_2$, then $|F(g(t)) - L| < \epsilon$. This says exactly that $\lim_{t \to t_0} F(g(t)) = L$.
Remark: Note that no continuity hypotheses were necessary. A continuity hypothesis on $F$ would be needed to show instead that $\displaystyle \lim_{t \to t_0}F(g(t)) = F(h_0)$.
You fix $x \neq 0$ and $h=tx$ for $t \to 0$. You have proved that $$ \lim_{h \to 0} \frac{|\lambda (h)-\mu (h)|}{|h|}=0, $$ so $$ \lim_{t \to 0} \frac{|\lambda (tx)-\mu (tx)|}{|tx|}=0 $$ as well. But $\lambda (tx)=t \lambda (x)$ and $\mu (tx)=t \mu (x)$ by linearity, and you conclude.
The second chain of inequality shows that $$\lim _{h\to 0} \frac{|\lambda (h) - \mu (h)|}{|h|}=0.$$ A necessary condition for the convergence is that $\phi (t)\to 0$ if $t\to 0$ , where $\phi$ is the quotient calculated in $tx$. It's something like “$a_n \to a$ implies $a_{n_k}\to a$ for each subsequence $a_{n_k}$”.