Derivative of the magnitude of a vector. Does it exist, or not?

CASE $1$: $\vec r(t)\ne 0$

Note that for $\vec r \ne 0$, we can write

$$\begin{align} \frac{ds(t)}{dt}&=\frac{\vec r(t)\cdot \frac{d\vec r(t)}{dt}}{|\vec r(t)|}\\\\ &=\bbox[5px,border:2px solid #C0A000]{\hat r(t)\cdot \frac{d\vec r(t)}{dt}} \tag 1 \end{align}$$

where in $(1)$, $\hat r(t)=\frac{\vec r(t)}{|\vec r(t)|}$ is the position unit vector. However, the unit vector $\hat r$ is undefined at the origin.

This fact does not automatically imply that the derivative $s'(t)$ fails to exist at $\vec r=0$. In the ensuing analysis, we will explore whether $s'(t)$ exists at $\vec r=0$.


CASE $2$: $\vec r(t)=0$

Assume that at $t_0$, $\vec r(t_0)=0$. We assume that $\vec r''(t)$ exists. Then, the derivative of $s(t)$ at $t_0$, if it exists, is given by

$$\begin{align} s'(t_0)&=\lim_{h\to 0}\left(\frac{\left|\vec r(t_0+h)\right|-\left|\vec r(t_0)\right|}{h}\right)\\\\ &=\lim_{h\to 0}\frac{\left|\vec r(t_0+h)\right|}{h} \\\\ &=\lim_{h\to 0}\frac{\left|\vec r'(t_0)h+O(h^2)\right|}{h} \\\\ &=\bbox[5px,border:2px solid #C0A000]{\lim_{h\to 0}\left(\frac{|h|}{h}\,\left|\vec r'(t_0)+O(h)\right|\right)} \tag2 \end{align}$$

If $\vec r'(t_0)=0$, then from $(2)$ we see that $s'(t_0)=0$ also. However, if $\vec r'(t_0) \ne 0$, then the limit fails to exist since the limits from the right-hand side and left-hand side are unequal.


Putting everything together, we find that

$$s'(t)=\begin{cases}\hat r(t)\cdot \frac{d\vec r(t)}{dt} &,\vec r(t) \ne 0\\\\0&,\vec r(t)=\vec r'(t)=0\\\\\text{fails to exist}&,\vec r(t) =0,\vec r'(t)\ne 0\end{cases}$$


Your confusion stems from a subtlety regarding differentiation of composite functions. In particular, your calculation of $s'(t)$ uses the chain rule, but the chain rule has hypotheses.

Define $s$ to be $f\circ g$ where $g$ sends $t\mapsto\mathbf{x}(t)$ and $f$ sends $\mathbf{x}\mapsto\|\mathbf{x}\|$. The chain rule allows us to conclude that $s'(t)$ exists and is what you say it is, but only under the hypotheses that $g$ is differentiable at $t$ and $f$ is differentiable at $\mathbf{x}(t)$. Unfortunately, the function $f$ is not differentiable when $\mathbf{x}=\mathbf{0}$, so the chain rule does not apply in the case where the particle is at the origin. Are we to conclude that $s'(t)$ must not exist when $\mathbf{x}=\mathbf{0}$?

No! There is no such converse to the chain rule; the derivative of the composite may still exist. In other words, the chain rule supplies sufficient but not necessary conditions for the derivative of a composite to exist. See this question, which involves an example similar to your example with $x=y=t^2$.