Is this matrix always orthogonal?

First, not every orthogonal matrix is of the stated form, only rotation matrices. There are orthogonal reflection matrices, with determinant $-1$: $$ \left[ \begin{array}{@{}rr@{}} \cos\theta & \sin\theta \\ \sin\theta & -\cos\theta \end{array}\right]. $$

Second, your conjecture is not true: Only unit speed parametrizations have the stated property, because the rows of an orthogonal matrix form an orthonormal set (in the obvious sense), and particularly must be unit vectors.

The third point implies that the parametrization goes over the circle at a constant speed $1$, which is, of course, false in general.

Take for example $v(t)=\langle \sin t^2,\cos t^2\rangle $ for $t\in[0,\sqrt{2\pi})$.

In general, (3) is not true. Since $\left\vert v \right\vert^2 = x(t)^2 + y(t)^2 = 1$ , we can write $$x(t) = \cos \theta(t), \quad y(t) = \sin \theta(t)$$ for some function $\theta(t)$. But computing gives $\left\vert\dot v\right\vert^2 = \dot\theta(t)^2$, so the condition $|\dot v|^2 = 1$ (which just says that $v$ is a unit speed parameterization) forces $\dot\theta(t) = \pm 1$, and this condition yields just the matrices of the given form (the rotations). Note that a composition of any rotation with a reflection is another reflection and hence orthogonal, but these reflections cannot be of the form in the question: Reflections have determinant $-1$ but matrices of the given form have determinant $(\cos \theta)(\cos \theta) - (\sin \theta)(-\sin \theta) = +1$.