Continuous version of the fundamental theorem of invariant theory for the orthogonal group

Yes. It suffices to show that if one has a sequence $\vec v^{(n)} = (v^{(n)}_1,\dots,v^{(n)}_m) \in E^m$ whose Gram matrix $(\langle v^{(n)}_i, v^{(n)}_j \rangle)_{i,j=1,\dots,m}$ converges to a Gram matrix $(\langle v_i, v_j \rangle)_{i,j=1,\dots,m}$ of a tuple $\vec v = (v_1,\dots,v_m) \in E^m$, then after applying linear isometries to each of the $\vec v^{(n)}$, that $\vec v^{(n)}$ converges to $\vec v$.

By permuting indices we may assume that $v_1,\dots,v_k$ are linearly independent, and $v_{k+1},\dots,v_m$ are in the span of $v_1,\dots,v_k$, for some $0 \leq k \leq m$. Applying the Gram-Schmidt process to $v_1,\dots,v_k$ and transforming $v^{(n)}_1,\dots,v^{(n)}_k$ appropriately we may assume that $v_1,\dots,v_k$ are orthonormal. It is not difficult to inductively apply isometries to the $\vec v^{(n)}$ so that $\vec v^{(n)}_i$ converges to $v_i$ for $i=1,\dots,k$. If we write each $v_j, j=k+1,\dots,m$ as a linear combination $v_j = a_{j1} v_1 + \dots + a_{jk} v_k$ of $v_1,\dots,v_k$, then from the Gram matrix convergence we see that $\| v_j^{(n)} - (a_{j1} v^{(n)}_1 + \dots + a_{jk} v^{(n)}_k) \|^2$ converges to zero, hence $v_j^{(n)}$ converges to $v_j$ by the triangle inequality, and the claim follows.

Reflecting on the answer of Terence Tao, I guess it boils down to the fact that an injective proper map between locally compact spaces is a homeomorphism onto its image. Since we are working with ${\bf R}^n$ here, there is a simple characterisation of proper maps that leads to the following statement.

Let $\Phi : {\bf R}^n \longrightarrow {\bf R}^k$ be a continuous map satisfying $$ \|\Phi(x)\| \longrightarrow \infty \quad when \quad {\|x\| \rightarrow \infty}. $$ Let us define the fiber relation on ${\bf R}^n$ by $x \sim x' \iff \Phi(x) = \Phi(x').$

Then $({\bf R}^n/\sim)$ is a locally compact metric space and $\bar{\Phi} : ({\bf R}^n/\sim) \longrightarrow \Phi({\bf R}^n)$ is a homeomorphism.

The condition on the norm is there to ensure that for all compact set $K \subset {\bf R}^k$, $\Phi^{-1}(K)$ is closed and bounded (hence compact). In particular, the fibers $\Phi^{-1}(\{y\})$ are compact and thus we can define a distance on the quotient as follows: $$ d(\bar{x}, \bar{x}') = d(\Phi^{-1}(\{\bar{\Phi}(\bar{x})\}), \Phi^{-1}(\{\bar{\Phi}(\bar{x}')\})). $$

For the problem at hand, we take $\Phi(v_1,...,v_l) = (\langle v_i, v_j \rangle)$ and note that the fibers of $\Phi$ are the orbits of the elements of ${\bf R}^n$ under the action of the orthogonal group. Then, for any invariant $f$, we have $$ f(v_1,...,v_l) = \bar{f}(\bar{\Phi}^{-1}(\bar{\Phi}(\pi(v_1),..., \pi(v_l)))) = \bar{f} \circ \bar{\Phi}^{-1}(\langle v_i, v_j\rangle). $$ The norm condition also ensures that $\Phi({\bf R}^n)$ is closed, so $\bar{f} \circ \bar{\Phi}^{-1}$ can be extended to all ${\bf R}^k$ if needed.