Is there a formula for the Frobenius-Schur indicator of a rep of a Lie group?

Let us write the Frobenius-Schur indicator of the representation $V$ with highest weight $\lambda$ as $\mathrm{FS}(\lambda)$. Let $R$ denote the root system, and let $\Pi=\{\alpha_1,\dots,\alpha_l\}$ be a basis. Let $\Pi^\vee=\{\alpha_1^\vee,\dots,\alpha_l^\vee\}$ denote the corresponding basis of the dual root system $R^\vee$. Write $\lambda_i=\langle \lambda,\alpha_i^\vee\rangle$.

Exercise 4.3.12 on page 196 of the book by Onishchik and Vinberg "Lie Groups and Algebraic Groups", Springer-Verlag, 1990, says that $$\mathrm{FS}(\lambda)=(-1)^{\sigma(\lambda)},$$ where $$\sigma(\lambda)=\sum_{i=1}^l r_i\lambda_i\tag{*} $$ and $r_i$ are the coordinates of of the sum of positive coroots $2\rho^\vee$ in the basis $\Pi^\vee$ of $R^\vee$. No proof is given.

The coordinates $r_i$ of $2\rho^\vee$ in the basis $\Pi^\vee$ can be found in the tables of Bourbaki "Groupes et algèbres de Lie, Ch. 4,5,6", formula (VII) in each table. Formulas for $r_i\ \mathrm{mod}\ 2$ are given in the last column of Table 3 on page 298 of the book by Onishchik and Vinberg and seem to agree with the formulas of Bourbaki.

For the reader's convenience I give below formulas for $\sigma(\lambda)$ following Table 3 of Onishchik and Vinberg, with the numbering of roots as in Bourbaki. \begin{align*} A_l,\qquad &\sigma(\lambda)=\lambda_1+\lambda_3+\dots+\lambda_l\quad \text{for } l=2p+1,\text{ otherwise } 0;\\ B_l,\qquad & \sigma(\lambda)=\lambda_l\quad\text{for } l=4q+1,\ l=4q+2,\text{ otherwise } 0;\\ C_l\qquad &\sigma(\lambda)=\lambda_1+\lambda_3+\dots\quad\text{ for any } l;\\ D_l\qquad &\sigma(\lambda)=\lambda_{l-1}+\lambda_l\quad\text{for }l=4q+2,\ l=4q+3,\text{ otherwise } 0;\\ E_7\qquad &\sigma(\lambda)=\lambda_2+\lambda_5+\lambda_7\,. \end{align*} For all the other connected Dynkin diagrams, $\sigma(\lambda)=0$ for all $\lambda$ corresponding to self-dual representations.

Note that since we assume that $V$ is self-dual, the highest weight $\lambda$ is invariant under the involution $\nu$ of the Dynkin diagram that is the only nontrivial symmetry for $A_l$ ($l\ge 2$), $D_{2q+1}$ and $E_6$, and is trivial for all the other connected Dynkin diagrams. Taking this into account, we obtain that for $A_l$ the number $\sigma(\lambda)$ can be odd only for $l=4q+1$, and then we have $\sigma(\lambda)\equiv\lambda_{(l+1)/2}\pmod{2}$, and that for $D_{4q+3}$ the number $\sigma(\lambda)$ is always even. Thus only $A_{4q+1}$, $B_{4q+1}$, $B_{4q+2}$, $C_l$, $D_{4q+2}$, and $E_7$ survive. This is what Exercise 4.3.13 in the book by Onishchik and Vinberg says. This agrees with the table in the answer of Claudio Gorodski.

The Frobenius-Schur indicator (of a self-dual finite dimensional representation) is $$ \chi_\pi(\exp(2\pi i\rho^\vee)) $$ where $\chi_\pi$ is the central character of $\pi$, $\rho^\vee$ is half the sum of the positive coroots, so $\exp(2\pi i\rho^\vee)$ is an element of order $2$ in the center of $G$. Equivalently if $\lambda$ is the highest weight then this equals $$ e^{2\pi i\langle\lambda,\rho^\vee\rangle} $$ This is, of course, equivalent to the other formulas cited, but is conceptually simpler. In particular: if $G$ is adjoint every (irreducible, finite dimensional, self-dual) representation is orthogonal. See Bourbaki, Lie Groups and Lie Algebras, Chapters 7-9, Chapter IX, Section 7.2, Proposition 1. The proof is included, and is the one sketched by Borovoi. For a simpler proof based on the Tits group see this preprint.

Proposition 7 in http://www.ams.org/journals/tran/1985-288-01/S0002-9947-1985-0773051-1/ contains a criterion (including a formula and proof; the proof is on page 131) to decide the type of the complex irreducible representation $\pi_\lambda$ of the compact connected semisimple Lie group $G$, where $\lambda$ is the highest weight, in terms of a maximal subset $\mathcal O=\{\beta_1,\ldots,\beta_\ell\}\subset\Delta^+$ of strongly orthogonal roots. Namely, $\pi_\lambda$ is unitary (not self-contragredient) if and only if $\lambda$ does not belong to the real span of $\mathcal O$. Otherwise it is symplectic (resp. orthogonal), meaning that it leaves a quaternionic (resp. real) structure invariant, if and only if $$ k(\lambda) = \sum_{i=1}^\ell \frac{(\beta_i,\lambda)-(\beta_i,s_0\lambda)}{(\beta_i,\beta_i)} $$ is an odd (resp. even) integer. Here $(,)$ is the inner product defined from the Killing form, and $$s_0 := s_{\beta_1}\cdots s_{\beta_\ell}$$ is the Weyl group element that maps the Weyl chamber to its negative.

The idea of the given proof is essentially to reduce the problem to a subgroup of $G$ isomorphic to the product of $\ell$ copies of $SU(2)$.

The integer $k(\lambda)$ has an interesting meaning. Let $\mathcal U^k(\mathfrak g_\mathbb C)$ be the k-th level in the natural filtration of the universal enveloping algebra of the complexified Lie algebra of $G$. Choose a highest weight vector $v_\lambda$ for $\pi_\lambda$. Then $s_0v_\lambda \in\mathcal U^{k(\lambda)}(\mathfrak g_\mathbb C)v_\lambda$ but $s_0v_\lambda\not\in\mathcal U^{k'}(\mathfrak g_\mathbb C)\lambda$ for any $k'<k(\lambda)$. Here
$$ s_0v_\lambda = \pi(X_{-\beta_1})^{n_1}\cdots\pi(X_{-\beta_\ell})^{n_\ell}v_\lambda$$ where $$n_i=\frac{(\beta_i,\lambda)-(\beta_i,s_0\lambda)}{(\beta_i,\beta_i)} $$ and the $X_{-\beta_i}$ are root vectors.

The table above lists the numbers $n_i$ for the fundamental representations of simple groups.