Definition of Hecke operators on orthogonal modular forms

For any reductive group $G$ over a number field $F$, so in particular for the orthogonal group of a quadratic form, there exists a theory of Hecke operators. Let's say you have a congruence arithmetic group $\Gamma\subset G({\mathbb Q})$ and $\alpha\in G({\mathbb Q})$. Let's also say that a modular form is a $\Gamma$-left-invariant function $f$ on $G$, then the function $L_{\alpha}f(x)=f(\alpha^{-1}x)$ is not $\Gamma$-invariant any more, but only invariant under $\Sigma=\alpha\Gamma\alpha^{-1}\cap\Gamma$. Fortunately, this group has finite index in $\Gamma$, so you can form the Hecke-operator $T_\alpha$ defined by $$ T_\alpha f(x)=\sum_{\gamma\in \Gamma/\Sigma} L_\gamma L_\alpha f(x). $$ Then $T_\alpha f$ is $\Gamma$-invariant again, so the Hecke-operator maps modular forms for $\Gamma$ to modular forms for $\Gamma$.

Depending on where you want to go, there are other definitions of Hecke operators, for instance, you can go adelic, then modular fomrs become $G({\mathbb Q})$-invariant functions on $G({\mathbb A})$, where $\mathbb A$ is the adele-ring of $F$. Then Hecke-operators become integral operators, where you integrate the $G(F_v)$-action for a local completion $F_v$ of $F$. As to literature, this usually splits between books that mainly treat the $GL(n)$-case and others that go totally general. But anyway, there's a vast literature, you only need to google ''automorphic form'' and ''Hecke operator''.

To supplement Anton's general answer, here are some specific references for modular forms on orthogonal groups:

• Freitag's notes for O(2,$n$) on his webpage (though he doesn't do loads with Hecke operators--there are also some papers that deal with cases for special $n$, but I haven't read them)

• Pei-Yu Tsai's thesis (though she only works locally)

• Many people have studied Hecke operators for PGSp(4) = SO(5). Besides the well known stuff for the split form (corresponding to usual degree 2 Siegel modular forms), people have also worked on non-split forms. E.g., see Lansky and Pollack for some explicit work with the nonsplit form, or more recent work by Dembele and Cunningham. This fits in Gross's context of algebraic modular forms.