Finite nonabelian groups with few positive real character values?
Here are some classes of such groups I can think of:
- extraspecial $p$-groups (and probably some other $p$-groups)
- metacyclic groups, where $\lvert G: G'\rvert$ has odd order
- the groups $AGL(1,q)$ (this subsumes Tim Dokchitser's examples and the $A_4$ example). More generally, suppose a group $H$ with few positive real values acts sharply transitive on the nonzero elements of a finite vector space $V$. Then the semidirect product $HV$ has also only characters with at most one strictly positive value. An example is $Q_8$ acting on $C_3\times C_3$.
Here is an argument that rules out some groups: If a group $G$ acts $2$-transitively on some set, then the permutation character $\pi$ has the form $\pi= 1+\chi$ with $\chi \in \operatorname{Irr}(G)$. By assumption, if $g$ acts nontrivially, then $\chi(g)\leq 0$. This implies that every element not in the kernel $K$ of the action has at most one fixpoint. But then $G/K$ is a Frobenius group, the elements with no fixpoints form the Frobenius kernel.
While I'm not an expert for finite simple groups, I suppose this argument rules out many (all?) finite simple groups, namely all that admit a $2$-transitive permutation representation.
Since the property in the question is inherited by factor groups, this would also yield that there are no perfect groups with that property.
EDIT: Thanks to Jack for putting that right. Anyway, simple groups are ruled out by ARupinksi's comment.
Here is an argument that resticts the structure of such groups of even order, which expands on ARupinksi's comments above (inspiration came from a nice argument in this paper).
Namely, suppose that $1\neq t$ is a real element in $G$, where $G$ is as in the question. Form the following normal subgroups:
$$ K = \langle t \rangle^G = \bigcap_{\chi(1)=\chi(t)} \ker \chi
\quad \text{and} \quad
L = \bigcap_{\chi(t) \neq 0} \ker \chi < K , $$
where the intersections run over irreducible characters $\chi$ of $G$. Then we have
$$ t^G = K \setminus L,$$
that is, the elements in $K\setminus L$ are all conjugate in $G$ to $t$. In particular, it turns out that $t$ is rational in the sense that it is conjugate to every generator of $\langle t \rangle$, which is equivalent to every character having rational value at $t$.
Why is this true? Well, column orthogonality yields $\DeclareMathOperator{\Irr}{Irr}$
$$ 0 = \sum_{\chi \in \Irr G} \chi(t)\chi(1) = |G:K| + \sum_{\chi(t)<\chi(1)}\chi(t)\chi(1). $$
Now let $y\in K$ and suppose that $t$ and $y$ are not conjugate. Then plugging in $y$ instead of $1$ in the last formula, we see that
$$ \sum_{\chi(t)<\chi(1)} (-\chi(t))\chi(y) = |G:K|
=\sum_{\chi(t)<\chi(1)} (-\chi(t))\chi(1) .$$
Since $-\chi(t)>0$ and $|\chi(y)|\leq\chi(1)$, it follows that $y$ is in the kernel of every irr char $\chi$ with $\chi(t)< 0$, that is, $y\in L$. Thus $K\setminus L = t^G$.
(Added later:) If $t$ is an involution, then it follows that $x^t=x^{-1}$ for all $x\in L$, in particular $L$ must be abelian, and elements of $L$ are real.
The "dual" argument (exchanging the roles of characters and conjugacy classes) shows the following: Suppose $1\neq \chi$ is real valued, and let $V= \operatorname{\mathbf{V}}(\chi)$ be the vanishing-off group of $\chi$, generated by all group elements on which $\chi$ is non-zero. Then
$$ \Irr( G/\ker \chi ) = \{ \chi \} \cup \Irr(G/V). $$
From this it follows easily that $V/\ker\chi$ is a conjugacy class of $G/\ker\chi$ and that the only value of $\chi$ besides $0$ and $\chi(1)$ is $-\chi(1)/(|V/\ker\chi|-1)$. (In particular, any real character is rational.) Groups with such an character have been studied by Zhmud, where more information can be found.
I suppose there is also literature on groups having normal subgroups $L\subset K$ such that $K\setminus L$ is a conjugacy class of $G$. The notion of a Camina pair/group seems to be related (see this paper and papers that refer to it).