How can I find real, symmetric matrices $A,B$ such that $Ax =\lambda B x $ has no real eigenvalues and eigenvectors?
Projection matrices seem to be the natural place to start with this condition. They are symmetric, and it is easy to consider matrices where the condition of no real eigenvalues are satisfied.
Let $S$ be any subspace of $\mathbb{R}^n$. Let $A$ be a projection matrix with range $S$ and let $B$ be a projection matrix for $S^\perp$. Then the ranges of $A$ and $B$ have trivial intersection, so there the pair will not have any real eigenvalues. The only possibility to check is $\lambda=0$. If $Ax=0$, then $Bx \neq 0$ if $x \neq 0$ since $x$ would be by necessity in $S^\perp$.
For example, let $n=3$. Let $S$ be spanned by $\langle 1,0,0 \rangle$ and $\langle 1,0,0 \rangle$. Then $S^\perp$ is spanned by $\langle 0,0,1 \rangle$. The corresponding projection matrices are $$ A =\begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}$$ and $$ B =\begin{pmatrix} 0 & 0 & 0\\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}$$ It is easy to see that the only way $Ax=\lambda Bx$ is if $x$ is the zero vector. This works because $S$ and $S^\perp$ have trivial intersections, and so no nonzero vector gets mapped to the same direction by both $A$ and $B$.
The strategy? Maybe a strategy: One of the simplest scenarios where such symmetric matrices $A$ and $B$ may fail to form such a pair with eigenvalues, observing that it doesn’t fail for invertible multiples of the identity, is for the matrices $A$ and $B$ to be square diagonal of dimension $n = 2$. So try to look there.
What would be the simplest examples of two such diagonal square matrices of dimension $2$?
Update. Actually, since the definition is not symmetric in $A$ and $B$, it may fail in dimension $1$ for $A = 1$ and $B = 0$ – as it does in every dimension.