Necessary and sufficient condition for the matrix $A = I - 2 x x^t$ to be orthogonal

You're right that for $A$ to be orthogonal, you need $AA^T = I$. You may have made a mistake in your derivation. You should get $$AA^T = (I - 2xx^T)(I - 2xx^T) = I - 4xx^T + 4xx^Txx^T = I - 4(1 - x^Tx)(xx^T).$$ In the last step, we use the fact that $x^Tx$ is a scalar and so can be pulled out of the middle of $xx^Txx^T$. So now, for $AA^T$ to equal $I$, either of the two parenthesized terms on the right should be zero. What does this tell you about the vector $x$?