A plausible inequality

The inequality is true for all $n$.

First of all, we can simplify it a little - from Douglas Zare's comment, we can assume $a_0 = 1$, $a_n = -1$, and try to maximize the LHS by varying the $a_i$s. Since the set of values for the $a_i$s under these conditions is compact, there is a maximum value, and the LHS is a convex function of each $a_i$ so we must have $a_i = \pm 1$ for all $i$. Then, we see that the LHS is clearly maximized when we take $a_i = \frac{|x^2-y^2|}{x^2-y^2}$, so we just have to prove that:

$(\frac{\sum |x_i^2-y_i^2|}{2})^2 \le 1 - (\sum x_iy_i)^2$

whenever $\sum x_i^2 = \sum y_i^2 = 1$. This follows from plugging $\alpha_i = \mbox{max}(x_i,y_i)$, $\beta_i = \mbox{min}(x_i,y_i)$ into the inequality

$(\frac{\sum \alpha_i^2 - \sum \beta_i^2}{2})^2 \le (\frac{\sum \alpha_i^2 + \sum \beta_i^2}{2})^2 - (\sum \alpha_i\beta_i)^2$,

which is just Cauchy-Schwarz in disguise.

[Wrong ounter-example deleted]

This it true for all $n$. The case $n=2$ is handled by Hailong Dao, let's reduce the general case to $n=2$.

First, we may assume that $a_1=1$ and $a_n=0$ as others mentioned. So remove the denominator in LHS. Then forget the condition that $a_i$ are monotone, let's only assume that they are in $[0,1]$ (we can rearrange the indices anyway). Also we may assume that the sum under the square in the LHS is nonnegative - otherwise swap $x$ and $y$.

Then, for every $i$ such that $x_i^2-y_i^2>0$, set $a_i=1$, otherwise $a_i=0$. The LHS grows, the RHS stays. So it suffices to prove the inequality for $a_i\in\{0,1\}$. Rearrange indices so that the first $k$ of $a_i$'s are 1. We arrive to $$ \left(\sum_{i=1}^k x_i^2-\sum_{i=1}^ky_i^2 \right)^2 \le 1 - \left(\sum_{i=1}^n x_iy_i\right)^2 . $$ Define $X_1,X_2,Y_1,Y_2\ge 0$ by $$ X_1^2 = \sum_{i=1}^k x_i^2, \ \ \ X_2^2 = \sum_{i=k+1}^n x_i^2, \ \ \ Y_1^2 = \sum_{i=1}^k y_i^2, \ \ \ Y_2^2 = \sum_{i=k+1}^n y_i^2. $$ Then the LHS equals $(X_1^2-Y_1^2)^2$ and $X_1^2+X_2^2=Y_1^2+Y_2^2=1$. By Cauchy-Schwarz, $$ \left|\sum x_iy_i\right| \le X_1Y_1+X_2Y_2 , $$ so the RHS is greater or equal to $1-(X_1Y_1+X_2Y_2)^2$. Now the inequality follows from $$ (X_1^2-Y_1^2)^2 \le 1-(X_1Y_1+X_2Y_2)^2 $$ which is the same inequality for $n=2$.

Here is a proof for the case $n=2$:

Some general facts: As Douglas pointed out, one can assume $a_1=1, a_n=0$. Also, note that the RHS is:

$$(\sum x_i^2)(\sum y_i^2) -(\sum x_iy_i)^2 = \sum_{i<j}(x_iy_j-x_jy_i)^2$$

Now let $n=2$. The inequality becomes: $$(x_1^2 -y_1^2)^2 \leq (x_1y_2-x_2y_1)^2$$ or

$$|x_1^2 -y_1^2| \leq |x_1y_2-x_2y_1|$$

Let $x_1 = cos(\alpha), x_2=sin(\alpha), y_1=cos(\beta), y_2=sin(\beta)$. Then LHS is $$|\frac 12 (cos(2\alpha) -cos(2\beta))| = |sin(\alpha-\beta)sin(\alpha+\beta)|$$

while the RHS is $|sin(\alpha-\beta)|$