Given that $a\cos\theta+b\sin\theta+A\cos 2\theta+B\sin 2\theta \leq 1$ for all $\theta$, prove that $a^2+b^2\leq 2$ and $A^2+B^2 \leq 1$

We have $$0 \leq F(\theta) + F(\theta + \pi) = 2 - 2(A \cos 2\theta + B \sin 2\theta).$$

Let $u$ be the vector $(A,B)$. Pick $\theta$ so that $v = (\cos 2\theta, \sin 2\theta)$ is a unit vector in the same direction as $u$. Then

$$\sqrt{A^2 + B^2} = |u| = u \cdot v = A \cos 2\theta + B \sin 2\theta \leq 1.$$

For the second part, write $$0 \leq F(\theta) + F(\theta + \pi/2) = 2 - (b + a)\cos \theta - (b- a)\sin \theta.$$

Now pick $\theta$ so that $v = (\cos \theta,\sin \theta)$ is a unit vector in the same direction as $u = (b + a,b - a)$. Then $$\sqrt{2}\sqrt{a^2 + b^2} = \sqrt{(b + a)^2 + (b - a)^2} = |u| = u \cdot v \leq 2.$$