Is there any continuous function that satisfies the following conditions?
What you want is a function that is highly peaked around $x=2a$. And you need to enforce the conditions that $f(0)=f(1)=0$; for that you can put in a factor of $x(1-x)$.
So try
$$ f(x) = x(1-x)e^{-20(x-\frac12)^2} $$ with $a=\frac14$ and $b=\frac34$.
There, $$f(a)^2 - f(2a)f(b) + f(b)^2 \approx -0.00766 <0$$
All you have to do is choose numbers $a,b$ so that $0\lt a\lt 2a\lt b\lt1$ (or $0\lt a\lt b\lt2a\lt1$), then find numbers $y_1,y_2,y_3$ satisfying $y_1^2-y_2y_3+y_3^2\lt0,$ and then construct a continuous function (e.g. a fourth degree polynomial) $f(x)$ with $f(0)=f(1)=0,\ f(a)=y_1,\ f(2a)=y_2,\ f(b)=y_3.$ For example:
$$f(x)=x(x-1)(4x-1)(8x-7),\ a=\frac14,\ b=\frac34$$