Find all continuous functions over reals such that $f(x)+f(y) = f(x+y)-xy-1$ for all $x,y \in \mathbb{R}$

Let $g(x) = f(x) + 1-x^2/2$. Then $g$ is continuous and

$$g(x+y) = g(x) + g(y)$$

The only functions with those properties (continuous and additive) are those of the form $g(x) = ax$, for $a \in \mathbb{R}$. (See Continuous and additive implies linear )

Hence $f(x) = -1+ax + \frac{x^2}{2}$ for some constant $a$.

EDIT: Why that $g$? First, $xy = \frac{1}{2} (x+y)^2 - \frac{1}{2}x^2 - \frac{1}{2} y^2$, so the relation can be written as

$$f(x) - \frac{1}{2}x^2 + f(y) - \frac{1}{2} y^2 = f(x+y) - \frac{1}{2}(x+y)^2 -1. $$

If $h(x) = f(x) -\frac{1}{2} x^2$, then

$$h(x)+h(y) = h(x+y) -1.$$

There are two $h$ on the left and only one on the right, and the balance is $-1$. So if we define $g(x) = h(x) +1$, then

$$g(x)-1 + g(y) -1 = g(x+y)-1 -1.$$


Since you found $f(0)+f(0)=f(0)-0-1$so that $f(0)=-1$, if you hold $y$ constant and take the derivative with respect to $x$, you get $$f^{\prime}(x)=f^{\prime}(x+y)-y$$ Then rearrange to $$f^{\prime\prime}(x)=\lim_{y\rightarrow0}\frac{f^{\prime}(x+y)-f^{\prime}(x)}{y}=\lim_{y\rightarrow0}1=1$$ So $f(x)=\frac12x^2+C_1x+C_2$. Since $f(0)=C_2=-1$, we are down to $f(x)=\frac12x^2+C_1x-1$, and since this works in the original equation, we are done.

EDIT: OK, so here's a solution that doesn't require twice differentiable functions. Rewrite the recurrence relation as $$\frac{f(x+y)-f(x)}y=\frac{f(y)-f(0)}y+x$$ Given that we know $f(0)=-1$. Then $$\begin{align}\lim_{y\rightarrow0}\frac{f(x+y)-f(x)}y&=\lim_{y\rightarrow0}\frac{f(y)-f(0)}y+\lim_{y\rightarrow0}x\\ &=f^{\prime}(x)=f^{\prime}(0)+x\end{align}$$ On integration we get $$f(x)=\frac12x^2+f^{\prime}(0)x+C_3$$ And recall that we knew $$-1=f(0)=C_3$$ So we are back to verifying that $$f(x)=\frac12x^2+f^{\prime}(0)x-1$$ works for all values of $f^{\prime}(0)$.