Find all the functions which satisfy the functional equation $f(y+f(x))=f(x)f(y)+f(f(x))+f(y)-xy$.

For any $x,y\in \mathbb R$ we have $$\begin{align} f(x)f(y)+f(f(x))+f(y)-xf(y) &=f(x)f(f(y))+f(f(x))+f(f(y))-xf(y)\\ &=f(x+f(f(y)))\\ &=f(x+f(y))\\ &=f(x)f(y)+f(f(x))+f(y)-xy\\ \end{align}$$ since $f(f(y))=f(y)$. Thus $xf(y)=xy$. Setting $x=1$ shows that $f$ is the identity.


I would like to give a solution of both the instances of the problem which appeared here in different times, i.e. the version posted before the OP edited his post and the current one. First I want to solve the problem as it is stated by the OP.

Problem I

$$f(y+f(x))=f(x)f(y)+f(f(x))+f(y)-xy.$$ As already noticed, setting $y=0$, one arrives to $$f(f(x))=f(0)f(x)+f(f(x))+f(0)$$ which leads to $f(x)\equiv -1$, which is not a solution of the problem, or $f(0)=0$. Incidentally, even though it is not crucial for the solution, notice that if we assume that there is an $\bar x$ such that $f(\bar x)=0$, setting $x=\bar x$ we infer that $f(y)=f(y)-\bar xy$. In particular, setting $y=1$, we conclude $\bar x=0$.

Next, plug inthe equation $f(y)$ instead of $y$ to get $$f(f(y)+f(x))=f(x)f(f(y))+f(f(x))+f(f(y))-xf(y).$$ Interchange the roles of $x$ and $y$ to obtain similarly $$f(f(x)+f(y))=f(f(x))f(y)+f(f(y))+f(f(x))-yf(x).$$ Compare these two equations to conclude that $$\frac{f(f(x))+x}{f(x)}=\frac{f(f(y))+y}{f(y)}=k,$$ where $k$ is a suitable constant.

Substitute in the original equation to get $$f(y+f(x))=f(x)f(y)-x+kf(x)+f(y)-xy.$$ Denote $a:=f(-1)$ and choose $y=-1$ to obtain $$f(f(x)-1)=(a+k)f(x)+a$$

Edit Thanks to Dejan who pointed out a flaw in my proof

Plug in the original equation $f(y)-1$ instead of $y$ to obtain $$\begin{aligned}f(f(y)+f(x)-1)&=f(x)[(a+k)f(y)+a]-x+kf(x)-x(f(y)-1)\\&=(a+k)(f(x)+f(y))-xf(y).\end{aligned}$$ Exchange $x$ and $y$ to obtain $$f(f(y)+f(x)-1)=(a+k)(f(x)+f(y))-yf(x).$$ Compare these two equations to deduce that $$\frac{f(x)}{x}=\frac{f(y)}{y}=a.$$

Substitute in the original equation to find out that the admissible values for $a$ are just $\pm 1$ i.e. $$f(x)=\pm x.$$

Problem II

Now let's see what happens if the equation is written instead as $$f(x+f(y))=f(x)f(y)+f(f(x))+f(y)-xy.$$ Setting $x=y=0$ one deduces that $$f(0)^2+f(0)=0$$ from which $f(0)\in \{-1, 0\}$.

Assume then $f(0)=0.$ Then plug in the equation $x=0$ to deduce that $f(y)=f(f(y)).$

Let's see that $f$ is injective. Suppose there are $y\neq z$ with $f(y)=f(z)$. Use the equation with $y$ and $z$ and deduce that in this case $x(y-z)=0$ should hold for all $x\in\mathbb R$, which is plainly impossible. Then $f$ is injective. Now, recall that in our hypothesis the equation states as $$f(x+f(y))=f(x)f(y)+f(x)+f(y)-xy.$$ Exchange $x$ and $y$ to obtain $$f(y+f(x))=f(y)f(x)+f(y)+f(x)-yx.$$ It follows from the injectivity of $f$ that $$f(x)-x=f(y)-y=k=0.$$ It is easy to verify that the identity is indeed a solution of our equation.

Assume now $f(0)=-1$. Then, choosing $x=0$, one deduces $$f(f(y))=f(f(-1))=a.$$ Substitute then $x$ with $f(x)$ to derive $$f(f(x)+f(y))=af(y)+f(a)+f(y)-f(x)y.$$ Similarly, also $$f(f(x)+f(y))=af(x)+f(a)+f(x)-f(y)x$$ holds. Therefore we finally find that $$\frac{f(x)}{(a+1+x)}=\frac{f(y)}{(a+1+y)}=c.$$ It follows that there are constants $c,d\in\mathbb R$ such that $$f(x)=cx+d.$$ From the fact that $f(f(x))=a$, we deduce $c=0$, and from $f(0)=-1$, it follows $f\equiv -1$ which is not a solution of the problem.


Partial progress: Suppose $f:\mathbb R\to\mathbb R$ satisfies the functional equation $$f(y+f(x))=f(x)f(y)+f(f(x))+f(y)-xy.\tag{$*$}$$

Proposition 1. Suppose $f(x_0)=0$. Then $x_0=0$.

Proof. Plugging $x=x_0$ and an arbitrary $y$ into the equation and using the fact that $f(0)=0$ yields $0=-x_0y$. Picking $y=-1$ yields the result. $\square$

Proposition 2. The function $f$ is injective.

Proof. Plugging an arbitrary $x$ and $y=f(x)$ into the equation tells us that $$f(2f(x))=f(x)f(f(x))+2f(f(x))-xf(x)\tag{1}$$ holds for every $x$. Now, suppose $f(x_1)=f(x_2)$ for some $x_1,x_2\in\mathbb R$. Then $$f(2f(x_1))=f(2f(x_2))$$ must hold. By $(1)$ this means that $$f(x_1)f(f(x_1))+2f(f(x_1))-x_1f(x_1)=f(x_2)f(f(x_2))+2f(f(x_2))-x_2f(x_2)$$ holds. Using the fact that $f(x_1)=f(x_2)$ again, this implies that $$-x_1f(x_1)=-x_2f(x_1).$$ Now, if $f(x_1)$ is non-zero, we can divide by it and we are done. Otherwise, $f(x_1)=f(x_2)=0$ and by Proposition 1, $x_1=x_2=0$. This completes the proof. $\square$

ADDED:

Proposition 3. Suppose there exists an $x_1\in\mathbb R$ such that $f(x_1)=-1$. Then $f=\pm\operatorname{id}_{\mathbb R}$.

Proof. If such an $x_1$ exists, plug it into the equation in place of $x$. Plug in an arbitrary $y$. This gives us $$f(y-1)=f(-1)-x_1y.$$ Since $y$ was arbitrary, this implies that $$f(y)=f(-1)-x_1(y+1).$$ This means that $f$ is linear. It is easy to verify that the only linear solutions of the equation are given by $f(x)=x$ and $f(x)=-x$. $\square$

So, to complete the proof that these two solutions are the only ones, we would only need to show that $f(x_1)=-1$ for some $x_1\in\mathbb R$. (If this is indeed true.)