How find this function $f(1+xy)=f(x)f(y)+f(x+y)$

This question is very tough, and I give only a partial solution. It shows that for the restriction to $\def\Z{\Bbb Z}\Z[\frac12]=\{\,\frac n{2^k}\mid n\in\Z, k\in\Bbb N\,\}$, which is a dense subset of $\def\R{\Bbb R}\R$, there are precisely two solutions. Beyond this subring one solution clearly extends in a continuous way, but might also have other discontinuous extensions; the other (discontinuous) solution may or may not have extensions (edit the comment by DejanGovc shows that it does not extend). To settle the question of uniqueness of the only obvious solution $f:x\mapsto x-1$ (which was indicated by Jonas Meyer) appears to require new ideas that I have not found.

In view of that solution, it will be convenient to reason in terms of the translation $g:\R\to\R$ of$~f$ given by $g:x\mapsto f(x+1)$. The equation then becomes $$ g(xy)-g(x+y-1)=g(x-1)g(y-1) \qquad\text{for all $x,y\in\R$.} \tag 1 $$ Put $c=g(-2)$, which value is given to be nonzero. Let $z\in\R$; applying $(1)$ with $(x,y)=(-1,z+1)$ gives $g(-z-1)-g(z-1)=cg(z)$ and applying it again but with $(x,y)=(-1,1-z)$ gives $g(z-1)-g(-z-1)=cg(-z)$, and adding these two up and dividing by $c$ gives (replacing the name $z$ by $x$) $$ g(-x)=-g(x) \qquad\text{for all $x\in\R$.} \tag 2 $$ It is then clear that $g(0)=0$, and putting $y=0$ in $(1)$ gives $g(x-1)(g(-1)+1)=0$ for all$~x$, which since $g(-2)\neq0$ implies $g(-1)=-1$. Now from $(2)$ also $g(1)=1$.

The instances of $(1)$ where $x$ or $y$ is in $\{0,1\}$ now provide no further information. For $y=2$ one gets $$ g(2x)=g(x-1)+g(x+1) \qquad\text{for all $x\in\R$,} \tag 3 $$ and for $y=-1$ one gets, using $(2)$, the "recurrence relation" $$ g(x-2)+cg(x-1)+g(x)=0 \qquad\text{for all $x\in\R$.} \tag 4 $$ Substituting $x+1$ for $x$ in$~(4)$ and subtracting this from $(3)$ gives $$ g(2x)=-cg(x) \qquad\text{for all $x\in\R$.} \tag 5 $$

One has $g(2)=-c$ from$~(2)$, then $g(3)=c^2-1$ from$~(4)$, and $g(4)=c^2$ from $(3)$ (or $(5)$) with $x=2$. But $(4)$ for $x=4$ gives $0=-c+c^3-c+c^2=c(c-1)(c+2)$, so since $c\neq0$ one must have $c=1$ or $c=-2$.

In both cases we can deduce unique solutions restricted to the ring $\def\Z{\Bbb Z}\Z[\frac12]$ by first solving $(4)$ to find the values $g(n)$ for $n\in\Z$, and then repeatedly applying $(5)$ to go halves. For $c=-2$ we get for $g$ the identity function on $\Z[\frac12]$. For $c=1$, combining two instances of $(4)$ gives $g(x)=g(x-3)$ so the values on $\Z$ are given by $g(n)=g(n\bmod 3)\in\{0,1,-1\}$, and we find for $n\in\Z$ and $k\in\Bbb N$: $$ g(\frac n{2^k})=(-1)^k r(n) \qquad\text{where } r(n)=(r+1)\bmod 3-1 = \begin{cases}1&\text{if $n\equiv1\pmod3$},\\ 0&\text{if $n\equiv0\pmod3$},\\ -1&\text{if $n\equiv-1\pmod3$.}\\ \end{cases} $$ Since $r(-n)=-r(n)$ and $2^k\equiv(-1)^k\pmod3$ it is easy to see that this definition is consistent across various way of expressing the same fraction, and to see that it indeed gives a solution if$~(1)$ restricted to $\Z[\frac12]$, one may choose $k,l$ even (to avoid multiple cases), for which the desired equation $$ g(\frac n{2^k}\frac m{2^l}) - g(\frac n{2^k}+\frac m{2^l}-1) = g(\frac n{2^k}-1)g(\frac m{2^l}) $$ reduces to $$ r(nm)-r(2^ln+2^km-2^{k+l})=r(n-2^k)r(m-2^l) $$ and since $r$ has period $3$ this becomes $$ r(nm)-r(n+m-1) = r(n-1)r(m-1) $$ which can be checked to hold for all combinations of classes in $\Z/3\Z$.

If one would like to prove that the identity is the only solution on$~\R$ for $g$, one might want to establish that $g$ is increasing, or (which would imply this) that $g(x)>0$ for $x>0$, but I don't see how one can derive such things from the given functional equation.

I pick up where Marc van Leeuwen has stopped. If we put $g(x)=f(x+1)$, his answer has shown the following fundamental properties :

$$ \begin{array}{ll} g(x)=x \ \text{when } x\in {\mathbb Z}, & (1) \\ g(2x)=2g(x) \ \text{when } x\in {\mathbb R} & (2) \\ g(-x)=-g(x) \ \text{when } x\in {\mathbb R} & (3) \\ g(x-1)+g(x+1)=2g(x) \ \text{when } x,y\in {\mathbb R} & (4) \\ g(xy)-g(x+y-1)=g(x-1)g(y-1) \ \text{when } x,y\in {\mathbb R} & (5) \\ \end{array} $$

I will deduce from those properties that $$ g(ux+v)=ug(x)+v(g(x+1)-g(x)) (\ \text{when } x\in {\mathbb R},u,v\in{\mathbb Q},u\neq 0) \tag{6} $$ Clearly, it suffices to show (6) when both $u$ and $v$ are integers. Thanks to (3), we may further assume that $u > 0$. We do this by induction on $u$.

Initial step : (i.e. $u=1$) : this follows easily from (4) alone.

Recurrence step : Suppose that the property is true for all $u' \leq u$. Using (5) with $y=u+1$, we deduce $g((u+1)x)=g(x+u)+g(x-1)g(u)$. Now $g(u)=u$ by (1), and $g(x+u)=g(x)+u(g(x+1)-g(x))$ and $g(x-1)=2g(x)-g(x+1)$ by (4). So $$ g((u+1)x)=g(x)+u(g(x+1)-g(x))+(2g(x)-g(x+1))u=(u+1)g(x) \tag{7} $$ Let us now define the auxiliary function $h(v)=g((u+1)x+v)-(u+1)g(x)-v(g(x+1)-g(x))$ for $v\in{\mathbb Z}$. By (4), $h$ is harmonic over $\mathbb Z$. But by (7) above, $h$ is zero on all multiples of $u+1$. So $h$ is zero everywhere, and we have just finished the proof of (6).

Let $t$ be any number. If we put $a=g(t)$ and $b=g(t+1)-g(t)$, then using (5) twice, first with $x=y=t$, second with $x=2t,y=2t$, we see that

$$ g(t^2)=2a-b+(a-b)^2 \ g(4t^2)=4a-b+(2a-b)^2 \tag{8} $$

But we know that $g(4t^2)=4g(t^2)$ by (2). So we must have $4(2a-b+(a-b)^2)=4a-b+(2a-b)^2$, i.e. $(b-1)(4a-3b)=0$. Since there is more than one possible value for $a$ ($a$ can be any integer), this forces $b=1$. We have just shown : $$ g(x+1)-g(x)=1 (\ \text{when } x\in {\mathbb R}) \tag{9} $$

So that (6) becomes $$ g(ux+v)=ug(x)+v (\ \text{when } x\in {\mathbb R},u,v\in{\mathbb Q}) \tag{6'} $$

Combining (6') with (5) (used with $x=y$), we have $$ g(x^2)=g(x)^2, (\ \text{when } x\in {\mathbb R}) \tag{10} $$

So $g$ is positive on ${\mathbb R}_{+}$. This positivity, together with (6'), implies that $g$ is nondecreasing. Since $g$ is the identity on ${\mathbb Q}$, it must be the identity everywhere.

I believe I've managed to complete Marc van Leeuwen's solution (i.e. uniqueness of the solution in the case $c=-2$.) The following argument should prove that $g(x)>0$ for $x>0$. (Which is sufficent, according to Marc van Leeuwen; Ewan Delanoy has also provided a nice argument for this fact in his answer.)

I would like to emphasize that Marc van Leeuwen deserves the credit for the solution. This is really just an extended comment to his answer. The numbers of the form $(n)$ (where $n\in\mathbb N$) refer to equations in his answer.

Here's the argument: letting $y=-x$ in $(1)$ we obtain $$g(-x^2) - g(-1) = g(x-1)g(-x-1).$$ Using $(2)$ and $g(-1)=-1$, this becomes $$g(x^2) - 1 = g(x-1)g(x+1). \tag{I}$$ Letting $y=1-x$ in $(1)$, we obtain $$g(x(1-x)) - g(0) = g(x-1)g(-x).$$ Using $(2)$ and $g(0)=0$, we obtain $$g(x(x-1))=g(x)g(x-1).\tag{II}$$ Multiplying this last equation by $4$ and using $(5)$, we obtain that $$g(2x(2x-2)) = g(2x)g(2x-2).$$ Substituting $x=\frac{z+1}2$ into this, we get $$g((z+1)(z-1))=g(z+1)g(z-1).$$ Comparing this with $(\mathrm I)$ yields $$g(x^2)-1=g(x^2-1).$$ This means that for $z\geq 0$, $$g(z)-1=g(z-1)\tag{III}$$ holds. Using $(2)$ again we can see that $(\mathrm{III})$ holds for all $z\in\mathbb R$, since for $z\leq1$ we have $$g(z-1) = -g(1-z)=-(g(1-z)-1+1)=-(g(-z)+1)=g(z)-1.$$ Using $(2)$ on this in a different way, we get $$g(z+1)=-g(-z-1)=-(g(-z)-1)=g(z)+1\tag{IV}$$ Applying $(\mathrm{III})$ and $(\mathrm{IV})$ to $(\mathrm I)$, we see that $$g(x^2)-1=(g(x)-1)(g(x)+1)=g(x)^2-1.$$ Therefore $g(x^2) = g(x)^2$ holds for all $x\in\mathbb R$, which implies that $g(x)>0$ holds for all $x>0$.