Find all functions $f:\mathbb R \rightarrow \mathbb R$, such that: $(x^2 − y^2)\cdot f(xy) = x\cdot f(x^2y) − y\cdot f(xy^2)$
Since my first answer contained a mistake so I am posting this revised version which only gives a partial result to the question.
Claim: Assume $f\colon\mathbb R\to\mathbb R$ satisfies $(x^2-y^2)f(xy) = xf(x^2y)-yf(xy^2)$ for all real $x,y$, and is continuous in $1$. Then $f(x) = f(1)x$ for all $x$; i.e. $f$ is linear.
Proof: Plugging in $y=-x$ yields $$ 0 = xf(-x^3) +xf(x^3) \implies f(x) = -f(-x) \qquad\text{for } x\neq 0 $$ Moreover since $f(0)=0$, as seen by plugging in $(x,y)=(-1,0)$, we conclude that $f$ is an odd function. Now if we plug in $y=1$ instead we find that for all $x\neq 0$: $$ (x^2 -1)f(x) = xf(x^2) -f(x) \implies x^2 f(x) = xf(x^2) \implies \boxed{\frac{f(x)}{x} = \frac{f(x^2)}{x^2}}\qquad (I) $$ For $x>0$ we can subsitute $x\to\sqrt{x}$ to find $\frac{f(x)}{x} = \frac{f(\sqrt{x})}{\sqrt{x}} = \frac{f(\sqrt{\sqrt{x}})}{\sqrt{\sqrt{x}}} = \ldots = \frac{f(\sqrt[2^k]{x})}{\sqrt[2^k]{x}}$ for all $k\ge0$ by repeatedly applying the equation to itself. Then we can use the well known fact that $\lim\limits_{n\to\infty}\sqrt[n]{x} =1$ for all $x>0$, which together with the continuity of $f$ in $1$ implies that for $x>0$: $$ f(1) =\lim_{k\to\infty}\frac{f(\sqrt[2^k]{x})}{\sqrt[2^k]{x}} = \lim_{k\to\infty} \frac{f(x)}{x} = \frac{f(x)}{x} $$ Thus by symmetry we have $f(x)=f(1)x$ for all $x$.
Remark: I wasn't able to deduce continuity of $f$ in $1$, but one possible approach might be to set $y = \frac{1}{x}$ and use $xf(x) = f(x^2)$, a variation of $(I)$, to deduce:
\begin{gather} (x^2 - \frac{1}{x^2})f(1) = xf(x) - \frac{1}{x}f(\frac{1}{x}) = f(x^2) - f(\frac{1}{x^2}) \\ \implies \boxed{f(1) = \frac{f(x) - f(\frac{1}{x})}{x-\frac{1}{x}}} \end{gather} Which holds not only for positive $x$, but by symmetry for all $x\neq 0$, and then somehow use that $x\approx 1 \iff \frac{1}{x} \approx 1$ to show continuity. Alternatively one might try to find a pathological counter-example that is not continuous, similarly as is the case with Cauchy's functional equation.
The assumption of continuity at $ x = 1 $ is not necessary. As @Hyperplane has shown, we know that:
- $ f ( 0 ) = 0 $.
- For $ x \ne 0 $ we have $ f ( - x ) = - f ( x ) $ and thus it suffices to find $ f ( x ) $ for $ x > 0 $.
- $ f \big( x ^ 2 \big) = x f ( x ) $.
Now let $ g ( x ) = f ( x ) - a x $ where $ a $ is a constant real number. Using the original functional equation, it's easy to see that $ g $ satisfies the same equation, i.e. $$ \big( x ^ 2 - y ^ 2 \big) g ( x y ) = x g \big( x ^ 2 y \big) - y g \big( x y ^ 2 \big) \text . \tag 0 \label 0 $$ Thus $ g $ satisfies the three properties above that hold for $ f $. We can also let $ a = f ( 1 ) $ and have the additional property $ g ( 1 ) = 0 $. Now multiplying \eqref{0} by $ x y $ and using the third property, we get $$ \big( x ^ 2 - y ^ 2 \big) g \big( x ^ 2 y ^ 2 \big) = g \big( x ^ 4 y ^ 2 \big) - g \big( x ^ 2 y ^ 4 \big) $$ and hence for $ x , y > 0 $ we have $$ ( x - y ) g ( x y ) = g \big( x ^ 2 y \big) - g \big( x y ^ 2 \big) \text . \tag 1 \label 1 $$ Now letting $ y = \frac 1 x $ in \eqref{1} we get $$ g \bigg( \frac 1 x \bigg) = g ( x ) \text . \tag 2 \label 2 $$ Again, letting $ y = \frac 1 { x ^ 2 } $ in \eqref{1} and using \eqref{2}, we have $$ \bigg( x - \frac 1 { x ^ 2 } \bigg) g ( x ) = - g \big( x ^ 3 \big) \tag 3 \label 3 $$ and substituting $ \frac 1 x $ for $ x $ in \eqref{3} and using \eqref{2} we also have $$ \bigg( \frac 1 x - x ^ 2 \bigg) g ( x ) = - g \big( x ^ 3 \big) \text . \tag 4 \label 4 $$ Subtracting \eqref{3} from \eqref{4} yields $$ \bigg( x + \frac 1 x + 1 \bigg) \bigg( x - \frac 1 x \bigg) g ( x ) = 0 $$ which shows $ g ( x ) = 0 $ for $ 1 \ne x > 0 $. Therefore $ g $ is the constant zero function and thus $ f ( x ) = a x $.