Find the limit or prove that it does not exist $\lim_{(x, \space y) \to (0, \space 0)} f(x, y) $ where $f(x, y) = \frac{x^5-y^5}{x^4-2x^2y^2+y^4}$

The limit doesn't exist. If $n\in\mathbb N$, then$$f\left(\sqrt{\frac1{n^2}+\frac1{n^4}},\frac1n\right)=\left(\left(\frac1{n^2}+\frac1{n^4}\right)^{5/2}-\frac1{n^5}\right)n^8$$and$$\lim_{n\to\infty}\left(\left(\frac1{n^2}+\frac1{n^4}\right)^{5/2}-\frac1{n^5}\right)n^8=\infty.$$

I think your substitution gives you a hint. The angles $\tan\phi =\pm 1$ are the problematic ones, i.e., the limit along the lines $\pm x=y$. You can check that along these lines there's a divergence.