How can I find the integers $x,y$?

You can solve this by talking the first equation and solving for y. Then substituting y in the second equation, as so:

$$ax+by=c$$

$$ax-c=-by$$

$$(ax-c)/(-b)=y$$

Now substitute into second equation:

$$ax^{2}+\frac{b\left(ax-c\right)^{2}}{b^{2}}=d$$

Expand:

$$ax^{2}+\frac{a^{2}x^{2}-2acx+c^{2}}{b}=d$$

Simplify:

$$\frac{bax^{2}}{b}+\frac{a^{2}x^{2}-2acx+c^{2}}{b}=d$$

$$\frac{bax^{2}+a^{2}x^{2}-2acx+c^{2}}{b}=d$$

Subtract d from both sides:

$$\frac{bax^{2}+a^{2}x^{2}-2acx+c^{2}-bd}{b}=0$$

Simplify more...

$$\frac{\left(a^{2}+ba\right)}{b}x^{2}-\frac{2ac}{b}x+\frac{\left(c^{2}-bd\right)}{b}=0$$

This is just a quadratic. You can solve it with the quadratic equation:

$$x=\frac{-g+\sqrt{g^{2}-4fh}}{2f}$$

And

$$x=\frac{-g-\sqrt{g^{2}-4fh}}{2f}$$

Where $$f=\frac{\left(a^{2}+ba\right)}{b}$$

$$g=-\frac{2ac}{b}$$

$$h=\frac{\left(c^{2}-bd\right)}{b}$$

Try it out: https://www.desmos.com/calculator/dw9xe3ppxe