Find all prime solutions of equation $5x^2-7x+1=y^2.$

Since $(0,1)$ is a solution to $5x^2-7x+1=y^2$, it can be used to parametrize all rational solutions to $5x^2-7x+1=y^2$. That will give us:

$$x:=\frac{-2ab - 7b^2}{a^2 - 5b^2}$$ and $$y:=\frac{a}{b}x+1$$

where $a,b\in \mathbb{Z}$, $b\neq 0$.

Since $x$ is prime, it follows that either $b=1$ or $x=b$.

• case $b=1$

  This gives us $x:=(-2a - 7)/(a^2 - 5)$. Since $x\in \mathbb{Z}$, we get $a=\pm 2$, and then $x=11$ or $x=3$. Those will give $y=23$ or $y=5$, repectively.

• case $x=b$.

We have that $\frac{-2ab - 7b^2}{a^2 - 5b^2}=b$ gives

$$(*) \hspace{2cm} a(a + 2) =5b^2 - 7b.$$

Since $y=a+1$ and $x=b$ are prime, and $x=2$ or $y=2$ do not give solutions to $5x^2-7x+1=y^2$, we conclude that $y=a+1$ and $x=b$ are ODD primes. In particular, $a(a + 2)\equiv 0 \mod 4$. Now reducing (*) $\mod 4$, contradicts the fact that $b$ is odd. Thus, the case $x=b$ does not occur.

Therefore $(x,y)= (11,23)$ and $(x,y)=(3,5)$ are the only solutions where both coordinates are prime numbers.

Try working mod $3$ and mod $8$. Assuming $x, y>3$, we have $x,y = \pm 1$ mod $3$. Since $x, y$ are odd we have $x^2, y^2=1$ mod $8$, so $$x^2, y^2 = 1 \text{ mod } 24.$$ Substituting in the equation gives $$x = 24k+11 $$ for some integer $k$. Rearranging the original equation we get $$x(5x-7)=(y-1)(y+1), \tag{1}$$ therefore $x |y-1$ or $x|y+1$, since $x$ is a prime number.

Solving for $x$ gives $$ x = \frac{17}{10} + \frac{1}{10}\sqrt{20y^2+29}>\frac{1}{3}(y+1).$$ Note that $x$ is odd and $y \pm 1$ is even, so $x \ne y\pm1$. This forces $x = \frac{1}{2} (y \pm1)$, or $$y = 2x \pm 1 = 48k + 22 \pm 1 \Rightarrow y = 48k+23.$$ $48k+21$ is rejected being divisible by 3. Plugging these in $(1)$ gives the solution $k=0$ or $$x = 11, \space y = 23.$$

Completing the square and dividing by $5$, we have

$$ (10 x - 7)^2 - 20 y^2 = 29$$

Thus $z = 10 x - 7$ and $w = 2 y$ are solutions of the Pell-type equation

$$ z^2 - 5 w^2 = 29$$

The positive integer solutions of this can be written as

$$\pmatrix{z\cr w\cr} = \pmatrix{9 & 20\cr 4 & 9\cr}^n \pmatrix{7\cr 2\cr} \ \text{or}\ \pmatrix{9 & 20\cr 4 & 9\cr}^n \pmatrix{23\cr 10\cr}$$ for nonnegative integers $n$.

Now you want $w$ to be even and $z \equiv 3 \mod 10$. All the solutions will have $w$ even, and $z$ altermately $\equiv \pm 3 \mod 10$. Thus for $n$ even you get integers for $x,y$ with $$ \pmatrix{z_n\cr w_n\cr} = \pmatrix{9 & 20\cr 4 & 9\cr}^n \pmatrix{23\cr 10\cr}$$ and for $n$ odd, $$ \pmatrix{z_n\cr w_n\cr} = \pmatrix{9 & 20\cr 4 & 9\cr}^n \pmatrix{7\cr 2\cr}$$ You do get primes for $n=0$ ($z_0 = 23, w_0 = 10, x_0 = 3, y_0 = 5$) and $n=1$ ($z_1 = 103, w_1 = 46, x_1 = 11, y_1 = 23$). In general,

• $x_n \equiv 0 \mod 3$ for $n \equiv 0$ or $3 \mod 4$.
• $x_n \equiv 0 \mod 17$ for $n \equiv 2$ or $3 \mod 6$.
• $x_n \equiv 0 \mod 5$ or $y_n \equiv 0 \mod 5$ for $n \equiv 0,3, 4, 5, 6, 9 \mod 10$.
• $x_n \equiv 0 \mod 11$ for $n \equiv 1, 8 \mod 10$.
• $x_n \equiv 0 \mod 13$ for $n \equiv 3, 5, 6, 7, 8, 10 \mod 14$.
• $y_n \equiv 0 \mod 23$ for $n \equiv 1, 2, 5, 6 \mod 8$.

And every integer $n$ is in at least one of these classes. We conclude there are no other prime solutions.