Find all rational points on $x^2 + y^2 = 17$

While you already have answers in one direction, I figure it's worth showing a different angle of approach to the problem: since 'by inspection' we have a particular solution $x=4, y=1$ to the problem, and using the sum-of-two-squares product identity, we can construct a map between solutions of $x^2+y^2=1$ and solutions of $x^2+y^2=17$.

In more detail: the equation $x^2+y^2=17$ is most easily thought about conceptually as $|z|^2=17$, with $z=x+iy$. Seen through this lens, it's clear that given our specific solution $z_0=4+i$, then we have $\left|\frac z{z_0}\right|^2=1$ whenever $|z|^2=17$; contrariwise, if $|w|^2=1$, then $|z_0w|^2=17$. Since the complex numbers form a field, these maps are bijections: if $z_0z=z_0w$ then $z=w$, and similarly if $\frac{z}{z_0}=\frac{w}{z_0}$. What's more, since $z_0$ is a 'rational' complex number, then the bijections take rationals to rationals.

But solving the equation $x^2+y^2=1$ in rationals the same as just solving the Pythagorean equation $a^2+b^2=c^2$, and its family of primitive solutions is well-established; you can use these to write explicit solutions to your equation.

(I'll add post-facto that this is really just an algebraic way of looking at the approach that you took; multiplication by $z_0$ is the same as rotating and scaling the complex plane, carrying the circle $|z|^2=1$ to the circle $|z|^2=17$, and carrying the point $(1,0)$ — i.e., $z=1$ — to the point $(4,1)$, which is just an $xy$ swap away from being the point that you were using for your projection, and one way of finding the canonical solutions to the Pythagorean equation is by projecting from the circle in much the same way as was done in the original post here. So the two approaches are really the same thing every step of the way, but I find the algebraic approach a little easier to understand.)

Note that if we can get all the solutions inside the first quadrant, we are done.

Starting with the OP's two equations,

$\qquad y=m(x-1)+4$

and

$\qquad m^2(x-1)^2 + 8m(x-1) + x^2 - 1 = 0$

after factoring out $x - 1$ and employing some algebra, we can get all rationals on the 1st quadrant arc $(0,\sqrt 17) \text{ to } (\sqrt 17,0)$ in a bijective manner by taking $m \in \mathbb Q$ and belonging to the open interval

$\tag 1 m \in (-\frac{4}{\sqrt {17} -1},\; 4 - \sqrt {17})$

and mapping it

$\tag 2 x\text{-coordinate}(m) = 1 -\frac{(8m+2)}{(m^2+1)}$

Checking the validity of (2): For $m = -1/2$, we get $x$, and then solving for $y$, we find the solution $(\frac{13}{5}, \frac{16}{5})$.

Afterthought: The mapping (2) actually is all that is required to answer the question. When I started working on this I restricted attention to the first quadrant, but that only made things more complicated. It is interesting to see a bijective corresponce to the quadrant, but ...

Now, when using (2) for all real numbers $m$, you have to keep in mind that only one value of $y$ found by taking the square root strictly works, i.e. represents the intersection of the line on the circle not equal to $(1,4)$. So you can describe a bijection again, but the algorithm is a bit wordy:

"throw out the $y$ of the two found that does not work"

As a final geometric check of our algebra, note that as $m$ goes to $\pm\infty$, the $x\text{-coordinate}$ found with (2) goes to $1$, as expected (picture the lines drawn through $(1,4)$). In the same vein, just plug $m = 0$ into (2).

---

Here is the 'factoring' algebra that the OP was looking for:

$\quad m^2(x-1)^2 + 8m(x-1) + x^2 - 1 = 0 \text{ iff }$
$\quad m^2(x-1)^2 + 8m(x-1) + (x - 1)^2 + 2(x-1) = 0 \text{ iff }$
$\quad m^2(x-1) + 8m + (x - 1) + 2 = 0 \; \text{ or } x = 1 \text{ iff }$
$\quad (m^2 + 1)(x-1) + 8m + 2 = 0 \; \text{ or } x = 1$

So ignoring the $x = 1$ solution, we have

$\quad x = 1 - \frac{8m + 2}{m^2 + 1}$

which can also be expressed as $\frac{m^2 -8m - 2}{m^2 + 1}$, as demonstrated in Eric Towers' answer.