certain proofs of the irrationality of $\sqrt{2}$

You might like the geometric proof posted here: http://blog.plover.com/math/sqrt-2-new.html

I think it can be converted into the form you presented.

Edit: 20 proofs of the irrationality of $\sqrt{2}$ can be found here, but none with the exact form you mentioned.

You can use that if $a^2=2b^2$ then $(2b-a)^2=2(a-b)^2$ to show that if $\frac ab $ is a square root of 2 then so is $\frac {2b-a}{a-b}$. So there is not a fraction with smallest denominator.

Also if $\frac ab$ is an approximation to $\sqrt 2$, then $\frac {a+2b}{a+b}$ is in general a better one, which may be what you are remembering.

If $$2-\frac{a^2}{b^2}=\epsilon$$ then

$$\frac{(a+2b)^2}{(a+b)^2}-2 = \frac{2b^2-a^2}{(a+b)^2}=\epsilon \left(\frac b{a+b}\right)^2$$