Find a circle orthogonal to two other circles

Algebraic method:

Consider a general circle $x^{2}+y^{2}+2gx+2fy+c=0$. The circle passes through $(8,4)$, which gives $16g+8f+c=-80$.

Also, the condition for $x^{2}+y^{2}+2mx+2ny+c=0$ and $x^{2}+y^{2}+2px+2qy+d=0$ to be orthogonal to each other is $2mp+2nq=c+d$.

Hence, making our circle orthogonal to the given 2 circles, we get

$-8f-c=0$ and $16g-4f+c=-64$

Solving the above three relations in $g,f,c$, we get $g=(-5), f=(-4/3), c=(32/3)$

Hence the equation of the required circle is

$x^{2}+y^{2}-(10)x-(8/3)y+(32/3)=0$

or $(x-5)^{2}+(y-4/3)^{2}=(145/9)$

Consider $\bigcirc A$ with radius $a$ and $\bigcirc B$ of radius $b$, and $\bigcirc R$ of radius $r$ orthogonal to both.

That $\bigcirc R$ is orthogonal to $\bigcirc A$ means that either point of intersection determines a right triangle with legs $r$ and $a$ and hypotenuse $|AR|$. Likewise for $\bigcirc B$. Thus, $$a^2 + r^2 = |AR|^2 \qquad\qquad b^2 + r^2 = |BR|^2$$ If point $P$ is on $\bigcirc R$, then also $$r^2 = |PR|^2$$ Writing $R(h,k)$, the above equations make a non-linear system in unknowns $h$, $k$, $r$. The non-linearness is daunting, but it can be overcome. In fact, the specific nature of your problem makes it easier than it would be in general.

In your problem, we have $A = (0, 4)$, $B = (8,-2)$, $P = (8,4)$, and $a = 4$, $b = 2$. The above equations become $$\begin{align} 16 + r^2 &= ( h- 0 )^2 + ( k - 4 )^2 \\ 4 + r^2 &= ( h - 8 )^2 + ( k + 2 )^2 \\ r^2 &= ( h - 8 )^2 + ( k - 4 )^2 \end{align}$$ where I won't multiply-out the terms. Instead, I'll use the last equation to replace $r^2$ in the first two and get some convenient cancellation (which is what makes your specific problem easier): $$\begin{align} 16 + ( h - 8 )^2 + ( k - 4 )^2 &= ( h- 0 )^2 + ( k - 4 )^2 \quad \to \quad 16 + ( h - 8 )^2 = h^2 \\ 4 + ( h - 8 )^2 + ( k - 4 )^2 &= ( h - 8 )^2 + ( k + 2 )^2 \quad \to \quad \phantom{1}4 + ( k - 4 )^2 = ( k + 2 )^2\\ \end{align}$$ Now, multiplying-out, we see that the quadratic terms $h^2$ and $k^2$ cancel in their respective equations, and we solve to get $$h = 5 \qquad k = \frac{4}{3} \qquad \text{and, thus} \qquad r^2 = (5-8)^2 + \left(\frac{4}{3} - 4 \right)^2 = \frac{145}{9}$$ so that the equation of $\bigcirc R$ is $$\left( x - 5 \right)^2 + \left( y - \frac{4}{3} \right)^2 = \frac{145}{9}$$

This agrees with @Apurv's answer.

(If we hadn't gotten the convenient cancellation, we would still have been able to cancel $h^2$ and $k^2$ from the equation pair. This would leave a linear system in $h$ and $k$ that could be solved.)

---

Note: One can prove @Apurv's orthogonality condition by observing that $$x^2 + y^2 + 2 m x + 2 n y + c = 0 \quad\text{and}\quad x^2 + y^2 + 2 p x + 2 q y + d = 0$$ represent circles about $U(-m,-n)$ and $V(-p,-q)$, with respective radii $u$ and $v$ satisfying $u^2 = m^2+n^2-c$ and $v^2 = p^2+q^2-d$. Thus, $$\begin{align} u^2 + v^2 = |UV|^2 \quad&\implies\quad m^2 + n^2 + p^2 + q^2 - (c + d) = (m-p)^2 + (n-q)^2 \\ &\implies\quad c + d = 2 m p + 2 n q \end{align}$$