Check if a point lies in a circle defined by three other points.

I get the answer finally.

I see the question about how to determine the equation of a circle defined by three points. Zaz's answer mentioned this page, which can solve my problem indirectly.

Notice that: $D = -p_x M_{41} + p_y M_{42} - (p_x^2 +p_y^2) M_{43} + M_{44}$ and if we regard $p$ as a moving point, then $D=0$ is actually an equation of a circle:

$(p_x+\frac{1}{2}\frac{M_{41}}{M_{43}})^2+(p_y-\frac{1}{2}\frac{M_{42}}{M_{43}})^2=(\frac{1}{2}\frac{M_{41}}{M_{43}})^2+(\frac{1}{2}\frac{M_{42}}{M_{43}})^2+\frac{M_{44}}{M_{43}}$

i.e. the circle locate at $(x_0,y_0)=(\frac{1}{2}\frac{M_{41}}{M_{43}},-\frac{1}{2}\frac{M_{42}}{M_{43}})$ with radius $r_0^2=x_0^2+y_0^2+\frac{M_{44}}{M_{43}}$.

When $D>0$ the circle equation become $(x-x_0)^2+(y-y_0)^2<r_0^2$

So it's clear that if $(p_x,p_y)$ fall inside that circle, then $D>0$ and vice versa.

Note: According to Cramer's rule, $(-\frac{M_{41}}{M_{43}},\frac{M_{42}}{M_{43}},\frac{M_{44}}{M_{43}})$ is exactly the solution of the equation $ \begin{pmatrix} a_x & a_y & 1 \\ b_x & b_y & 1 \\ c_x & c_y & 1 \end{pmatrix} \boldsymbol{x}=\begin{pmatrix}a_x^2+a_y^2 \\ b_x^2+b_y^2 \\c_x^2+c_y^2\end{pmatrix} $

And this 3 parameters perfectly define a circle pass through $a,b,c$.One can easily verify when $p$ is $a,b$ or $c$, $D=0$ by the equation above.