Show that the area of a triangle is given by this determinant

Another possibility is to use the formal properties of the determinant and see how they correspond to the properties of the area. This seems lengthy, but it explains also why you have such a relation between determinant and area.

You start with the determinant: $$ \left|\begin{array}{ccc} 1 & 1 & 1 \\ x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \end{array} \right| $$ then you can subtract from the second row $x_1$ times the first row and from the third row you can subtract $y_1$ times the first row to get: $$ \left|\begin{array}{ccc} 1 & 1 & 1 \\ 0 & x_2-x_1 & x_3-x_1 \\ 0 & y_2-y_1 & y_3-y_1 \end{array} \right|. $$ This operation does not change the determinant (which is multilinear). And it does not change the area of the triangle, because it corresponds to a translation, namely the translation of vector $-(x_1,y_1)$ which sends the first vertex of the triangle to the origin.

Now suppose to fix ideas that $0<x_3<x_2$ and consider the triangle (which now has vertices: $p_1= (0,0)$, $p_2=(x_2-x_1,y_2-y_1)$, $p_3=(x_3-x_1, y_3-y_1)$) divided in two by the line $x=x_3-x_1$, and let $p$ be the point with $x$ coordinate equal to $p_3$ which lies on the edge $p_1 p_2$. If we consider $pp_3$ as the base of the two smaller triangles, you see that you can move the vertices $p_2$ and $p_1$ vertically (which means parallel to the common base $pp_3$ without changing the area of the resulting quadrilateral (which is the union of the two smaller triangles). Or, which is the same, you can keep $p_1$ fixed and move the base $pp_3$ vertially. If you want to move the triangle so that both the point $p$ and $p_2$ go to the $x$-axis, you find the mapping $$ \begin{cases} x' = x\\ y' = y - \frac{y_2-y_1}{x_2-x_1}x \end{cases} $$ which correspond to subtract from the third line of the matrix ($y$ coordinates) a multiple of the second line ($x$ coordinate). So, again, this transformation maintains the area of the triangle, and the determinant of the matrix.

You get: $$ \left|\begin{array}{ccc} 1 & 1 & 1 \\ 0 & x_2-x_1 & x_3-x_1 \\ 0 & 0 & (y_3-y_1)-\frac{y_2-y_1}{x_2-x_1}(x_3-x_1) \end{array} \right|. $$ So now you have a triangle (with same area of the original one) whose basis is on the $x$ axis has length $x_2-x_1$ and whose height is given by the $y$ coordinate of the third point which is the entry in the lower-right corner of the matrix. So the area is half the product of the entries on the diagonal, and in fact in this case (triangular matrix) this is indeed the determinant of the matrix.

You could also complete the diagonalization process, by moving the third vertex to the $y$ axis. In that case you would get a right triangle and a diagonal matrix.

Hint: It may be easier to show that $\frac 12\det(M)$ (i.e. without taking absolute value) computes the oriented area of the triangle.

One possible way is to show that $\det M$ is invariant under translation and shearing and gives the correct answer for the special case $A=(0,0)$, $B=(1,0)$, $C=(0,1)$.

Note that translations correspond to adding multiples of the first row of $M$ to the second or third row, shearing corresponds to adding a multiple of the second row to the third or vice versa.

Here is a multi-variable calculus approach. Suppose $S$ is the triangle spanned by vertices $(x_1,y_1),(x_2,y_2),(x_3,y_3)$. We intend to compute the double integral $$\iint_{S} 1 dydx$$ which is the double integral that gives us the area of $S$. Now, we try to come up with a suitable change of variables, so that the legs of this triangle are on the coordinate axes. Suppose $$u=(y_1-y_2)x+(x_2-x_1)y,v=(y_1-y_3)x+(x_3-x_1)y$$.

The Jacobian Determinant of this transformation $$\frac{\partial(u,v)}{\partial(x,y)}=\det \begin{bmatrix}\frac{\partial u}{\partial x} && \frac{\partial u}{\partial y} \\ \frac{\partial v}{\partial x} && \frac{\partial v}{\partial y}\end{bmatrix}=\begin{vmatrix} y_1-y_2 && x_2-x_1 \\y_1-y_3 && x_3-x_1 \\ \end{vmatrix}=(y_1-y_2)(x_3-x_1)-(y_1-y_3)(x_2-x_1)$$ But the need to take the reciprocal $\frac{\partial(x,y)}{\partial(u,v)}=\frac{1}{(y_1-y_2)(x_3-x_1)-(y_1-y_3)(x_2-x_1)}$

$S$ transforms into a right triangle $T$ whose vertices are $(0,0)$, $(y_1-y_2)(x_3-x_1)+(x_2-x_1)(y_3-y_1),0)$, $(0,(y_1-y_3)(x_2-x_1)+(y_2-y_1)(x_3-x_1))$.

You can verify this by plugging in the vectors $(x_2-x_1,y_2-y_1),(x_3-x_1,y_3-y_1)$ into the transformation.

So the double integral becomes under the transformation: $$\iint_{S}1dydx=\iint_{T}\left | \frac{\partial(x,y)}{\partial(u,v)}\right | dudv=\left |\frac{1}{(y_1-y_2)(x_3-x_1)-(y_1-y_3)(x_2-x_1)} \right |\iint_{T}1dudv$$ Now we can use geometry to compute $\iint_{T}1dudv$, which is the area of $T$. Since $T$ is a right triangle, we just need to multiply its base and height and divide by $2$. Looking at the vertices, the base (measured on the $u$ axis) is $$\left|(y_1-y_2)(x_3-x_1)+(x_2-x_1)(y_3-y_1)\right|$$ and the height (measured on the $v$ axis) is $$\left|(y_1-y_3)(x_2-x_1)+(y_2-y_1)(x_3-x_1)\right|$$

Notice that the base cancels out with the the Jacobian determinant. So the area of $S$ is: $$\iint_{S}1dydx=\frac{\left|(y_1-y_3)(x_2-x_1)+(y_2-y_1)(x_3-x_1)\right|}{2}$$ Now expand out $$\frac{\left|(y_1-y_3)(x_2-x_1)+(y_2-y_1)(x_3-x_1)\right|}{2}$$ to get: $$\frac{\left|y_1x_2-y_3x_2+y_3x_1+x_3y_2-x_3y_1-x_1y_2\right|}{2}$$ Rearrange the terms around: $$\frac{\left|-x_2y_3+y_2x_3+x_1y_3-y_1x_3-x_1y_2+x_2y_1\right|}{2}$$ And Use the fact $$x_iy_j-x_jy_i= \begin {vmatrix} x_i && x_j \\ y_i && y_j \\ \end{vmatrix}$$ to see: $$\frac{\left|-x_2y_3+y_3x_3+x_1y_3-y_1x_3-x_1y_2+x_2y_1\right|}{2}=\frac{\left|-\begin {vmatrix} x_2 && x_3 \\ y_2 && y_3 \\ \end{vmatrix}+\begin {vmatrix} x_1 && x_3 \\ y_1 && y_3 \\ \end{vmatrix}-\begin {vmatrix} x_1 && x_2 \\ y_1 && y_2 \\ \end{vmatrix}\right|}{2}$$ Factoring out an $\left |-1\right |$ in the numerator, we see this is just $$\frac{\left|\begin {vmatrix} x_2 && x_3 \\ y_2 && y_3 \\ \end{vmatrix}-\begin {vmatrix} x_1 && x_3 \\ y_1 && y_3 \\ \end{vmatrix}+\begin {vmatrix} x_1 && x_2 \\ y_1 && y_2 \\ \end{vmatrix}\right|}{2}$$. This is just (by the reverse of cofactor expansion):

$$\frac{1}{2}\left |\begin {vmatrix} 1 && 1 && 1 \\ x_1 && x_2 && x_3 \\ y_1 &&y_2&&y_3 \\ \end{vmatrix}\right |,$$ which verifies the determinant formula.