Check whether a point is within a 3D Triangle
A common technique in a computer program is to use barycentric coordinates.
Barycentric coordinates are a lot easier to find than any web resources indicate, so I'm not linking to them.
The easiest way to obtain barycentric coordinates of a point P, given a triangle with vertices described by the vectors A, B, C is likely this method:
$ AreaABC = \frac{ \left| \overline{AB} \times \overline{AC} \right| }{ 2 } $
$ \alpha = \frac{ \left| \overline{PB} \times \overline{PC} \right| }{ 2AreaABC } $
$ \beta = \frac{ \left| \overline{PC} \times \overline{PA} \right| }{ 2AreaABC } $
$ \gamma = 1 - \alpha - \beta $
Here $\alpha$ is the ratio of the area of a subtriangle PBC over the area of the whole triangle ABC, as shown in this image from Peter Shirley's book:
If ALL of the following 4 restrictions are met:
- $ 0 \le \alpha \le 1 $
- $ 0 \le \beta \le 1 $
- $ 0 \le \gamma \le 1 $
- $\alpha + \beta + \gamma = 1$
then the point P is inside the triangle.
Note if you compute $\gamma$ as I did above (using $\gamma = 1 - \alpha - \beta$) then you don't have to check $\alpha + \beta + \gamma = 1$, but you would if you found $\gamma$ using areas (as shown in the diagram).
If ANY of $\alpha$,$\beta$,$\gamma$ are outside those ranges, or if the sum of $ \alpha + \beta + \gamma \ne 1 $ then the point P is not inside the triangle.
Note also when one of $\alpha$,$\beta$,$\gamma$ is 0, and the other 2 coordinates are between 0 and 1, the point P is on an edge of the triangle.
When one of $\alpha$,$\beta$,$\gamma$ is 1 and the other two are 0, then the point P is exactly at a vertex of the triangle.
Of course, these computations assume P is already in the plane of the triangle. If P is not in the plane of the triangle, then you should project it there first, before computing the barycentric coordinates.
Try to solve the system $$ x * (P_1 - P_0) + y * (P_2 - P_0) = P - P_0 $$ If it is solveable, the point $P$ lies on the plane. If in addition $x \geq 0$, $y \geq 0$, and $x + y \leq 1$, then $P$ lies inside the triangle.
Vectors $V_{01}=P_1-P_0$ and $V_{02}=P_2-P_0$ lie in the plane of the triangle, and $V_{01}\times V_{02}$ is normal to this plane. Let $V_{0p}=P-P_0$, and if $V_{0p} \cdot (V_{01}\times V_{02}) = 0$ then $P$ lies in the plane.
Let $P=cP_0+aP_1+bP_2$ where $c=1-a-b$. Then $P=(1-a-b)P_0+aP_1+bP_2$ or $V_{0p}=aV_{01}+bV_{02}$. If $0 \le a,b,c \le 1$ then $P$ lies in the triangle or on its edge.
Vector $V_{01} \times (V_{01}\times V_{02})$ is orthogonal to $V_{01}$ so $V_{02} \cdot(V_{01} \times (V_{01}\times V_{02}))b=V_{0p} \cdot(V_{01} \times (V_{01}\times V_{02}))$ can be solved for b.
Likewise $V_{02} \times (V_{01}\times V_{02})$ is orthogonal to $V_{02}$ so $V_{01} \cdot(V_{02} \times (V_{01}\times V_{02}))a=V_{0p} \cdot(V_{02} \times (V_{01}\times V_{02}))$ can be solved for a.