Angle Between Two Vectors Facing A Point

Unless $A-B$ and $C-B$ are linearly dependent, you can uniquely express $P-B$ in terms of them: $$ P-B=r\cdot(A-B)+s\cdot(C-B).$$ Now $P$ is in the smaller of the two possible angles if and only if $r$ and $s$ are both positive. It is on the ray $BA$ if and only if $s=0$ and $r\ge0$, and on the ray $BC$ if and only if $r=0$ and $s\ge 0$. In all other cases, it is in the larger angle.

Except if $A,B,C$ are collinear, for any $P$ on the non-reflex side of $\angle ABC$, the unique solution $(t,u)$ to $\vec{BP}=t\vec{BA}+u\vec{BC}$ will have $t,u\ge0$. This leads to the following algorithm:

1. Translate all the points so $B$ is at the origin.

2. In these new coordinates, solve the linear system $$\begin{bmatrix}x_A&x_C\\y_A&y_C\end{bmatrix}\begin{bmatrix}t\\u\end{bmatrix}=\begin{bmatrix}x_P\\y_P\end{bmatrix}$$

3. If both $t$ and $u$ are non-negative, take the non-reflex angle $\alpha$ resulting from the application of the standard formula for angle between two vectors. Otherwise, take the complementary reflex angle $\theta$.

4. If $A,B,C$ are collinear, the linear system in step 2 will be indeterminate, and it is easy to check whether $\angle ABC=0$ (in which case $2\pi$ is returned) or $\pi$ (in which case $\pi$ is returned).

You can sort this out by comparing the signs of three determinants. Let $$d_0 = \det\begin{bmatrix}B_x&B_y&1\\A_x&A_y&1\\C_x&C_y&1\end{bmatrix}.$$ This tells you the direction in which the vertices of $\triangle{BAC}$ are traversed: positive means counterclockwise, negative clockwise, and the points are colinear if it’s zero. Note that the area of this triangle is equal to $\frac12\lvert d_0\rvert = \frac12\lVert A-B\rVert\,\lVert C-B\rVert\,\lvert\sin\phi\rvert$, with $\lvert\phi\rvert\in[0,\pi)$, so when the points aren’t colinear this is another way to find the angle between them. Moreover, unlike the dot product, this determinant can tell left from right. On the other hand, it can’t tell the order of the points when they’re colinear, which the dot product can.

Now, assuming that $d_0\ne0$, compare the signs of the corresponding determinants for $\triangle{APB}$ and $\triangle{BPC}$ $$d_A=\det\begin{bmatrix}A_x&A_y&1\\P_x&P_y&1\\B_x&B_y&1\end{bmatrix} \text{ and } d_C=\det\begin{bmatrix}B_x&B_y&1\\P_x&P_y&1\\C_x&C_y&1\end{bmatrix},$$ respectively, to the sign of $d_0$: If they are all the same, then $P$ lies within the “narrow” side and you can use the angle $\theta$ computed in your question; if any differ, then $P$ is on the “wide” side and you want $2\pi-\theta$. If either $d_A$ or $d_C$ is zero, ignore it; if both are zero, then $P=B$ and you might as well use $\theta$.

If you expand these determinants, you might find that the resulting expressions look familiar. For example, $d_0 = (A-B)\wedge(C-B)$, where $\mathbf v\wedge\mathbf w=v_x w_y-v_y w_x$ is the “two-dimensional cross product” of the vectors.