How to determine if a point lies on the same side of the plane that origin does?
Basic method
Simply plug in both points $A=(2,-1,1)$ and $O=(0,0,0)$ to the expression: $$ 5x-3y+z-18 $$ and compare signs. A point to the same side of the plane as the direction of the normal vector $\vec n=(5,-3,1)$ will result in a positive value of the expression above, whereas a point in the plane results in the value zero, and a point opposite the normal vector yields a negative value. $$ 5\cdot 2-3\cdot(-1)+1-18=10+3+1-18=-4 $$ and for $(0,0,0)$ all terms but $-18$ vanish, so indeed they both lie opposite $\vec n$. In the negative half-space defined by the plane and the normal vector.
Background
The reason for this is that the plane equation has come about based on the dot product: $$ a(x-x_0)+b(y-y_0)+c(z-z_0)=0 $$ where we have the normal vector $$ \vec n= \begin{pmatrix} a\\ b\\ c\end{pmatrix} $$ and the vector from a point in the plane $P=(x_0,y_0,z_0)$ to a variable point $Q=(x,y,z)$: $$ \overrightarrow{PQ}= \begin{pmatrix} x-x_0\\ y-y_0\\ z-z_0\end{pmatrix} $$ Considering the well-known result $\vec v\cdot\vec w=|\vec v||\vec w|\cos(\varphi)$ for two vectors $\vec v,\vec w$ and the angle $\varphi$ between them, we have:
- $\vec v\perp\vec w\iff \vec v\cdot\vec w=0$
- $\varphi=\angle(\vec v,\vec w)<90^\circ\iff \vec v\cdot\vec w>0$ (positive dot product implies normal vector in same direction as vector from $P$ in plane to $Q$)
- $\varphi=\angle(\vec v,\vec w)>90^\circ\iff \vec v\cdot\vec w<0$ (negative dot product implies normal vector in opposite direction as vector from $P$ in plane to $Q$)
The line $(x,y,z)=(2,-1,1)t$ passes through the origin, and your point. Consider values of the form $$A=5(2t)-3(-t)+t-18.$$ Solving for the value of $t$ that makes $A$ zero tells us the value of $t$ when the line passes through the plane. If that value of $t$ is between $0$ and $1$, the line passes through the plane after leaving the origin, but before arriving at your point, so the two points are on different sides. If $t<0$ or $t>1$, then the line intersects both points either before or after intersecting the plane, so they are on the same side.
Let $F(x,y,z)=5x-3y+z-18$. The plane is defined by $F(x,y,z)=0$. Then check $$ F(2,-1,1)=10+3+1-18=-4 < 0 \\ F(0,0,0) = -18 < 0. $$ These points are on the same side of the plane.