Postgres slow query with order by, index and limit

$M$ of coordinates $(x,y)$ is inside the rectangle iff

$$(0<\textbf{AM}\cdot \textbf{AB}<\textbf{AB}\cdot \textbf{AB}) \land (0<\textbf{AM}\cdot \textbf{AD}<\textbf{AD}\cdot \textbf{AD})$$ (scalar product of vectors)

Let $P(x,y)$, and rectangle $A(x_1,y_1),B(x_2,y_2),C(x_3,y_3),D(x_4,y_4)$

Calculate the sum of areas of $\triangle APD, \triangle DPC, \triangle CPB, \triangle PBA$.

1. If this sum is greater than the area of the rectangle, then $P(x,y)$ is outside the rectangle.

2. Else if this sum is equal to the area of the rectangle (observe that this sum cannot be less than the latter),

   1. if area of any of the triangles is $0$, then $P(x,y)$ is on the rectangle (in fact on that line corresponding to the triangle of area$=0$). Observe that the equality of the sum is necessary; it is not sufficient that area$=0$),

   2. else $P(x,y)$ is is inside the rectangle.

Acceptably this approach needs substantial amount of computation. This approach can be employed to any irregular polygon, too.

---

Another way is to calculate the perpendicular distances of $P(x,y)$ from all the 4 lines $AB,CD, AD,BC$

To be inside the rectangle, the perpendicular distances from $AB, P_{AB}$(say) and from $CD, P_{CD}$(say) must be less than $|AD|=|BC|$ and the perpendicular distances from $AD, P_{AD}$(say) and from $BC, P_{BC}$(say) must be less than $|CD|=|AB|$. Here , the areas of each of the four triangles < $\frac{1}{2}$the area of the rectangle.

1. If one of the perpendicular distances is greater than the respective length, then $P(x,y)$ is outside the rectangle.

   This essentially implies and is implied by the statement : the area of the respective triangle > $\frac{1}{2}$the area of the rectangle (as commented by Ben Voigt) as $\triangle APD=\frac{1}{2}AD\cdot P_{AD}$.

2. Else if $ P_{AB}=0$ and $P_{CD}=|AD|$ , then $P(x,y)$ is on AB . So, $\triangle PBA=0$ and $\triangle PCD=\frac{1}{2}$the area of the rectangle.

   Observe that in this case, the remaining two perpendicular distances $P_{AD}, P_{BC}$ must be ≤ $|AB|=|CD|$, $P_{BC}=|AB|\implies P(x,y)$ is lies on AD i.e, P coincides with A as it is already on AB .

I would use a "point-in-convex-polygon" function; this works by checking whether the point is "to the left of" each of the four lines.