Given four points, determine a condition on a fifth point such that the conic containing all of them is an ellipse

One can always construct two parabolas passing through $p_1$, $p_2$, $p_3$, $p_4$ (green and pink in the figure below), each one possibly degenerating into a couple of parallel lines if two opposite sides of quadrilateral $p_1p_2p_3p_4$ are parallel. Point $p_5$ will determine an ellipse if it lies inside either parabola but not in their intersection.

This follows from the fact that five points always determine a conic section, and because the parabola is a limiting case between ellipse and hyperbola: each time $p_5$ crosses the boundary of a parabola, conic section $p_1p_2p_3p_4p_5$ switches from ellipse to hyperbola (or viceversa).

I'll expand-upon my hint, because the raw calculations get pretty ugly, yet there is a reasonably-tame way to express the desired condition.

---

The equation of a conic through $P$, $Q$, $R$, $S$, $T$ is given by:

$$\left|\begin{array}{cccccc} x^2 & y^2 & x y & x & y & 1 \\ P_x^2 & P_y^2 & P_x P_y & P_x & P_y & 1 \\ Q_x^2 & Q_y^2 & Q_x Q_y & Q_x & Q_y & 1 \\ R_x^2 & R_y^2 & R_x R_y & R_x & R_y & 1 \\ S_x^2 & S_y^2 & S_x S_y & S_x & S_y & 1 \\ T_x^2 & T_y^2 & T_x T_y & T_x & T_y & 1 \\ \end{array} \right| = 0 \tag{1}$$

Expanding the determinant yields an equation of the form $$A x^2 + B x y + C y^2 + D x + E y + F = 0 \tag{2}$$ This represents an ellipse when $$B^2-4AC < 0 \tag{3}$$ (likewise, a hyperbola when $>0$, and a parabola when $=0$).

Condition $(3)$, in $xy$-coordinates, turns out to be an expression in over $14,000$ terms. We can collapse the complexity a bit by using a self-induced coordinate system; specifically, we'll use barycentric coordinates based on $\triangle PQR$ (which we'll assume is non-degenerate).

We can give $S$ and $T$ respective coordinates $(s_P:s_Q:s_R)$ and $(t_P:t_Q:t_R)$. That is, we can write $$S = \frac{s_P P + s_Q Q + s_R R}{s_P+s_Q+s_R} \qquad\qquad T = \frac{t_P P + t_Q Q + t_R R}{t_P + t_Q+t_R} \tag{4}$$ Substitution into $(3)$ collapses the relation into a mere $21$ terms (and a discardable factor corresponding to the area of $\triangle PQR$). This is better, but still a little messy. It cleans up nicely when reciprocating the elements, however; defining $x' := 1/x$, we can write

$$\begin{align} &\phantom{4}\left( s^{\prime}_P t^{\prime}_Q + s^{\prime}_Q t^{\prime}_R + s^{\prime}_R t^{\prime}_P + s^{\prime}_Q t^{\prime}_P + s^{\prime}_R t^{\prime}_Q + s^{\prime}_P t^{\prime}_R \right)^2 \\[4pt] <\; &4 \left(s^{\prime}_P s^{\prime}_Q + s^{\prime}_Q s^{\prime}_R + s^{\prime}_R s^{\prime}_P \right) \left(t^{\prime}_P t^{\prime}_Q + t^{\prime}_Q t^{\prime}_R + t^{\prime}_R t^{\prime}_P \right) \\ \left(\quad =\; \right. & \frac{4}{s_P s_Q s_R\,t_P t_Q t_R}\left.\left(s_P+s_Q+s_R\right)\left(t_P+t_Q+t_R\right)\quad\right) \end{align} \tag{$\star$}$$

If we know $P$, $Q$, $R$, $S$ (and therefore $s_P$, $s_Q$, $s_R$ and $s^\prime_P$, $s^\prime_Q$, $s^\prime_R$), then $(\star)$ gives a condition on parameters defining $T$.

(There's probably a nice projective interpretation of $(\star)$ that could've saved all the trouble of deriving it. :)

Adding to the direct answer in terms of parabolas: If you are reduced to ruler and compass or just do not want to actually draw the two bounding parabolas, you may want to transform the problem to a geometrically simpler one.

The method described below does this. It can be extended to find slopes of asymptotes (if any) and of principal axes when given five points of a conic section.

To simplify the description, I assume a projective geometry context. Thus I may speak of points at infinity and the line at infinity. Points at infinity occur as intersections of parallel lines and can be interpreted as slopes of those lines. A point is at infinity if and only if it lies on the line at infinity.

Relabel $p_1,\ldots,p_4$ to $A,B,C,D$. Any permutation is admissible. (Some details will depend on the permutation, but the overall result won't.) Let me also rename $p_5$ to $E$. We will use $ABC$ as reference triangle below.

Relative to that reference triangle $ABC$, there is a transformation of points called isogonal conjugation. The inverse transformation is again isogonal conjugation. However, isogonal conjugation of the vertices $A,B,C$ is undefined. Therefore isogonal conjugation is a bijection only for points outside the lines $\overline{AB}$, $\overline{BC}$, $\overline{CA}$. In our use case, the resulting singularities can be removed by continuity.

Finding the isogonal conjugate of a point can be done using only ruler and compass. Furthermore:

• Applying isogonal conjugation to the points of a line yields a conic section through $A,B,C$ (a circumconic of $ABC$), and every non-degenerate circumconic of $ABC$ can be obtained that way.
• Particularly, applying isogonal conjugation to all points at infinity (i.e. the line at infinity) yields the circumcircle of $ABC$.
• Extension: The slope of a line (i.e. its point at infinity) determines the slopes of the principal axes of the circumconic obtained by pointwise isogonal conjugation of the line.

Given points $D,E$, we can construct their isogonal conjugates $D',E'$ and join them with a line $g = \overline{D'E'}$. That line is the pointwise isogonal conjugate of the conic section through $A,B,C,D,E$.

(You could now place a point $F'$ arbitrarily on $g$ and construct its isogonal conjugate $F$ to obtain another point on the conic.)

Now consider the following cases:

1. If the line $g$ passes through one of the vertices of the reference triangle $ABC$, e.g. $A$, then isogonal conjugation runs into a singularity. However, algebraically, we can still argue that there exists a unique corresponding circumconic, degenerated to a pair of lines, one of which passes through that same vertex $A$, and the other matches the opposite side $\overline{BC}$.

   Consequently, for a line through two vertices, e.g. $\overline{AB}$, the corresponding conic consists of the other two extended sides $\overline{BC}, \overline{CA}$.

2. If $g$ intersects the circumcircle $U$ of $ABC$ in two distinct real points $P_1',P_2'$, this means that the corresponding conic intersects the line at infinity in two distinct points $P_1,P_2$ (at infinity). A conic with two distinct points at infinity is a hyperbola (possibly degenerated to a pair of nonparallel lines if case 1 applies), and those points at infinity identify the slopes of its asymptotes.

   Extension: $P_1,P_2$ can be obtained as isogonal conjugates of $P_1',P_2'$. When breaking this down to more elementary steps, each $P_i$ will arise as intersection of parallel lines. Since you only need to know the associated slope, you are done as soon as you have constructed one of those lines.

3. If $g$ touches the circumcircle $U$ of $ABC$ in one real point $P'$, this means that the corresponding conic has a double point $P$ at infinity. Such a conic is a parabola (possibly degenerated to a pair of parallel lines if case 1 applies), and its point at infinity gives the slope of the parabola's symmetry axis.

4. If $g$ does not intersect the circumcircle $U$ of $ABC$, this means that the corresponding conic contains no point at infinity. Such a conic is an ellipse.

Extension: if $g$ is not the line at infinity, the slopes of the corresponding circumconic's principal axes can be obtained by determining the points $H_1',H_2'$ on the circumcircle $U$ with tangents parallel to $g$ and transforming them back to points $H_1,H_2$ at infinity.

The solution to your problem is therefore to construct the isogonal conjugate $D'$ of $D$ and its tangents to the circumcircle $U$ of $ABC$. Then, when given another point $E$ of the conic, construct its isogonal conjugate $E'$ and test whether $\overline{D'E'}$ intersects $U$. The region for $E'$ where such intersection happens is bounded by the tangents to $U$ through $D'$ and is tinted green in the figure below.

• If $E'$ is inside the green region, the conic is a hyperbola, possibly degenerated to a pair of nonparallel lines (if $\overline{D'E'}$ intersects a vertex of $ABC$).
• If $E'$ lies on a bounding tangent, the conic is a parabola, possibly degenerated to a pair of parallel lines (if the point of tangency is a vertex of $ABC$).
• Otherwise the conic is an ellipse.

For those bounding tangents to exist, $D'$ must not be inside $U$. Isogonal conjugation translates this requirement to $A,B,C,D$ being vertices of a convex 4-gon, as given in the problem statement.