Determining if collection of coordinates (polygon) is ellipse?

I had a project where I needed to classify geometries as circles, ellipses, or irregular polygons. I found that after locating the center of the figure, I could easily classify two coordinates as "closest" and "farthest" point to that center, which then would allow me to derive a possible orientation of the ellipse, and its semi-major and semi-minor axis. Then I just calculated the distance from that center to each of the vertices, and what the hypothetical distance at that angle would be if the figure were an ellipse. If the sum of the deltas between actual and hypothetical, divided by the number of vertices was relatively small, then I could classify the shape as an ellipse, and if semi-major was roughly equal to semi-minor, then it was a circle, otherwise it was a generic polygon.

There were some minor flourishes in the orientation determination (using the two closest and two farthest points), and possibly a square root of sum of squares in the delta determination (I don't have access to the code anymore) but it seemed reliable enough over the hundreds of shapes I had to test against. I had a further complication that the distances all had to be calculated on a WGS84 spheroid, but it even handled high latitude geometries correctly. It's possibly not the most efficient solution, but it wasn't too bad [O(n)], and it was effective.

Assuming a 2d planar surface (since these are projected coordinates).

Where n is the number of points....

For the case where n < 5, they always define an ellipse. For any four points, you can construct an ellipse that goes through all four points. In fact, there can be multiple ellipses which go through all four points.

For n = 5, you can use the coordinates to solve the generalized conic section equation Ax^2 + Bxy + Cy^2 + Dx + Ey + F. The solution to the equation may be a circle (A=C B=0) an ellipse (Ax^2 + Bxy + Cy^2 = -F), a parabola or a hyperbola.

So, the solution path is straightforward, though not necessarily easy. You take the first five points and solve for A, B, C, D, E, & F in the generalized conic section equation. You test the solution to see if Ax^2 + Bxy +Cy^2 = -F for those first five points. If so, you now have the equation for your potential ellipse. You now have to plug in all remaining coordinates after the first five points to test if the equation still holds. If even the very last coordinate is off the ellipse, you no longer have an ellipse.

If all coordinates are valid for the ellipse equation you solve for from the first five coordinates, you have an ellipse. The hard part? Solving your five equations six unknowns to get your conic section equation. I think you can do this with matrix math in a straightforward way, but I am not sure off the top of my head.