How many points are needed to uniquely define an ellipse?
The equation $\left(\frac{x-h}{a}\right)^2 + \left( \frac{y-k}{b}\right)^2 = 1$ is the equation for an ellipse with major and minor axes parallel to the coordinate axes. We expect such ellipses to be unchanged under horizontal reflection and under vertical reflection through their axes. In this equation, these reflections are effected by $x \mapsto 2h - x$ and $y \mapsto 2k -y$.
This means, if all you have is one point on the ellipse and the three reflected images of this point, you do not have $8$ independent coordinates; you have $2$ and uninformative reflections forced by the equation.
We can see this by plotting two ellipses at the same center (same $h$ and $k$), intersecting at $4$ points, with, say, semiaxes of length $1$ and $2$.
These clearly have four points of intersection. But as soon as you know an ellipse is centered at the origin and contains any one of the four points of intersection, by the major and minor axis reflection symmetries, it contains all four. This is still true if you use generic ellipses, which can be rotated.
Remember that the reflections are through the major and minor axes, wherever they are.
Of course, there are other ways for two ellipses to intersect at four points.
So just knowing those four points are on an ellipse cannot possibly tell you which one is intended.
Returning to the first diagram, corresponding to the diagram you have where the four known points are the vertices of a square... Symmetries force the center of the ellipse to be the center of the square, but that's not a very strong constraint.
The equation of an ellipse is: $$ ax^2+by^2+cxy+dx+ey+f=0 $$ Hence you need $5$ points to obtain the coefficients: $(a,b,c,d,e,f)$, assuming that the center is unknown.
If, on the the other hand, the center is known then $3$ points are enough, since every point's reflection in respect to the center is also a point of the ellipse and you technically have $6$ known points.
In this case, assuming that the blue-painted point is the center, the points you've been given are symmetric in respect to it and thus it's like having the center and $2$ points, which is not enough to uniquely determine an ellipse.
This is exactly the question discussed several months ago here (Chinese). The central problem is the hidden constraints put on the ellipse.
When you claim that ellipses are determined by the equation $\frac{(x-h)^2}{a^2}+\frac{(y-k)^2}{b^2}=1$, you are implicitly assuming that there are no 'slant' ellipses. For instance, the equation $x^2+y^2+xy=1$ also characterize an ellipse, but it is not included in your equation.
If by 'determine' you mean that the ellipse is the unique one passing through the points, the answer is 5. Since two ellipses can intersect at four points, these 4 points cannot determine a unique one. On the other hand, you can easily construct the unique quadratic curve passing through any 5 given point.
However, you can, in fact, use only one point to specify an ellipse. Since the set of all ellipses $E$ is equipotent to $\mathbb R^5$, as discussed above, and $\mathbb R^5$ is equipotent with $\mathbb R^2$, there is a one-to-one correspondence between $\mathbb R^2$ and $E$. You can see a formal discussion here.