Why do the Perseids only happen once per year, not twice or four times?

Comets do orbit in ellipses, but there is no requirement for their orbital planes to match the Earth's exactly. Short-period comets have planes that are relatively parallel to the Earth's (with a good degree of lee-way), and long-period comets can come in from any direction and with any orbital inclination.

Here is a good representation of the orbit of comet 109P/Swift-Tuttle:

When the comet's orbit crosses the orbital plane of the Earth again, it is way out past Jupiter, and this means that there is just the single meteor shower from this debris track.

Also, it's worthwhile noting that while ellipses in general can intersect four times, keplerian orbits share a focus, which means that they can intersect at most twice (which they do when they're coplanar and the sizes are right).

Swift-Tuttle is orbiting the Sun in a different plane than the Earth: its orbital path has an inclination of 113° with respect to the mean ecliptic, i.e. its orbit is almost perpendicular to the Earth's orbit. Therefore their orbits only intersect in one place, as this picture makes clear (the cyan/blue curve is the comet's path):