Why don't electron-positron collisions release infinite energy?

This is a great question! It can be answered on many different levels.

You are absolutely right that if we stick to the level of classical high school physics, something doesn't make sense here. However, we can get an approximately correct picture by "pasting" together a classical and a quantum description. To do this, let's think of when the classical picture breaks down.

In relativistic quantum field theory, the classical picture of point particles breaks down when we go below the Compton wavelength $$\lambda = \frac{\hbar}{mc}.$$ It isn't impossible to localize particles smaller than this length, but generically you will have a significant probability to start creating new particles instead. Now, the electric potential energy released by the time we get to this separation is $$E = \frac{e^2}{r} = \frac{e^2 m c}{\hbar}$$ in cgs units. Here one of the most important constants in physics has appeared, the fine structure constant which characterizes the strength of electromagnetism, $$\alpha = \frac{e^2}{\hbar c} \approx \frac{1}{137}.$$ The energy released up to this point is $$E \approx \alpha m c^2$$ which is not infinite, but rather only a small fraction of the total energy.

Past this radius we should use quantum mechanics, which renders the $1/r$ potential totally unapplicable -- not only do the electrons not have definite positions, but the electromagnetic field doesn't even have a definite value, and the number of particles isn't definite either. Actually thinking about the full quantum state of the system at this point is so hairy that not even graduate-level textbooks do it; they usually black-box the process and only think about the final results, just like your high school course is doing. Using the full theory of quantum electrodynamics, one can show the most probable final outcome is to have two energetic photons come out. In high school you just assume this happens and use conservation laws to describe the photons long after the process is over.

For separations much greater than $\lambda$, the classical picture should be applicable, and we can think of part of the energy as being released as classical electromagnetic radiation, which occurs generally when charges accelerate. (At the quantum level, the number of photons released is infinite, indicating a so-called infrared divergence, but they are individually very low in energy, and their total energy is perfectly finite.) As you expected, this energy is lost before the black-boxed quantum process starts, so the answers in your school books are actually off by around $1/137$. But this is a small enough number we don't worry much about it.

Positronium is what you're describing in your first idea.

Until/unless a believable model of the electron is developed, pretty much the best we can do is say that the rest mass of the electron-positron system is the only energy available to be converted to radiation in an annihilation event.

This experiment will reveal a lot more about what happens in very close encounters between electrons and positrons.