Must magnetic field lines close upon themselves or go to infinity?

I had the same question a few years ago, both for magnetic field lines and for current lines in the quasi-static approximation. The following papers from the American Journal of Physics convinced me that, indeed, magnetic field lines are not always closed or extending to infinity.

• "Lines of force in electric and magnetic fields": J. Slepian, Am. J. Phys. 19 no. 2, 87 (1951)
• "Topology of steady current magnetic fields", K.L. McDonald, Am. J. Phys. 22 no. 9, 586 (1954)
• "The magnetic fields lines of a helical coil are not simple loops", M. Lieberherr, Am. J. Phys. 78 no. 11, 1117 (2010)

Nope. Conventional wisdom is wrong about this one. Magnetic field lines can have much more interesting configurations.

Magnetic field lines are trajectories of Hamiltonian systems and, as such, are typically chaotic, which means they never close on themselves (if they did, they'd be periodic, not chaotic). This case is made very clear in this paper by Phil Morrison: Magnetic field lines, Hamiltonian dynamics, and nontwist systems (e-print, without figures).

The dense covering of a toroidal surface you mention is also typical: in fact, these so-called quasiperiodic lines are infinitely more common than their periodic counterparts - essentially because irrational numbers are infinitely more common than rational numbers.

Again, in the same way that a real-life mechanical system is typically chaotic rather than periodic, real life magnetic fields will most often also be chaotic, even for simple configurations and that's actually probably good.

In the image bellow, electric current flows through the black lines (belonging to two perpendicular wire loops), generating the field line showed in red. From a paper that makes a good case that

chaotic magnetic-field lines are ubiquitous in the modern technological world.