What are the waveguide modes of twisted pairs as discussed here?
At a certain frequency, you start getting issues because you can fit more than one wavelength in between the wire pairs. This is also the reason coaxial cable and it's connectors get smaller as we go up in frequency (to 0.8mm connectors Anritsu is working on).
I've give an short explanation of 2 unconventional modes they might be referring to. Ofcourse, I've not seen the actual presentation, so this is just a guess on my part. I've never heard of anything like a "twisted pair" mode. The only thing that might be somewhat similar is a slotline, but I'm not very familiar with slotlines.
Surface Waves
Surface waves have been known about for a long time now, however, I've not seen them used outside of research. When using these, you are using the copper more as a optical-fiber style wave guide, and not a pair. The wave would be trapped between either the copper and the polymer covering it, or (but I don't think this is possible in telephone twisted pairs) the oxide forming on top of the conductor could server as the trapping dielectric. If my memory serves this right, Sommerfeld first came up with a description of these waves. The topic was later popularized by Goubau, which is why they are still sometimes called Goubau waves or the Goubau mode. The lines carrying these are then refered to G-lines.
Some references to look at on this topic:
Goubau’s paper On the Excitation of Surface Waves, Proceedings of the IRE, vol. 40, Issue 7, 1952.
G. Goubau, Open wire lines, IRE Trans. Microw. Theory Tech., vol. MTT-4, no. 10, pp.197-200, Oct. 1956.
M. J. King and J. C. Wiltse, Surface-wave propagation on coated or uncoated metal wires at millimeter wavelengths, IRE Trans. Antennas Propag., vol. AP-10, no. 5, pp. 246-254, May 1962.
There are also some more recent papers on the topic, I believe one of the researchers on the topic is/used to be Tahsin Akalin, but I don't know if he is still working on them. The main issue are losses that tend to still be quite strong. There is also the matter that in any research I've seen, coupling losses are quite high, and the launching structures are big or expensive, and would need to be installed at the users home to use this. There is also the matter that at any frequency this high, you are going to loose large amounts of signal whenever there is any significant bend, and from how I've seen the telephone twisted pair lines treated in average homes...
Polymer Waveguide mode
Another interesting development in research is the polymer waveguide (note, I'm working on this myself soon, so I might be more excited about this than it actually is).
In short, at mm-wave frequencies, copper (and any metal really) become poor performing cariers. However, we can start using polymers (the ones I've commonly seen used are PTFE, Styrene and LCP) as our waveguides. Problems here are the launching structures and fabrication of the high-quality polymer carriers. While these modes can be excited in hollow polymer waveguides, I don't know if it will still work/how things change when we introduce a copper core. (I suspect not because of the high losses within the metal).
Dielectric waveguides (the family of waveguides that polymer waveguides belong to) have been studied for a long time, and optical fiber is the most common member of this family. However, in recent years interest has turned to polymer waveguides for mm-wave applications as they provide flexible alternatives to rigid waveguides, and seem to be able to perform as well or better than coaxial cable (and, to higher frequency). Another advantage over rigid waveguides is that they tend to be higher bandwidth. One of the main issues now if finding launching structures that are good, and potentially integrating these into connectors.
There are a lot of papers on polymer waveguides, so I'm not going to list my favorites.