Does QED really break down at the Landau pole?

You are completely correct that the perturbative calculation of the Landau pole can't be trusted, as it will clearly become invalid long before the putative pole is reached. The only method that we know of that can give accurate predictions for the high-energy behavior of QED is numerical simulation. According to https://arxiv.org/abs/hep-th/9712244 and http://www.sciencedirect.com/science/article/pii/S092056329700875X, numerics suggests that QED is indeed quantum trivial (i.e. $e$ always renormalizes to zero for any choice of bare coupling), but not because of a Landau pole, which is the usual explanation. Instead, chiral symmetry breaking kicks in before the Landau pole is reached. So there is no Landau pole at high energies, but there is a different phase transition that causes QED to break down.

The IR Landau pole in QCD doesn't render the theory inconsistent, but it is a hint of serious trouble. It's a harbinger of confinement: It's telling you that, absent a miracle, the interactions between quarks and gluons at low energies are so intense that their correlation functions will cease to be well-defined when one tries to separate them by more than this scale. There are composite operators in QCD which make sense below the confinement scale, but they're necessarily complicated combinations of quarks and gluons, like glueballs and hadrons.

The same problem occurs in QED in reverse. Absent a mathematical miracle, the correlation functions between electrons will fail to be defined if you bring them too close together. You might get lucky and discover that there are analogues of the hadron operators that make sense to arbitrarily high energy. But this would be a mathematical curiousity: You'd have a QFT where the elementary fields emerged from their composites at low energies.. It's hard to imagine such a theory could be unitary.