Covariance in gauge theories: why should the Lagrangian be gauge invariant

We do not start from the assumption that the Lagrangian "should" be invariant under gauge transformations. This assumption is often made because global symmetries are seen as more natural than local symmetries and so writers try to motivate gauge theory by "making the global symmetry local", but this is actually nonsense. Why would we want a local symmetry just because there's a global one? Do we have some fetish for symmetries so that we want to make the most symmetrical theory possible? One can derive gauge theory this way but as a physical motivation, this is a red herring.

The actual point is not that we "want" gauge symmetry, but that it is forced upon us when we want to describe massless vector bosons in quantum field theory. As I also allude to in this answer of mine, every massless vector boson is necessarily described by a gauge field. A Lagrangian gauge theory is equivalently a Hamiltonian constrained theory - either way, the number of independent degrees of freedom that are physically meaningful is less that the naive count, since we identify physical states related by gauge transformations.

The true physical motivation for gauge theories is not "we want local symmetries because symmetries are neat". It's "we want to describe a world with photons in it and that can only covariantly be done with a gauge theory".

A non-quantum motivation of gauge theory can also be given: If you write down the Lagrangian of free electromagnetism, motivated because its equations of motion are the Maxwell equations, not because we like gauge symmetry, then you find it comes naturally with a $\mathrm{U}(1)$ gauge symmetry, corresponding to the well-known fact that adding a gradient to the vector 4-potential is physically irrelevant. Now, if you want to couple other fields to this free electromagnetism, you need to make the additional terms gauge invariant, too, else the theory is no longer "electromagnetism coupled to something else" in any meaningful sense since suddenly adding gradients can change the physics. Once again, gauge symmetry is something one discovers after physically motivating the Lagrangian from something else, not some sort of a priori assumption we put in.