Who pays travel & per diem costs for invited lecturers?

The short answer is that it depends (no surprise there). If it's to give a talk in a department colloquium or seminar, travel cost are usually covered (at least, that's been my experience in mathematics), though that can vary a bit, especially since smaller departments may have very limited budgets for that kind of thing.

Food expenses vary even more wildly, and they depend heavily on the source of funding. I've been at places that had a standard per diem for visitors, places that did nothing, and places where the faculty would take the speaker to dinner and split the cost among them.

In my experience (as a visiting guest lecturer while a grad student elsewhere) the travel expense and accommodation were covered not by the university that hosted me, but by grant money from the professor that invited me. That being said, these visits were for a few days and included both the guest lecture, and some research/discussion on ongoing projects.

I did not claim a per diem for food, but I am sure that if I had it would have been covered, too.

When you are an invited guest of a university department, in most cases you will have your travel costs paid by the department, or in the case of an individual institute or professor extending the invitation, by the specific people involved. In the case of a short visit (less than a day, for instance), when a per diem becomes impractical, the cost of the meal after the conference might be split among the attendees.

However, I have been an invited lecturer at conferences that were organized at various schools. In those cases, the conferences were operating on a very limited budget, and the "compensation" was essentially a free registration to the conference.