Gift for mentor after his thesis defense?

A short (1-2 paragraph) hand written note on a card or postcard would be an appropriate gift.

I have mentored about a dozen undergraduates and I still have the thank you notes they sent to me on my pin board in my office. The notes are nice reminders about the impact I had on the students.

Actually, the thing that would be most appreciated - and valued - is a hand written letter on nice stationery giving congratulations and thanking him for his help in your own work.

Short, professional, sincere.

He will save it forever.

This is culturally dependent. You should follow local cultural norms.

When I was in Sweden, the PhD candidate would, after their successful PhD defence, organise a party, with typically between 30 and 50 people invited, such as family and friends from the office and outside. If they were supervising a few undergraduate students, they would probably invite them. At such parties, it was entirely common that people would bring gifts, which would all be placed on a table. If such a party exists and you are invited, by all means bring a small gift, similar to what you would bring if invited to a birthday party. I received food, books, souvenirs, and a few joke presents.

However, if such a party does not exist or you are not invited, and such gifts are unusual in your location, then it may be more appropriate to stick with a card such as suggested in other answers.