Is the Dirac Lagrangian Hermitian?

$$(\psi^\dagger \gamma^0 \psi)^* = \psi^\dagger \gamma^0 \psi$$ because $\gamma^0$ is hermitian. Also,

$$ \begin{align} (\psi^\dagger i \gamma^0 \gamma^\mu \partial_\mu \psi)^* &= -i \partial_\mu\psi^\dagger \gamma^{\mu\dagger} \gamma^0 \psi\\ &= -i \partial_\mu\psi^\dagger (\gamma^0 \gamma^\mu \gamma^0)\gamma^0 \psi\\ &= -i \partial_\mu\psi^\dagger \gamma^0 \gamma^\mu \psi\\ &= i \psi^\dagger \gamma^0 \gamma^\mu \partial_\mu\psi + \mathrm{surface\,\,term}\\ \end{align} $$ For the second line I used $\gamma^{\mu\dagger} = \gamma^0 \gamma^\mu \gamma^0$ and for the last line I integrated by parts. I think your question hinges on this part, because the last "index" we sum over is the spacetime index $x^\mu$, i.e., integration. It is the same reason why the quantum mechanical momentum operator $p = i \tfrac{\partial}{\partial x}$ is hermitian.

Edit: Something I glossed over is that the spinors are also Grassmann numbers, so care has to be taken. In particular, this means that the components of the spinors satisfy

$$(\psi_i \phi_j)^* = \phi_j^* \psi_i^*$$

(more about that here). One already interchanges the objects when taking the Hermitian conjugate by the rules of matrix algebra, and there is a temptation to want to introduce a minus sign because they are Grassmann numbers, but this would be redundant. Borrowing from the linked math.se answer: $$ \begin{align} (\eta\xi)^*&=[(a+ib)(c+id)]^*\\ &=(ac-bd+ibc+iad)^*\\ &=ca-db-icb-ida\\ &=(c-id)(a-ib)=\xi^*\eta^* \end{align} $$

In principle what matters is the action and not the Lagrangian density that, strictly speaking must not be considered as an observable, since it is defined up to boundary terms. Concerning the free Lagrangian you are considering, it is real up to a boundary term. You can also re-define it just by adding the Hermitean conjugate Lagrangian you wrote to the original one and taking one half the result. This new Lagrangian density is real and equivalent to the initial one.