Does swapping columns of a matrix cause the rows of the inverse matrix to be swapped?

Yes, this is always true. Note that swapping columns $i$ and $j$ is equivalent to multiplying on the right side by the elementary matrix $T_{ij}$ which is defined by swapping rows $i$ and $j$ of the identity matrix. You can check that this matrix is the inverse of itself. Also, multiplying by $T_{ij}$ on the left side is equivalent to swapping rows $i$ and $j$.

So if we call your matrix $A$ then $AT_{ij}$ is the matrix that you get that swapping columns $i$ and $j$. Then its inverse is $T_{ij}A^{-1}$ which is the matrix that you get when you swap rows $i$ and $j$ in $A^{-1}$.

Another way to think of it: you are just re-numbering the basis for one of the vector spaces: The domain space for $A$, which is the range space fo $A^{-1}$.

Your observation is spot on. You can swap columns of a matrix by right-multiplying by a permutation matrix $P$ that is the identity matrix with the corresponding columns swapped: $\hat A = AP$. We then have $$\hat A^{-1} = (AP)^{-1} = P^{-1}A^{-1}.$$ $P$ is its own inverse, and left-multiplying by $P$ swaps the rows of $A^{-1}$ that are swapped in $P$.