Compute the stationary distribution of a large transition matrix

You may run into memory problems because you are constructing a full matrix with IdentityMatrix[n] instead of a sparse one with IdentityMatrix[n, SparseArray]. By using a sparse identity matrix you can do 4000x4000 in about 8 seconds, without excessive memory use:

Dimensions[transit]
(*    {4000, 4000}    *)

eigenVector = First[NullSpace[
  N[transit - IdentityMatrix[Dimensions[transit], SparseArray]]]]; // AbsoluteTiming // First
(*    8.32465    *)

Just make sure you keep all matrices sparse and never construct a full ("normal") matrix.

edit: it's even simpler since this is a discrete-time Markov chain

I didn't know Roman's trick with SparseArray. Here's another trick: use Method->"Arnoldi", asking for only the eigenvector corresponding to the largest eigenvalue (approx. one) using 1, which is the stationary distribution.

evNull=First[NullSpace[N[transit]-IdentityMatrix[4000,SparseArray]]];//AbsoluteTiming
(* 15.6608 -- I must need a faster computer! *)

evArnoldi=Eigenvectors[N@transit,1,Method->{"Arnoldi"}][[1]];//AbsoluteTiming
(* 0.024535 *)

So there's another couple of orders-of-magnitude speedup for you. I'm not patient enough to wait for StationaryDistribution to finish!

The eigenvectors look the same when plotted BTW.