Gaussian matrix integration
We have $Tr(B^2)=\sum_i\sum_j B_{ij}B_{ji}=\sum_i\sum_j \mid B_{ij}\mid^2=\sum_i B_{ii}^2+2\sum_{i<j}Re(B_{ij})^2+Im(B_{ij})^2$. Thus the integrand becomes $\prod_i e^{-nx_{ii}^2/2}dx_{ii}\prod_{i<j}e^{-2nr_{ij}^2/2}dr_{ij} e^{-2nI_{ij}^2/2}dI_{ij}$, by independence of the components (I used a different notation for the dummy variables just for simplicity). Then, each term in the first product is $\int e^{-nx^2/2}dx=(2\pi/n)^{1/2}$, while each term in the second is $((\pi/n)^{1/2})^2$. There are N terms in the first product and $N(N-1)/2$ in the second, so we get $(2\pi/n)^{n/2}(\pi/n)^{n(n-1)/2}=2^{n/2}(\pi/n)^{n^2/2}$
Edit: I also think it is incorrect to say that $dB_{ii}$ is a gaussian measure. If you assume the entries are jointly distributed according to dx (normalized to 1), then it is true they are independent gaussians, but in the equation defining dx, the measure $dB_{ii}$ is just Lebesgue measure on the line (try to replace it with a gaussian measure and you will get a different answer)