What does Peskin's square root of a matrix mean?

Take a self-adjoint matrix $A$. There exists a unitary matrix $U$ such that $UAU^*$ is diagonal. Take the square root of every diagonal element in order to define $\sqrt{UAU^*}$ (now you are allowing for roots of negative numbers, so imaginary numbers as well). Then rotate the matrix back with $U^*$ and set $$\sqrt A := U^*\sqrt{UAU^*}U.$$ Both of your problems are now solved, because order doesn't enter here in any way once you rotate the square root back and if $\sqrt{p_k}$ is a real number, then $\sqrt{-p_k}$ is imaginary, hence the two matrices have different roots.

The form given in Peskin and Schroeder is useless except in the high energy and low energy limits and to remind the reader that taking the square of these matrices gives the result you would expect (e.g., $\sqrt{p.\sigma}^2=p.\sigma$). In practice a more useful form to use is, $$ \sqrt{p.\sigma} \equiv \frac{E_p+m-{\bf \sigma}.\bf{p}}{\sqrt{2(E_p+m)}} $$ and $$ \sqrt{p.\bar{\sigma}} \equiv \frac{E_p+m+{\bf \sigma}.\bf{p}}{\sqrt{2(E_p+m)}} $$ These can be derived from the other answer (though this is not a trivial exercise).