Parenthesis vs brackets for matrices
It is just a question of notation and what you are comfortable with. The only thing you might want to be aware is |A| stands for determinant of the matrix A and hence cannot be used in place of () or [] to represent the matrix
They're interchangeable. I think a lot of people tend to use parentheses just because they're easier to write by hand.
There are two notations for matrices, ( ), [ ]. Use the one with which you are comfortable with.
Below are examples of both for the 3x3 Identity matrix.
$$
\begin{pmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{pmatrix}
\\
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
$$