Why is the expected value (mean) of a variable written using square brackets?

Mathematicians usually adopt square bracket such that $f[g]$, instead of $f(g)$ to indicate $f$ is a functional. Obviously, random variable $X=X(\omega)$ is a function, which makes expectation, variance and so on functionals. But this is only an unestablished convention, not necessary.

I don't.

You can write in both ways, it doesn't matter. Some don't even use brackets but might instead write $EX$ or $VX$.