How many irreducible monic quadratic polynomials are there in $\mathbb{F}_p[X]$?

Monic reducible degree two polynomials have a root, so they are of the form $(X-a)(X-b)$.

There are $p$ polynomials of the form $(X+a)^2$ and $\binom{p}{2}$ of the form $(X+a)(X+b)$ with $a\ne b$ (as you computed). So $$ p^2 - p - \binom{p}{2}=\frac{p(p-1)}{2} $$ is the correct number.

Examples. For $p=2$ the reducible polynomials are $X^2$, $X(X+1)$ and $(X+1)^2$; so you get $4-3=1$, indeed the only one is $X^2+X+1$.

For $p=3$ we have $X^2$, $(X+1)^2$, $(X+2)^2$, $X(X+1)$, $X(X+2)$, $(X+1)(X+2)$. The three irreducible polynomials are $X^2+1$, $X^2+X+2$, $X^2+2X+2$.

Hint: the product of all monic irreducible polynomials of degree dividing $n$ with coefficients in $\mathbf F_p$ is $x^{p^n}-x$.

So, with $n=2$, you just have to exclude the linear monic polynomials...