Why can't the $\mathbb{Z}_2$ invariant be defined with a determinant instead of a pfaffian?

FWIW, I agree with you.

The index is

$$\frac{1}{2\pi i} \int_C d \vec k \cdot \nabla \log(P(k) + i \epsilon),$$

where $P(k)$ is the Pfaffian. The determinant is $D(k) = P(k)^2$ so we have

$$\log(D(k) + i2\epsilon) = \log((P(k) + i\epsilon)^2) = 2\log(P(k)+i\epsilon),$$

so the index is also

$$\frac{1}{4\pi i} \int_C d \vec k \cdot \nabla \log(D(k) + i \epsilon).$$

I think Kane and Mele might also agree with you. From their paper:

I wanted to point out that the fermionic ground state here is a Slater determinant, which is the point of the formula in the snippet for $P(\alpha)^2$. The determinant shows up because if you exchange indices of a matrix, ie. multiplying by a transposition matrix, the determinant picks up a minus sign. Thus, determinants are natural things for expressing fermionic ground states.

On the other hand, Pfaffians show up when computing (many-body) partition functions of real fermions. If we were computing some twisted partition function of this model, then we really would be concerned with its sign, since that would be precisely this $\mathbb{Z}_2$ invariant. It's an interesting general feature that winding invariants of single-particle ground states are captured by the many-body partition function. This is how the Pfaffian shows up in this classic (where you really need the Pfaffian and not the determinant), and I assume Kane and Mele had this on their mind when they wrote about the Pfaffian, see also Chiral Anomaly's answer. That square-root in the snippet looks reaaally spurious to me.

Pfaffians are not weird objects but very natural for fermionic systems. They follow very similar rules as determinants.

You are missing the topological information in the sign of the Pfaffian. Taking a square root changes the topology of the manifold on which the function is single valued - just like the complex square root is properly defined on a topologically nontrivial Riemannian surface and not just on the plane of complex numbers.

Because of periodicity, winding over a Brillouin zone is well-defined but winding over half a Brillouin zone is path-dependent (without additional arguments).

This is not my specialty, but the Bernevig and Hughes text looked interesting, so I spent some time with it.

I might be missing something, but as far as I can tell from the pieces I've digested, you're right. For the vorticity-based construction of the $Z_2$ invariant, the Pfaffian is being used only in what is presumably a connected region where it is continuous and nonzero (namely the boundary of the half-BZ); and in any such region, the Pfaffian can be reconstructed from its square except for an overall sign that has no bearing on the vorticity.

However, section 10.6 in the Bernevig and Hughes text highlights something that might help explain the general preference for using Pfaffians in this context, in addition to the reasons that were already mentioned in other comments/answers. Section 10.6 reviews two different ways of constructing the same $Z_2$ invariant, using the Pfaffians of two different quantities at different sets of points. Specifically, they define one matrix $m(\mathbf{k})$ that is antisymmetric for all $\mathbf{k}$ in the Brillouin zone (BZ), and they define another matrix $B(\mathbf{k})$ that is antisymmetric only at time-reversal invariant points. According to section 10.6, the same $Z_2$ invariant may be expressed in either of two ways:

• The modulo-$2$ vorticity of the Pfaffian of $m$ in half the BZ. This seems to be the construction used in the original Kane-Mele paper.

• The product of the signs of the Pfaffian of $B$ at those points where $B$ is antisymmetric, where the sign is defined by comparing it to one consistent branch of the square-root of the determinant. In this construction, knowing only the determinant of $B$ at those same points is clearly not sufficient.

These two constructions of the $Z_2$ invariant are also reviewed in "Notes on topological insulators," https://arxiv.org/abs/1501.02874. This paper and section 10.4 in the Bernevig and Hughes text both indicate that the second construction, the product-of-signs version, can also be applied in higher dimensions (not just 2d).

So instead of this:

And who doesn't prefer determinants to Pfaffians?

maybe Kane and Mele were thinking this:

And who doesn't prefer Pfaffians to determinants?

...because sometimes it does matter, and when it does, the Pfaffian is more informative.