Why can we assume torsion is zero in GR?

In standard formulations of general relativity, it is simply an assumption of the theory designed so that the affine geodesics given by the connection match the metrical geodesics given by extremizing the spacetime interval.

The Levi-Cevita connection is the unique connection that is both torsion-free and metric-compatible, but for GTR only the torsion-free assumption is necessary. Through the Palatini action given by the Lagrangian $\mathscr{L}_G = \sqrt{-g}g^{ab}R_{ab}\text{,}$ the connection coefficients being symmetric is enough to derive that they are necessarily $$\Gamma^a_{bc} = \frac{1}{2}g^{ad}\left[g_{db,c}+g_{dc,b}-g_{bc,d}\right]\text{.}$$ The Palatini approach is discussed in some introductory textbooks, e.g., Ray d'Inverno's Introducting Einstein's Relativity, and as an exercise in Sean Carroll's Spacetime and Geometry.

Physically, the no-torsion assumption allows the metric to take the role of a potential for the "gravitational field" of connection coefficients.

But in the end it's just an assumption of the theory; if you don't take it, you're doing something else, such as Einstein-Cartan theory or teleparallel gravity. Interestingly, here's what Einstein had to say about the relationship between the connection and the metric around the time when he was working on teleparallelism:

... the essential achievement of general relativity, namely to overcome "rigid" space (i.e. the inertial frame), is only indirectly connected with the introduction of a Riemannian metric. The directly relevant conceptual element is the "displacement field" ($\Gamma^l_{ik}$), which expresses the infinitesimal displacement of vectors. ... This makes it possible to construct tensors by differentiation and hence to dispense with the introduction of "rigid" space (the inertial frame). In the face of this, it seems to be of secondary importance in some sense that some particular $\Gamma$ field can be deduces from a Riemannian metric ...

There are (at least) two approaches to torsion in the geometric framework behind general relativity:

First, we can use it to encode a new degree of freedom of the theory: The coupling of spin to the gravitational field. This is Einstein-Cartan theory, which is (as far as I'm aware) neither supported nor excluded by observational evidence.

Second, we can use it to encode the existing gravitational degrees of freedom. There's a sort of gauge symmetry between curvature and torsion. Gauge fixing torsion to 0, we end up with general relativity, whereas fixing curvature to 0, we end up with its teleparallel equivalent.