Why aren't tangent spaces simply defined as vector spaces with same dimension as the manifold?

We don't just want to have a vector space to call the "tangent space". We want to do geometric things with the tangent space, and we can't do those things if it's just an arbitrary vector space of the right dimension. We need it to more specifically be a vector space that encodes the parts of the geometry of our manifold that we care about.

For instance, here's one geometric thing we'd like to be able to do: given a smooth curve $\gamma:\mathbb{R}\to M$ on a manifold $M$, we'd like to define a "tangent vector" at each point of the curve. For each $t\in\mathbb{R}$, this should give us a vector "$\gamma'(t)$" which is in the tangent space of $M$ at the point $\gamma(t)$. If the tangent space is just defined as some arbitrary $n$-dimensional vector space, there isn't going to be any natural way to define $\gamma'(t)$. But for the usual definition, there is: we just take the equivalence class of the curve $\gamma$.

Another very useful thing we like to do is differentiate functions between manifolds: if $f:M\to N$ is a smooth function between manifolds and $p\in M$, there should be a "derivative" $df$ which is a linear map from $T_pM\to T_{f(p)}N$. Again, there's not any natural way to get such a map if the tangent spaces are just some arbitrary vector spaces.

To put it another way, the tangent space to a manifold at a point is not merely a vector space. It is a vector space together with a bunch of extra structure relating it to other geometric features of the manifold. So we care about it not just as a vector space up to isomorphism, but as a "vector space plus extra structure" up to isomorphism. It's possible to axiomatize this extra structure, and then you could define the tangent space as just any abstract gadget satisfying those axioms, rather than restricting yourself to the usual definition by equivalence classes of curves. This isn't commonly done though because (as far as I know) there isn't any nice axiomatization that isn't essentially just saying "a vector space equipped with an isomorphism to this particular concrete construction", so you might as well just use the concrete construction itself.

If they are “just vector spaces” at each point, then every set would be a manifold.

The whole point is to link the topological structure of the manifold (using coordinate patches that fit together well) to linear algebra that goes on in the tangent spaces.

Another reason to give many different definitions is that they all expose some aspect of what a manifold does. In particular, I’ve found D. Holm’s chapter on manifolds in Geometric mechanics and symmetry very helpful, as it relates several equivalent definitions.

Finally, another contribution to your angst might be the books you’re reading. Some math expositions by physicists can be extremely frustrating for mathematicians, due to different styles in the two disciplines (to put it charitably), so if you’re reading books by such authors it may contribute to what you’re describing.