Is Minkowski space usually a vector space or an affine space?

Physically speaking, there is no preferred origin in the spacetime of special relativity, therefore an affine space (equipped with a Lorentzian scalar product) is a better model than a vector space. The Lorentz group acts on the tangent space at each event, this space being isomorphic to the space of four-displacements. The whole invariance group is the Poincare' one which includes translations.

The confusion arises as in many cases no distinction is made between the manifold of space-time and its tangent bundle, or even a characteristic fibre of it. Flat space-time is assumed to be a pseudo-Riemannian manifold $M$ with a pseudo-metric tensor $\eta\in T^*M^{\otimes 2}$ defined everywhere. The Minkowski vector space is any fibre $T_pM$, which is usually described with orthogonal inertial frames. Such frames are linked by transformations of the group $O(1,3)$, i.e. the full Lorentz group. The metric structure on $M$ can be used to turn the manifold $M$ itself into an affine space, since lengths in this space are translation-invariant. Hence the full isometry group of the space-time manifold $M$ is not just the full Lorentz group $O(1,3)$ but the larger Poincaré group $O(1,3)\ltimes\mathbb R^4$, i.e. the semidirect product with the translation group $\mathbb R^4$.

If I understood you correctly, you want to know the difference between the two.

Take a look at this question (and answer): What are differences between affine space and vector space?

Short answer: the only difference is that affine spaces don't have a special $\vec{0}$ element. But there is always an isomorphism between an affine space with an origin and the corresponding vector space.

In this sense, Minkowski space is more of an affine space. But you still can think of it as a vector space with a special 'you' point.