What's the term for a "physical vector space"?

There probably isn't quite a term for this, but I agree that probably what you want to talk about is not naked vector spaces but vector spaces $V$ equipped with an action of a suitable group $G$, namely the group of "physically meaningful" symmetries. "Physical" means something like $V$ being an irreducible representation of $G$. In the case of physical space we might have $V = \mathbb{R}^3$ and $G = \text{SO}(3)$ (this representation is irreducible), for example, while in the case of something like fruits in a fruit basket we might have $V = \mathbb{R}^3$ and $G$ at best is some abelian group acting by scaling on each coordinate individually (this representation is reducible).

Inner products don't seem to play any role in this discussion though.

I think you're looking at the problem backwards. It's not that your "physical vector space" requires extra structure to distinguish it from the other types: it's that the extra distinguishing information belongs to the other types of vector spaces.

For example, your vector space of fruit (once you fix up the technicalities) has extra information: we attach significance to a particular basis of the vector space, so that the coordinates with respect to this basis pick out how many of each particular kind of fruit we have.

Aside: note that "coordinate with respect to this particular basis we have chosen" is a basis-independent notion. We can "mix" the coordinates of this vector space all we like, and it doesn't change the result when we do the calculation to extract the coordinates with respect to our chosen basis.

The same deal with your energy-temperature example; you have a vector space along with a significant basis for the vector.

Your "physical vector space", incidentally, doesn't allow freely mixing the coordinates either, since you attach significance to the values of an inner product.

Your intuition to look at a symmetry group is a reasonable one; one of the standard methods to look at structure is to think instead in terms of operations that preserve the structure. In each of the cases above, you can look at the group that preserves the form of the relevant structure. e.g.

• The symmetry group of your fruit vector space together with its canonical basis is the trivial group.
• The symmetry group of your energy-temperature space is the group $\mathbf{R}^* \times \mathbf{R}^*$, since you put significance on keeping the two coordinates separate, but you're probably not attaching significance to scale, so the symmetry group allows you to rescale the two coordinates independently
• The symmetry group of your "physical" vector space is $SO(3)$, since that preserves the inner product as well as orientation (which you probably find significant)

And the symmetry group of just a 3-dimensional vector space, of course, is $GL(3)$: the group of invertible $3 \times 3$ matrices.