Why doesn't an inductor have polarity?

We only care about the polarity of an inductor when the polarity of the magnetic field is important.

Such as with electromagnets or common mode chokes and measurement transformers.

You might care. For example, if you have a filter choke that has a permanent magnet biasing it.

Or if there is another inductor magnetically coupled to it, then the relative polarity matters.