Why is a polar cone a closed set?

Actually, your argument works: $X^0$ is closed because it can be expressed as an intersection of closed sets.

if is infinite we can't conclude that the intersection of closed sets is also a closed set (as far as we are talking in terms of usual topology).

Yes you can. One of the axioms of topology is that any union of open sets is open. (This is easy to show directly for the standard topology on a metric space).

Taking complements, you get that any intersection of closed sets is closed.