I attended an Olympiad for student and failed. How should I write this down on my CV?

Firstly, you did not 'fail' - you were part of a team and competed - so congratulations on that.

Secondly, it can't hurt to include an entry in you CV under a heading like 'Other Achievements', something along the lines of:

(Year), Competed as part of (Team) for the Student Physics Olympiad.

If you have space, briefly describe positive aspects of your participation.

the fact that I had a chance to go to the competition is one of my competitive points. How can I write that down on my CV?

You almost wrote down exactly what you need.

• was selected to go to SPO, [...which is an accomplishment why?] an extremely selective program admitting only [top X% of, Y hundred, etc.] physics students nationwide

And it's always fine to say "placed in the top 1/2 of students" if true. Just from a resume writing standpoint anything worse than top 1/2 wouldn't add value to include.

As an aside, you are completely right that this is huge lauds. In high school I placed #8 in a similar state-wide competition. What if the top 10 were invited for an even more competitive competition and I came darn near in last? Don't fall for the Russian doll effect which more or less just tells you you're not #1 in the world (...but even the best undergraduate programs admit more than one physics student!) Obviously the admission would be an accomplishment.

Quantification is important. Numbers are best for a corporate resume, but perhaps just establishing the prestige of SPO is sufficient. Check their website for info on how prestigious it is to participate. Perhaps they have recognition from the U.S. President or something - whatever it is they'll be sure to brag about it somewhere. If the reviewer does not know of SPO, they will have no way to tell this isn't just an email list with 30 students submitting problems and answers with a leader board.