Why am I getting very high ping from this IP, but others aren't?

A friend of mine lives in the same city as me (less than a mile away), and hires the same internet service provider as me. When he pings that IP, he gets an average latency of 40ms. That indicates it's not an issue with my ISP, and not a geographical issue either

This doesn't matter, he could be routed through a different CO, or he could be on a different ATM or Trunk that has lower latency or some routing issues. If you have XO, you almost certainly have routing issues.

All computers in my home network suffer from this issue. I tried it on both wired and wireless computers. That indicates it's not an issue with my PC.

This is pretty standard as all the computers in your house will take the same route.

Like already suggested, run a test, traceroute is not a good idea, it won't tell you anything because you know the route completes and it really only gives you a brief snapshot, what you need is MTR or pathping to tell you the latency at each hop over a period of time. This will produce much more accurate results. If your ISP is peering with the network that is returning high ping times or it is on their network, they can redirect you (although it might take stubbornness on your part to get them to do it). If it is off-net for them, you're out of luck.