OS choice for High Performance Computing

For exploration purposes I would say OpenBSD to discover how stable, secure and simple it can be. The basics of Linux will be Slackware.

But this is in contradiction to your programming goals. So knowing about HP-Unix, AIX, Solaris, Linux Debian, RH and SuSE I would recommend using Linux CentOS 5.7 (or newer - keep it updated via yum-cron) to you.

CentOS 5 is binary compatible to RHEL 5 and there are many tools around for the el5-branch (use rpmforge as additional repository - see the CentOS page about that).