Rejecting a PhD offer after having accepted it. Advice?

I am assuming you are going to go to the better place, so asking our opinion is rhetorical.

The consequences depend on the people at the university: if they are reasonable folks, they should understand. But you never know.

You should do your best to make it as painless as possible FOR THEM. My suggestion is that you should decide where you want to go, and accept there. And the AS SOON AS POSSIBLE, send a very very nice and very very apologetic letter to the person you had said yes to, explaining that you have decided to go elsewhere. The key here is "as soon as possible" so they have the opportunity to offer acceptance to someone else -- that will lesson any annoyance they have.

Most US universities with serious graduate programs are signatories to the Council of Graduate Schools Resolution Regarding Graduate Scholars, Fellows, Trainees and Assistants, which pertains to situtations in which graduate acceptance comes with an offer of TA and/or RA support. This resolution explicitly allows students to retract acceptances made prior to April 15th, but not after April 15th:

In those instances in which a student accepts an offer before April 15, and subsequently desires to withdraw that acceptance, the student may submit in writing a resignation of the appointment at any time through April 15. However, an acceptance given or left in force after April 15 commits the student not to accept another offer without first obtaining a written release from the institution to which a commitment has been made.

This is not to say that programs will be happy to hear that you have retracted your acceptance, only that you have every right to do so as long as you do this before April 15th.

Of course this a tremendous decision, and one that will likely stay with you for the rest of your life. Trust me, while the folks at this institution are likely of a particular kind of 'good' sort, I would bet a boatload of money that each of the committee members that accepted you would likely do what was best for them should they have ever found themselves in your shoes. The worst possible scenario is that if you've paid some sort of deposit, you will likely not see this returned. Also, in the future, should you seek employment at this institution, you might find yourself in a precarious situation should one of these committee members find themselves on a search committee for a position you're interested in (unlikely). Perhaps the most disheartening impact will be felt by way of having to explain to those you might have already told that you'd be attending this particular institution about your change of mind. Rest assured, they will not judge you, and if they are in fact true friends, family, or kin, they'll unconditionally support your decision. Hope this helps..