How do I remove an existing claim from a ClaimsPrinciple?

You should use identity to add or remove a claim. Try this to add a claim.

var user = User as ClaimsPrincipal;
var identity = user.Identity as ClaimsIdentity;
identity.AddClaim(new Claim(ClaimTypes.Role, "somenewrole"));

To remove a claim,

var user = User as ClaimsPrincipal;
var identity = user.Identity as ClaimsIdentity;
var claim = (from c in user.Claims
                         where c.Value == "somenewrole"
                         select c).Single();
identity.RemoveClaim(claim);

BTW, it is better to use User from your controller instead of HttpContext.Current.User.

Something else that is important to add is to make sure you do not try to iterate over the collection of claims and remove items. I just stumbled upon buggy code written by someone else, and at first I didn't see the problem until I stepped through it.

The buggy code was:

        foreach (var claim in identity.Claims)
        {
            var name = claim.Type;
            if (!name.Equals("UserAccountId") && !name.Equals("Email") && !name.Equals("TenantIds"))
            {
                identity.RemoveClaim(claim);
            }
        }

The result was that claims were inconsistently removed from the list. The simple solution to the problem is to iterate over a list of claims and not the claims themselves, and remove them that way:

        var claimNameList = identity.Claims.Select(x => x.Type).ToList();

        foreach (var name in claimNameList)
        {
            if (!name.Equals("UserAccountId") && !name.Equals("Email") && !name.Equals("TenantIds"))
            {
                var claim = identity.Claims.FirstOrDefault(x => x.Type == name);
                if (claim != null)
                    identity.RemoveClaim(claim);
            }
        }

It's never a good idea to iterate over a collection and add or remove items. You will see sporadic errors and different results depending on the situation, and in some circumstances, such as iterating over items in HttpContext.Current.Items, you will see sporadic errors about the collection being modified.