Disable COD if subtotal is greater then xxxx amount

Go to System > configuration > Sales > Payment methods > select Cash on delivery.

you can see "Minimum order total" field, just enter xxxx , it will not allow to checkout if the amount is greater than xxxx