How do I delete a goal in Google Analytics?

It seems that you can't permanently delete goals in GA, the only action you can take is to deactivate them.

For the time being you can request to add that delete feature here "Google bugs project"

Some people recommend to add goals to separate profile than the main one, so when you are done with them you can easily delete the whole goals' profile without losing the collected data.