How do I clear out the ssh-agent entries (on Mac OS X )?

Your SSH keys should not get automatically added to the agent just because you SSH'ed to a server...

Run ssh-add -l to list the agent's keys, ssh-add -D to clean out all keys.

Another way to limit what keys are checked against is to add IdentitiesOnly yes to /.ssh/config and a list of entries

IdentityFile ~/.ssh/<nameofkey>

You may also add them to each Host entry:

Host server-name HostName <ipaddress> User <username> IdentityFile ~/.ssh/<nameofkey>

SSH Man Page has more options as well.