iTerm 2 profiles

The latest iTerm2 nightly (Build 2.9.20150329-nightly at the time of writing) allows you to do that easily. You can download it here.

Once you've installed and opened it:

1. Log in to your remote machine via ssh and click iTerm2 (the app menu) > Install Shell Integration. It will download a script with curl and install it. Do the same on your local machine.
2. Go to Preferences > Profiles.
3. Create a new profile for your local machine. Customize it to fit your needs (change background color, name, etc)
4. Go to the Advanced tab and scroll to the bottom.
5. In Automatic Profile Switching, click '+' and add the hostname of your local machine. The hostname is the one you get when running echo $HOST on the target machine. It is not always the one you see in your prompt.
6. Create another profile, this time for your remote machine, and customize it.
7. Add the hostname of the remote machine in Automatic Profile Switching.
8. Now, if you ssh into the remote machine, your profile will change, and if you exit out of the ssh session, you will be back to your local profile.

You can combine this solution with @esod's answer seamlessly.
Note: it didn't work for me until I created a profile specifically for the desktop instead of using the default profile.

See the documentation for more info.

for zsh users

lets say you have 2 profiles, one named mac (for your primary machine) and one for linux (your remote machine)

when entering the session, we need to tell zsh to load our profile

1. connect to remote linux
2. in ~/.zshrc add echo -e "\033]50;SetProfile=linux\a"
3. source your files for immediate effect: source ~/.zshrc
4. your new theme should be visible within the iterm session.

when exiting the session, we need to tell zsh to switch back to our original profile

1. connect to remote linux
2. in linux ~/.zlogout add the following

if [ "$SHLVL" = 1 ]; then
  echo -e "\033]50;SetProfile=mac\a"
  clear
fi

now you can swap profiles with ease <3.

if you are using bash, i believe the steps are similar but you would instead modify ~/.bashrc and ~/.bash_logout

demo

iTerm2 supports a custom escape code that changes the profile on the fly. Put it in your .bashrc or .bash_profile.

<esc>]50;SetProfile=X^G

where X is the profile. For instance, to change the profile to one called "Foo", us this shell script:

#!/bin/bash
echo -e "\033]50;SetProfile=Foo\a"

To change it back when you log out, put code to change the profile back to default in ~/.bash_logout.