How can I trigger a notification from a remote terminal in OS X

You have not specified what kind of notification you want nor what OS the remote server is running so I am going to have to make some assumptions here. I will assume you don't really care what type of notification it is as long as you are notified and that the remote server is running some flavor of *nix.

1. Send yourself an email. If sendmail is configured on the server, you could do

   ./long_process && echo "Job done" | sendmail [email protected]
   

2. ssh back to your local machine (assuming this is possible) and make it talk to you. See here for more cool ways of making OSX beep at you.

   ./long_process && ssh [email protected] say "Yo! All done"
   

   or

   ./long_process && ssh [email protected] terminal-notifier -message "Job finished!" -title "Info"
   

   If you are connecting from a dynamic IP and you have configured your router so that you can ssh to that dynamic IP, you can do this (assuming you are only currently connected from your remote machine):

   ip=$(who | grep $USER | perl -lne 's/\\((.+?)\\)\s*$//; print "$1"' | tail -n 1) &&
    ./long_process && ssh you@$ip terminal-notifier -message "Job finished!" -title "Info"
   

   You can set up password-less ssh in the normal way. It should not be affected by the dynamic IP. Once you have done so, the code above will work.

3. Use pushover and send a notification to your Android or iOS device (if you have one)

   ./long_process && pushover.pl "All done"
   

You can echo a bell - \a in the remote terminal: ./long_process ; echo -e '\a'

Some terminal emulators will play some sort of sound and might show a notifications. iTerm2, in particular, will show a notification and play a sound, if process' tab is not focused.

Also, notice that I used ; to separate commands. This way you will be notify even when command returns non-zero status while && will only run if command succeeds.