How can I limit the bandwidth used by a process?

One option that I just discovered is to use trickle.

trickle is a portable lightweight userspace bandwidth shaper. It can run in collaborative mode (together with trickled) or in stand alone mode.

trickle works by taking advantage of the unix loader preloading. Essentially it provides, to the application, a new version of the functionality that is required to send and receive data through sockets. It then limits traffic based on delaying the sending and receiving of data over a socket. trickle runs entirely in userspace and does not require root privileges.

You can use iptables to mark a packet (--pid-owner ...), then use tc to shape the traffic. Also "--sid-owner" can be used to include threads and children of that process.

http://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#OWNERMATCH

Match --pid-owner
Kernel 2.3, 2.4, 2.5 and 2.6
Example iptables -A OUTPUT -m owner --pid-owner 78
Explanation This match is used to match packets based on the Process ID (PID) that was responsible for them. This match is a bit harder to use, but one example would be only to allow PID 94 to send packets from the HTTP port (if the HTTP process is not threaded, of course). Alternatively we could write a small script that grabs the PID from a ps output for a specific daemon and then adds a rule for it. For an example, you could have a rule as shown in the Pid-owner.txt example

If you can write to a pipe (or stdout), you can install the pv (pipe viewer) command. It was originally written to display the progress of data transferred through a pipe.

tar cvf - /files/to/backup | pv -L 512k > /your/file/on/sshfs

   -L RATE, --rate-limit RATE
          Limit the transfer to a maximum of RATE  bytes  per  second.   A
          suffix of "k", "m", "g", or "t" can be added to denote kilobytes
          (*1024), megabytes, and so on.