what is the difference between `>> /dev/stderr` (with the white space) and `>&2`?

>& n is shell syntax to directly duplicate a file descriptor. File descriptor 2 is stderr; that's how that one works. You can duplicate other file descriptors as well, not just stderr. You can't use append mode here because duplicating a file descriptor never truncates (even if your stderr is a file) and >& is one token, that's why you can't put a space inside it—but >& 2 works.

>> name is a different permitted syntax, where name is a file name (and the token is >>). In this case, you're using the file name /dev/stderr, which by OS-specific handling (on Linux, it's a symlink to /proc/self/fd/2) also means standard error. Append and truncate mode both wind up doing the same thing when stderr is a terminal because that can't be truncated. If your standard error is a file, however, it will be truncated:

anthony@Zia:~$ bash -c 'echo hi >/dev/stderr; echo bye >/dev/stderr' 2>/tmp/foo
anthony@Zia:~$ cat /tmp/foo
bye

If you're seeing an error with /dev/stderr over ssh, it's possible the server admin has applied some security measure preventing that symlink from working. (E.g., you can't access /proc or /dev). While I'd expect either to cause all kinds of weird breakage, using the duplicate file descriptor syntax is a perfectly reasonable (and likely slightly more efficient) approach. Personally I prefer it.

The failure cases occur because the bash syntax for using & in redirections specifies a single >, and requires that it exists directly adjacent to the & sign:

[n]>&word

Use '>' to redirect (truncates if exists) or '>>' (appends if exists).

Use '>&' to duplicate a stream, for instance, if you want standard output AND standard error in the same file, you redirect to file '> output.log' and also error with '2>&'

myjob.sh > output.log 2>&