How to escape & in scp

Escaping both the spaces and the ampersand did the trick for me :

scp source_file "user@host:/dir\ with\ spaces\ \&\ ampersand"

The quotes are still needed for some reason.