vim - How to escape filename containing single and double quotes mix?

Building on Wildcard's answer using filename modifiers, :S gives you exactly what you want. According to the docs (:h %:S),

:S      Escape special characters for use with a shell command (see
        |shellescape()|). Must be the last one. Examples:
            :!dir <cfile>:S
            :call system('chmod +w -- ' . expand('%:S'))

To use your example:

$ touch '"I'\''m also a n00b.txt"'
$ ls
"I'm also a n00b.txt"

Then vim '"I'\''m also a n00b.txt"', and voilà:

:!ls %:S

"I'm also a n00b.txt"

The :S filename modifier is available in Vim 7.4.

From :help filename-modifiers:

The file name modifiers can be used after "%", "#", "#n", "<cfile>", "<sfile>",
"<afile>" or "<abuf>".  ...

...
    :s?pat?sub?
            Substitute the first occurrence of "pat" with "sub".  This
            works like the |:s| command.  "pat" is a regular expression.
            Any character can be used for '?', but it must not occur in
            "pat" or "sub".
            After this, the previous modifiers can be used again.  For
            example ":p", to make a full path after the substitution.
    :gs?pat?sub?
            Substitute all occurrences of "path" with "sub".  Otherwise
            this works like ":s".

So rather than just handling double quotes or single quotes, let's just backslash escape everything unusual:

:!ls -l %:gs/[^0-9a-zA-Z_-]/\\&/

Works perfectly with the test filename you provided.

To use an absolute path, which you may want for rsync, you can add :p at the end:

:!ls -l %:gs/[^0-9a-zA-Z_-]/\\&/:p

---

Actually, it also works just fine if you backslash-escape literally every character, and it's shorter to type:

:!ls -l %:gs/./\\&/:p

So, in your rsync command, instead of %, use %:gs/./\\&/:p.