How can I escape characters in SQLite via bash shell?

The trouble with MarkusQ's solution is knowing which characters are special inside double quotes - there are quite a lot of them, including back-ticks, dollar-open parenthesis, dollar-variable, etc.

I would suggest it is better to enclose the string inside single quotes; then, each single quote inside the string needs to be replaced by the sequence quote, backslash, quote, quote:

sqlite3.bin contacts.db 'select * from contacts
      where source = "Nancy'\''s notes"'

The first quote in the replacement terminates the current single-quoted string; the backslash-quote represents a literal single quote, and the final quote starts a new single-quoted string. Further, this works with Bourne, Korn, Bash and POSIX shells in general. (C Shell and derivatives have more complex rules needing backslashes to escape newlines, and so on.)