How do I remove a file I accidentally created named $file?

You can escape the $ with \:

rm \$file

Basically, if you want to do things literal with these weird characters, you need to escape it. In a shell there are several ways to do that. The first one is to prepend a '\' to every character you want to escape. So you can do rm \$file. Another way is to quote them with single quotes, for example, rm '$file' or rm '$'file. Some people also consider double quotes as a mean to "escape", but it only escapes white spaces. For example if you have a file named a file, you can do

rm a\ file

or

rm 'a file'

or

rm "a file"

You can also do

rm '$file'

Stuff in single quotes is taken as literal always,so globs and variables don't get expanded.