Delete all files starting with a question mark

No need for any fancy stuff. Simply escape the ? so that it's not considered part of the glob:

rm -f ./\?*

This works for ! too:

rm -f ./\!*

Or in one fell swoop:

rm -f ./{\?,\!}*

Update

Just noticed that you were suggesting to grep the output of ls. I wanted to bring your attention to the fact that you shouldn't parse the output of ls

In my case, the characters were not really question marks, but unicode characters that apparently could not be displayed in my console.

Using rm -i * worked for me. If you don't want to do this, you can also delete by inode, as described at http://www.cyberciti.biz/tips/delete-remove-files-with-inode-number.html.

To find the inode, use:

ls -il

Then do:

find . -inum [inode-number] -exec rm -i {} \;