Remove Unicode characters from textfiles - sed , other Bash/shell methods

For UTF-8 encoding of Unicode, you can use this regular expression for sed:

sed 's/\xc2\x91\|\xc2\x92\|\xc2\xa0\|\xe2\x80\x8e//g'

Clear all non-ASCII characters of file.txt:

$ iconv -c -f utf-8 -t ascii file.txt
$ strings file.txt

Options:

-c # discard unconvertible characters
-f # from ENCODING
-t # to ENCODING

If you want to remove only particular characters and you have Python, you can:

CHARS=$(python -c 'print u"\u0091\u0092\u00a0\u200E".encode("utf8")')
sed 's/['"$CHARS"']//g' < /tmp/utf8_input.txt > /tmp/ascii_output.txt

Related