Process a file that starts with a BOM (FF FE)

From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8:

iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt

dos2unix also removes BOMs and converts UTF-16 to UTF-8:

$ printf %s あ|recode ..utf16 >a;xxd -p a;dos2unix a;xxd -p a
feff3042
dos2unix: converting file a to Unix format...
e38182

dos2unix also removes UTF-8 BOMs:

$ printf %b '\xef\xbb\xbfa'>a;dos2unix a;xxd -p a
dos2unix: converting file a to Unix format...
61

Related