Convert UTF-16 to UTF-8 and remove BOM?
This is the difference between UTF-16LE
and UTF-16
UTF-16LE
is little endian without a BOMUTF-16
is big or little endian with a BOM
So when you use UTF-16LE
, the BOM is just part of the text. Use UTF-16
instead, so the BOM is automatically removed. The reason UTF-16LE
and UTF-16BE
exist is so people can carry around "properly-encoded" text without BOMs, which does not apply to you.
Note what happens when you encode using one encoding and decode using the other. (UTF-16
automatically detects UTF-16LE
sometimes, not always.)
>>> u'Hello, world'.encode('UTF-16LE')
'H\x00e\x00l\x00l\x00o\x00,\x00 \x00w\x00o\x00r\x00l\x00d\x00'
>>> u'Hello, world'.encode('UTF-16')
'\xff\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00w\x00o\x00r\x00l\x00d\x00'
^^^^^^^^ (BOM)
>>> u'Hello, world'.encode('UTF-16LE').decode('UTF-16')
u'Hello, world'
>>> u'Hello, world'.encode('UTF-16').decode('UTF-16LE')
u'\ufeffHello, world'
^^^^ (BOM)
Or you can do this at the shell:
for x in * ; do iconv -f UTF-16 -t UTF-8 <"$x" | dos2unix >"$x.tmp" && mv "$x.tmp" "$x"; done
Just use str.decode
and str.encode
:
with open(ff_name, 'rb') as source_file:
with open(target_file_name, 'w+b') as dest_file:
contents = source_file.read()
dest_file.write(contents.decode('utf-16').encode('utf-8'))
str.decode
will get rid of the BOM for you (and deduce the endianness).