how do I .decode('string-escape') in Python3?
You'll have to use unicode_escape
instead:
>>> b"\\123omething special".decode('unicode_escape')
If you start with a str
object instead (equivalent to the python 2.7 unicode) you'll need to encode to bytes first, then decode with unicode_escape
.
If you need bytes as end result, you'll have to encode again to a suitable encoding (.encode('latin1')
for example, if you need to preserve literal byte values; the first 256 Unicode code points map 1-on-1).
Your example is actually UTF-16 data with escapes. Decode from unicode_escape
, back to latin1
to preserve the bytes, then from utf-16-le
(UTF 16 little endian without BOM):
>>> value = b's\\000u\\000p\\000p\\000o\\000r\\000t\\000@\\000p\\000s\\000i\\000l\\000o\\000c\\000.\\000c\\000o\\000m\\000'
>>> value.decode('unicode_escape').encode('latin1') # convert to bytes
b's\x00u\x00p\x00p\x00o\x00r\x00t\x00@\x00p\x00s\x00i\x00l\x00o\x00c\x00.\x00c\x00o\x00m\x00'
>>> _.decode('utf-16-le') # decode from UTF-16-LE
'[email protected]'
The old "string-escape" codec maps bytestrings to bytestrings, and there's been a lot of debate about what to do with such codecs, so it isn't currently available through the standard encode/decode interfaces.
BUT, the code is still there in the C-API (as PyBytes_En/DecodeEscape
), and this is still exposed to Python via the undocumented codecs.escape_encode
and codecs.escape_decode
.
>>> import codecs
>>> codecs.escape_decode(b"ab\\xff")
(b'ab\xff', 6)
>>> codecs.escape_encode(b"ab\xff")
(b'ab\\xff', 3)
These functions return the transformed bytes
object, plus a number indicating how many bytes were processed... you can just ignore the latter.
>>> value = b's\\000u\\000p\\000p\\000o\\000r\\000t\\000@\\000p\\000s\\000i\\000l\\000o\\000c\\000.\\000c\\000o\\000m\\000'
>>> codecs.escape_decode(value)[0]
b's\x00u\x00p\x00p\x00o\x00r\x00t\x00@\x00p\x00s\x00i\x00l\x00o\x00c\x00.\x00c\x00o\x00m\x00'
If you want str-to-str decoding of escape sequences, so both input and output are Unicode:
def string_escape(s, encoding='utf-8'):
return (s.encode('latin1') # To bytes, required by 'unicode-escape'
.decode('unicode-escape') # Perform the actual octal-escaping decode
.encode('latin1') # 1:1 mapping back to bytes
.decode(encoding)) # Decode original encoding
Testing:
>>> string_escape('\\123omething special')
'Something special'
>>> string_escape(r's\000u\000p\000p\000o\000r\000t\000@'
r'\000p\000s\000i\000l\000o\000c\000.\000c\000o\000m\000',
'utf-16-le')
'[email protected]'