How to work with surrogate pairs in Python?

You've mixed a literal string \ud83d in a json file on disk (six characters: \ u d 8 3 d) and a single character u'\ud83d' (specified using a string literal in Python source code) in memory. It is the difference between len(r'\ud83d') == 6 and len('\ud83d') == 1 on Python 3.

If you see '\ud83d\ude4f' Python string (2 characters) then there is a bug upstream. Normally, you shouldn't get such string. If you get one and you can't fix upstream that generates it; you could fix it using surrogatepass error handler:

>>> "\ud83d\ude4f".encode('utf-16', 'surrogatepass').decode('utf-16')
''

Python 2 was more permissive.

Note: even if your json file contains literal \ud83d\ude4f (12 characters); you shouldn't get the surrogate pair:

>>> print(ascii(json.loads(r'"\ud83d\ude4f"')))
'\U0001f64f'

Notice: the result is 1 character ( '\U0001f64f'), not the surrogate pair ('\ud83d\ude4f').

Because this is a recurring question and the error message is slightly obscure, here is a more detailed explanation.

Surrogates are a way to express Unicode code points bigger than U+FFFF.

Recall that Unicode was originally specified to contain 65,536 characters, but that it was soon found that this was not enough to accommodate all the glyphs of the world.

As an extension mechanism for the (otherwise fixed-width) UTF-16 encoding, a reserved area was set up to contain a mechanism for expressing code points outside the Basic Multilingual Plane: Any code point in this special area would have to be followed by another character code from the same area, and together, they would express a code point with a number larger than the old limit.

(Strictly speaking, the surrogates area is divided into two halves; the first surrogate in a pair needs to come from the High Surrogates half, and the second, from the Low Surrogates. Confusingly, the High Surrogates U+D800-U+DBFF have lower code point numbers than the Low Surrogates U+DC00-U+DFFF.)

This is a legacy mechanism to support the UTF-16 encoding specifically, and should not be used in other encodings; they do not need it, and the applicable standards specifically say that this is disallowed.

In other words, while U+12345 can be expressed with the surrogate pair U+D808 U+DF45, you should simply express it directly instead unless you are specifically using UTF-16.

In some more detail, here is how this would be expressed in UTF-8 as a single character:

0xF0 0x92 0x8D 0x85

And here is the corresponding surrogate sequence:

0xED 0xA0 0x88
0xED 0xBD 0x85

As already suggested in the accepted answer, you can round-trip with something like

>>> "\ud808\udf45".encode('utf-16', 'surrogatepass').decode('utf-16').encode('utf-8')
b'\xf0\x92\x8d\x85'

Perhaps see also http://www.russellcottrell.com/greek/utilities/surrogatepaircalculator.htm