Confused about backslashes in regular expressions

An r character before the regular expression in a call to search() specifies that the regular expression is a raw string. This allows backslashes to be used in the regular expression as regular characters rather than in an escape sequence of characters. Let me explain ...

Before the re module's search method processes the strings that are passed to it, the Python interpreter takes an initial pass over the string. If there are backslashes present in a string, the Python interpreter must decide if each is part of a Python escape sequence (e.g. \n or \t) or not.

Note: at this point Python does not care whether or not '\' is a regular expression meta-character.

If the '\' is followed by a recognized Python escape character (t,n, etc.), then the backslash and the escape character are replaced with the actual Unicode or 8-bit character. For example, '\t' would be replaced with the ASCII character for tab. Otherwise it is passed by and interpreted as a '\' character.

Consider the following.

>>> s = '\t'
>>> print ("[" + s  + "]")
>>> [       ]           // an actual tab character after preprocessing

>>> s = '\d'
>>> print ("[" + s  + "]")
>>> [\d]                // '\d' after preprocessing

Sometimes we want to include in a string a character sequence that includes '\' without it being interpreted by Python as an escape sequence. To do this we escape the '\' with a '\'. Now when Python sees '\' it replaces the two backslashes with a single '\' character.

>>> s = '\\t'
>>> print ("[" + s  + "]")
>>> [\t]                // '\t' after preprocessing

After the Python interpreter take a pass on both strings, they are passed to the re module's search method. The search method parses the regular expression string to identify the regular expression's meta-characters.

Now '\' is also a special regular expression meta-character and is interpreted as one UNLESS it is escaped at the time that the re search() method is executed.

Consider the following call.

>>> match = re.search('a\\t','a\\t')        //Match is None

Here, match is None. Why? Lets look at the strings after the Python interpreter makes its pass.

String 1: 'a\t'
String 2: 'a\t' 

So why is match equal to None? When search() interprets String 1, since it is a regular expression, the backslash is interpreted as a meta-character, not an ordinary character. The backslash in String 2 however is not in a regular expression and has already been processed by the Python interpreter, so it is interpreted as an ordinary character.

So the search() method is looking for 'a escape-t' in the string 'a\t' which are not a match.

To fix this we can tell the search() method to not interpret the '\' as a meta-character. We can do this by escaping it.

Consider the following call.

>>> match = re.search('a\\\\t','a\\t')          // Match contains 'a\t'

Again, lets look at the strings after the Python interpreter has made its pass.

String 1: 'a\\t'
String 2: 'a\t'

Now when the search() method processes the regular expression, it sees that the second backslash is escaped by the first and should not be considered a meta-character. It therefore interprets the string as 'a\t', which matches String 2.

An alternate way to have search() consider '\' as a character is to place an r before the regular expression. This tells the Python interpreter to NOT preprocess the string.

Consider this.

>>> match = re.search(r'a\\t','a\\t')           // match contains 'a\t'

Here the Python interpreter does not modify the first string but does process the second string. The strings passed to search() are:

String 1: 'a\\t'
String 2: 'a\t'

As in the previous example, search interprets the '\' as the single character '\' and not a meta-character, thus matches with String 2.

The confusion is due to the fact that the backslash character \ is used as an escape at two different levels. First, the Python interpreter itself performs substitutions for \ before the re module ever sees your string. For instance, \n is converted to a newline character, \t is converted to a tab character, etc. To get an actual \ character, you can escape it as well, so \\ gives a single \ character. If the character following the \ isn't a recognized escape character, then the \ is treated like any other character and passed through, but I don't recommend depending on this. Instead, always escape your \ characters by doubling them, i.e. \\.

If you want to see how Python is expanding your string escapes, just print out the string. For example:

s = 'a\\b\tc'
print(s)

If s is part of an aggregate data type, e.g. a list or a tuple, and if you print that aggregate, Python will enclose the string in single quotes and will include the \ escapes (in a canonical form), so be aware of how your string is being printed. If you just type a quoted string into the interpreter, it will also display it enclosed in quotes with \ escapes.

Once you know how your string is being encoded, you can then think about what the re module will do with it. For instance, if you want to escape \ in a string you pass to the re module, you will need to pass \\ to re, which means you will need to use \\\\ in your quoted Python string. The Python string will end up with \\ and the re module will treat this as a single literal \ character.

An alternative way to include \ characters in Python strings is to use raw strings, e.g. r'a\b' is equivalent to "a\\b".