Python - How to split a string by non alpha characters

The two options mentioned by others that are best in my opinion are re.split and re.findall:

>>> import re
>>> re.split(r'\W+', '#include "header.hpp"')
['', 'include', 'header', 'hpp', '']
>>> re.findall(r'\w+', '#include "header.hpp"')
['include', 'header', 'hpp']

A quick benchmark:

>>> setup = "import re; word_pattern = re.compile(r'\w+'); sep_pattern = re.compile(r'\W+')"
>>> iterations = 10**6
>>> timeit.timeit("re.findall(r'\w+', '#header foo bar!')", setup=setup, number=iterations)
3.000092029571533
>>> timeit.timeit("word_pattern.findall('#header foo bar!')", setup=setup, number=iterations)
1.5247418880462646
>>> timeit.timeit("re.split(r'\W+', '#header foo bar!')", setup=setup, number=iterations)
3.786440134048462
>>> timeit.timeit("sep_pattern.split('#header foo bar!')", setup=setup, number=iterations)
2.256173849105835

The functional difference is that re.split keeps empty tokens. That’s usually not useful for tokenization purposes, but the following should be identical to the re.findall solution:

>>> filter(bool, re.split(r'\W+', '#include "header.hpp"'))
['include', 'header', 'hpp']

You can do that with a regex. However, you can also use a simple while loop.

def splitnonalpha(s):
   pos = 1
   while pos < len(s) and s[pos].isalpha():
      pos+=1
   return (s[:pos], s[pos:])

Test:

>>> splitnonalpha('#include"blah.hpp"')
('#include', '"blah.hpp"')

Your instinct on using regex is correct.

import re
re.split('[^a-zA-Z]', string_to_split)

The [^a-zA-Z] part means "not alphabetic characters".