Dealing with wrong spelling when matching text strings in R

I recommend you use Jaro-Winkler distance, a string similarity metric developed to solve this exact problem in US census data. It is more sophisticated than levenshtein distance and designed specifically for working with names. You can find an R implementation in the RecordLinkage package. You'll need to set a cut-off threshhold (e.g. 0.8) for how similar two strings must be.

install.packages('RecordLinkage','RSQLite')
require(RecordLinkage)

jarowinkler('William Clinton', "Willam Clntn")
# 0.96
jarowinkler('William Clinton', "Wuliam Clinton")
# 0.8462637
jarowinkler('William Clinton', "Hilary Clinton")
# 0.7790765

I'd recommend setting a reasonably high threshhold (maybe 0.9) for automated matching and then sending records below the high threshhold but above a secondary lower thresshhold (maybe 0.7) to human review. You should play with these numbers and see what works for you. These values will determine your sensitivity/specificity trade-off.


The spelling problems are commonly dealt with by using some variant of the soundex algorithm. There is an R implementation in RecordLinkage package. Then you need to compare not the strings themselves but their "phonetic codes":

> soundex('Clenton') == soundex('Clinton')
[1] TRUE

UPDATE: There is also another way of determining if two words are "close" to each other - which is a "distance" is some sense between the words. One standard measure of the distances is the minimum amount of one-letter replacements, deletions and insertions needed to transform the first word into the second one. It is called Levenshtein distance. RecordLinkage as well as vwr package have the appropriate functions:

> levenshteinDist('Clinton', 'Clenton')
[1] 1

> vwr::levenshtein.distance('Clinton', 'Clenton')
Clenton 
  1 

Then you can use the distances and consider the words "close" enough if the distance does not exceed some threshold.

UPDATE: soundexis also available in phonics package.