How do I configure custom URL handlers on OS X?

The file you seek is ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure.plist.

It holds an array called LSHandlers, and the Dictionary children that define an LSHandlerURLScheme can be modified accordingly with the LSHandlerRole.

Rather than manually editing this file, you can also use RCDefaultApp, which gives you a nice Preference Pane. It is said not to be working for macOS 10.12 and up, but I can confirm it runs under macOS 10.14.

An alternative is SwiftDefaultApps.

For example, here's SVN and SSH:

… and in RCDefaultApp:

Update Jul 2017: RCDefaultApp and lstool no longer work on macOS 10.12 or later. We'll have to find a new solution.

Edit Mar 2018
I found SwiftDefaultApps but can't test as I'm not on High Sierra yet.

---

Furthering the previous answer, if you would like to do this on the command line you can use the lstool command, which is the core of RCDefaultApp, found under RCDefaultApp.prefPane/Contents/Resources/lstool.

Its usage is straightforward:

[~]$lstool --help
Usage:

lstool read [<lsscheme> [<type>]]
lstool [-n] write <lsscheme> <type> <app>
lstool [-n] setoption <lsscheme> <type> login|ignorecreator YES|NO
lstool [-n] register {<app>}
lstool [-n] unregister {<path> | <app>}
lstool apps

-n means do not make changes
<lsscheme> is one of: internet, media, url, extension, uti, mime, ostype
<app> is the path to an application or a name to be looked up

Listing current LaunchServices URL handler settings on Apple OS X 10.10 (Yosemite):

defaults read com.apple.LaunchServices/com.apple.launchservices.secure