Adding custom Compose key sequences

You're almost there: the file is indeed ~/.XCompose. You can specify an alternate location with the XCOMPOSEFILE environment variable. See the Compose manual page.

The missing piece of the puzzle is that the file is read by each application when it starts. The Compose key is not handled by the X11 server, but by the X11 library. In theory, an application could detect that the file has been modified or offer a command to reload it but I don't know of any that does. You'll have to restart your applications. Start a new editor or terminal emulator for testing.