how to create a custom file extension in C#?

As a deployment point, you should note that ClickOnce supports file extensions (as long as it isn't in "online only" mode). This makes it a breeze to configure the system to recognise new file extensions.

You can find this in project properties -> Publish -> Options -> File Associations in VS2008. If you don't have VS2008 you can also do it manually, but it isn't fun.

File extensions are an arbitrary choice for your formats, and it's only really dependent on your application registering a certain file extension as a file of a certain type in Windows, upon installation.

Coming up with your own file format usually means you save that format using a format that only your application can parse. It can either be in plain text or binary, and it can even use XML or whatever format, the point is your app should be able to parse it easily.

There are two possible interpretations of your question:

What should be the file format of my documents?

You are saving currently your notes in the RTF format. No matter what file name extension you choose to save them as, any application that understands the RTF format will be able to open your notes, as long as the user knows that it's in RTF and points that app to that file.

If you want to save your documents in a custom file format, so that other applications cannot read them. you need to come up with code that takes the RTF stream produced by the Rich Edit control (I assume that's what you use as editor in your app) and serializes it in a binary stream using your own format.

I personally would not consider this worth the effort...

What is the file name extension of my documents

You are currently saving your documents in RTF format with .rtf file name extension. Other applications are associated with that file extension, so double-clicking on such file in Windows Explorer opens that application instead of your.

If you want to be able to double click your file in Windows Explorer and open your app, you need to change the file name extension you are using AND create the proper association for that extension.

The file extension associations are defined by entries in the registry. You can create these per-machine (in HKLM\Software\Classes) or per-user (in HKCU\Software\Classes), though per-machine is the most common case. For more details about the actual registry entries and links to MSDN documentation and samples, check my answer to this SO question on Vista document icon associations.