addition of a new network protocol in the linux kernel

To handle communication from userspace to your protocol, register your protocol with the kernel sockets API. This allows you to create a normal socket from userspace.

Have a look at the bluetooth/RFCOM socket implementation for relevant code samples.

static const struct proto_ops rfcomm_sock_ops = {
     .family         = PF_BLUETOOTH,
     .owner          = THIS_MODULE,
     .bind           = rfcomm_sock_bind,
     .connect        = rfcomm_sock_connect,
     .listen         = rfcomm_sock_listen,
     .
     .
     .
     .accept         = rfcomm_sock_accept,

};
 
static const struct net_proto_family rfcomm_sock_family_ops = {
     .family         = PF_BLUETOOTH,
     .owner          = THIS_MODULE,
     .create         = rfcomm_sock_create
};

---

To register a protocol you will have to fill the proto_ops structure. This structure follows the object oriented pattern observed elsewhere inside the kernel. This structure defines an interface to follow for developers implementing their own socket interface.

Implement the functions the interface defines such as bind, connect, listen, and assign the function pointer to the structure entry. Define ioctl's for functionality not covered by the operations interface.

You end up with a structure that later you embed at the socket struct we return from the create function.

Struct net_proto_family defines a new protocol family. This structure includes the create function where your function implementation should populate a socket struct filled with the proto_ops struct.

After that register the family with sock_register, and if everything is ok you should be able to create a proper socket from userspace.

Internally the protocol should probably use skbuffs (see here, and here) to communicate with the networking devices.

skbuffs are the universal way of handling network packets in the linux kernel. The packets are received by the network card, put into some skbuffs and then passed to the network stack, which uses the skbuff all the time.

This is the basic data structure and io path to implement a networking protocol inside the linux kernel.

I am not aware of a document that describes this procedure from start to finish. The source is with you on this one.