How to remove a failed kernel module

Use a virtual machine.

Once you make a NULL dereference or other such mistake, you've put the kernel into an unknown state. Even if you did manage to remove the module (which is unlikely to be possible; a kernel OOPS kills the calling thread, so it'll never have a chance to reduce the reference count - the module will never be removable) there may still be corruption left behind, and your new, 'fixed' module is just as likely to be in trouble.

Much better to just use a fast-to-reboot virtual machine - perhaps with a snapshot, to make restoration even faster.

As bdonlan pointed out, you would be better off with a virtual machine.

However, if you really want to do it your way, you have to:

1. check that your kernel config has CONFIG_MODULE_FORCE_UNLOAD enabled
2. write another module that sets the refcount of the broken module to zero. Check out the delete_module system call in kernel/module.c