Not able to remove a loadable kernel module

Maybe you forget to provide module_exit, so your module doesn't know how to exit.

This only happens to me when there is a bug in my driver which is causing the code in the module to panic or crash in some way. In my experience once this happens reboot is the only possible course.

As I said, the kernel usually panics so you should check out dmesg after inserting it or running you application to exercise it. I'm not certain but it might be possible that if the driver doesn't release a mutex this behavior will happen as well.