If I load a kernel module and list the loaded modules with
rmmod has a
--force parameter. If you know the stuff your module does, and have a kernel configured to support forcing unload, that might work. That's to save you from having to restart until you fixed the problem with the ref-counting), but it won't show you the cause of the load of your module. I think it's not possible to get why the module was loaded in the first place (i.e which exact code-path). Maybe
dmesg has logged something useful or you can add some useful logging into your module code.