Other than your everyday well-known antivirus techniques (e.g., looking at signatures of executables and comparing to a known list of viruses), are there any super reliable ways using low-level kernel operations to see any running executable/service that is accessing keyboard events?
Even if it's a big list, if you could eliminate most of the processes because they are known applications (say, by crowdsourcing a list of known safe processes), you could come up with a subset of them and decide for yourself if any look suspicious.
The best approach should be able to detect and stop even a custom tailored attack on a high value target, where none of the code is reused from known exploits, and every trick in the book is used to obfuscate and evade automated detection. I feel like an open source package like this that had been inspected by a lot of smart people and vetted to work reliably under Windows/Mac/Linux would be a huge benefit to global security.
In academics there is nothing really interesting about key loggers specifically. It's just a software, that reads memory.
You could whitelist processes, but the attacker will patch itself into whitelisted programs at runtime and run under their hood - low level kernel operations are the things, that an attacker ignores, when she already is a kernel module or sits in your ME. That's why Antivirus solutions are basically useless against a skilled attacker.
Perhaps a better way to approach this would be to turn the threat model around? If they have a keylogger, assume they have a rootkit. What needs to remain secure even when the platform is comprised?
Try to get it down to some cryptographic keys, which you can then store on a smartcard, that you use with a card reader with hardware PIN pad. There's no keylogging the PIN on one of those, as the PIN is sent directly to the smartcard, without relying on the host.
Obviously at that point the software asking you to sign something is probably compromised, but your cryptographic keys will remain protected, as will the PIN.