I got inspiration from this (https://github.com/corsix/amx) and I wondered if someone has done it before on a Linux-based system. I understand a userspace library could be created to access those instructions from userspace, but how would then they be implemented in the kernel? Through a proprietary kernel module built using a custom compiler? Or is that not needed at all and the library could just run on the processor taking advantage of the proprietary extensions?
As you said, it's possible to use instructions in userspace that the kernel doesn't know about; in general the kernel doesn't care what goes on in userspace. One twist is that the kernel needs to know about processor registers so that it can save/restore them during a context switch. If an ISA extension introduces new registers (e.g. matrix registers) then the kernel needs to know about them.
Many vendors are complying with the Linux license fairly loosely if at all. Maybe they have a modified kernel build and they never publish the source. Maybe they only give the source to people who ask, and the source has no comments and it requires a secret compiler that is not available.