Many drivers are written in C, but you still need to know the underlying assembly code.
At one point, Windows split driver development into 2 types. One at the user level and one in the OS.
Rumor was this was because of printers. Most of the calls for support at MS (blue screen of death) were caused by printer drivers. By moving them out of the OS and into user space the drivers only crashed their application.
I recall there once was a USB driver development kit you could buy. Not sure if it was tied to an OS. But it was a physical device you would plug into a USB port. Cost was like $100-
Pretty much, yeah. There's a reason not everyone is sitting around writing hardware drivers.
The alternative is reading through Linux kernel code. However it typically requires you digging through many layers of abstractions to get a fuller picture.
Generally, x86 is generally more mature, and the specs tend to be fairly large. ARM documentation should be a little simpler to dig into.
If you're just intellectually interested, I'd suggest starting with that and then create some toy hardware project on an Arduino and create a USB driver for it.