How might a Mac hacker get keyboard and mouse access?

Recently I returned to my computer after a 3 minute break and I observed a third party moving my mouse and clicking around.

I assume this means they had some sort of screen access?

My question is: What sort of things do I need to consider when assessing how they got access? What would it take? How did I slip up?

I run Little Snitch, and am fairly aware of apps making calls.

I update /etc/hosts regularly with malware and spam lists (eg: from https://urlhaus.abuse.ch)

All the "sharing" options in MacOS are disabled, like file sharing and screen sharing. No remote login or management. Nothing as full disk access. No unexpected login items.

I have a reputable VPN which also offers basic malware and spam blocking. (I wont name the VPN, but it has a good reputation, like Mulvad)

I do use homebrew and npm, so no doubt crap gets installed there, but again Little Snitch asks me to wave them through each and every time I use them.

I don't bother with virus scanners, and especially avoid any type of "remote help".

My modem firmware is up to date. An older 2022 DLink modem with no fancy "network scan" or "parental control" or "remote access" etc.

Noone is sharing my wifi. No mystery devices on my network. WPA3 only.

I just can't get my brain around how someone could possibly remotely observe and control my computer like that.

In no way am I an expert, but I am moderately savvy - but obviously prone to hacks!

It's really hard to find examples and explanations of other peoples experiences... need to learn from them!