Then I get a report of any unusual disk or network activity that occurred. For example I don't expect opening a RAR file to cause network activity[1]
We're happy to pay what it takes for this kind of thing, and I suspect a lot of people are. Is there such a service out there?
[1] https://digital.nhs.uk/cyber-alerts/2023/cc-4370
If you can chose your system QubesOS may be interesting. It would be interesting if Qubes developed the capability to have non local VMs transparently (they wouldn't necessarily need to be virtualized).
i've never used it, but i've been meaning to check it out. at least it should give you a jumping off point for further investigation.
if that is insufficient, use proofpoint.
for archives that are tickling bugs, you have to use a similar technique. it's not enough to analyze them and send them on as-is. you have to unpack in a sandbox (which will be detectable, no 2 ways about it, but the question is will anyone expend enough effort to detect -- no, not for your use case, seeing as how you're asking the question at all), process with dangerzone or dangerzone-like tool, then re-archive it and let the user see only that new archive.
You can use a VM, but that’s a pain.