Self-hosted VoIP server for private/covert conversations

Whenever there are discussions about anonymous or private communication channels several apps with E2E encryption pop up but I've never seen anyone suggest simply using a self-hosted PBX/SIP Trunk.

I think it is pretty easy to spin-up an Asterisk server with a let's encrypt certificate for SIP/TLS and SRTP support and parties in need of secure communication simply need to call each other using a sip client (android has a built-in one).

A part from the requirement to harden the server, am I overseeing something? Is SRTP not that secure?