How do you keep your backup server safe?

I use duplicity on a few servers to encrypt backups and send them to a backup server, which then sends another copy to rsync.net, and then, once a week, give it or take, I download these backups to a local server.

The problem I have with this is that to send these encrypted archives to the backup server, each server has a password-less SSH key that allows them to connect to the backup server.

While each server has its own user on the backup server, and the user only has permission to write to its own backup directory, I still fear that a compromised server — thinking ransomware, to be specific — could damage the backup server as well.

I thought about doing the inverse and having the backup server connect to the other servers, grab what it needs, and then shut itself down, but that seems worse, as a compromised backup server would have access to the entire server inventory.

So, I am wondering what you guys do to keep your backup servers safe?