What's a good business model for a product like Keybase

Open-core, or fully open-source with paid offerings (e.g. Zulip -- see tabbott's comment below [0]).

Let the DIYers host their own instances, but offer support and/or on-premise deployments for a fee. It seems to be what Mattermost, Zulip, and others are doing with some success.

Mattermost also includes use of their mobile notification relay servers in their paid offerings, which is compelling, because otherwise you have to build and distribute your own mobile apps.

[0] edited to include reference to important comment

Free for personal and small team use to gain brand and product awareness, then charge for support, additional features, higher usage limits, etc.

Basically the same as Slack, Trello, etc.

Be careful with per-user billing though, as this puts a lot of big enterprises off.