Here's roughly what I'd want out of the license:
1. The source code can't be used for commercial projects. I know this would be difficult to enforce, but having legal remedies for this in some jurisdictions is better than nothing.
2. People who contribute to the project would not own the source code. I know this stinks, but the way I look at it is if there's a bug in some proprietary software I'm using, I'd like the option of looking at the source code to fix it, and giving that commit to the owner/maintainer of the repo if it means I can have software with my bug fixed.
3. The license could allow people to self-host their own instance, but again, only if it's for non-commercial use. Ideally there's some remedy for if the company goes bust or in a direction the user doesn't like ... they could keep hosting it.
What existing type of license most closely matches this? Has anybody worked in a project with such a license? I can think of a few ways it would backfire, but hearing it from somebody with experience would be way better.
The intent is that I can continue to own the SaaS code while letting others fix things and self-host, if that's what they want to do.
https://www.mongodb.com/licensing/server-side-public-license
https://polyformproject.org/licenses/
https://www.elastic.co/licensing/elastic-license
AGPL is another possibility; technically it allows commercial use but in practice companies are too scared to touch it.
Let's say someone wants to use your "wildcat" license. They'd realistically need to hire a lawyer just to understand what the legal implications of your license are. If you're planning to write this license so that you can just easily understand the implications by reading it, that is not wise because unless you are a lawyer you don't even know what you're doing or what the license really means. This is an important part of the problem that free software licenses are supposed to solve.
That said, look at the GNU AGPL. It doesn’t prohibit commercial use but a lot of companies seem wary of it.