- Cost savings: With us being an ISP we will always be paying for rack space at multiple DCs so there's no money to be saved here
- Scalability: Our applications can handle an order of magnitude more people than they currently do and reworking our programs to work with this is another time sink
- Portability: See cost savings. We're never going to be switching cloud providers
- Deployment: Gitlab CICD handles it now. It's just SSHing, cargo build, systemctl restart. I love the simplicity of that over building a new docker image and maintaining some docker repo
I should also point out that we're not even using docker as of now.
Right now it seems like we'll end up with the same outcome of running Proxmox but we'll spend quite a bit of time getting up and running with Kubernetes. Please let me know if I'm not taking some things into consideration
This is 90% of shops. The reason people are deploying k8s is not because they _need_ it, it's because the devops working there want to get k8s under their belts and onto their resumes so they're more hireable long-term. It's a feedback cycle.
We hired a new guy and all he wants to do is tear out everything I have spent 3 years building - after I was denied being allowed to use Nomad or K8S three years ago, due to rational arguments around its complexity and our needs. Of course, he wants to replace it with k8s. No respect for what has been built, no appetite to help me go after the low-hanging fruit that would get us 80% of the way to k8s deployment speeds without having to significantly re-architect anything. Nope! Gotta be CLOUD NATIVE!!!
TBQH from what I have seen it's overengineered and only makes sense to use when someone else is running it for you. Then you just consume it as a cluster and don't worry about what's inside the black box. This makes sense for younger Zoomer devops who have never even touched VMware or any on-prem servers and think everything is always clouds all the time.
IMO, you should write up two alternate timelines: the one where you spend a few months getting K8S to a POC stage, and the one where you spend that same time tuning the deployment and management of your application in its current habitat. Probably the kinds of features and reliability you can deliver in the latter case will be far more appealing once you actually lay them out. Don't play in the "k8s ideal utopian state vs. actual current state" fight, play in the "k8s expected early outcomes vs. maturing current model" fight.
Kubernetes is great when it's all set up and working properly. The challenge is getting to that point and training enough of your staff to be able to navigate Kubernetes in case someone is on vacation.