Kubernetes at big, non-tech companies?

> Am I crazy for thinking that this misses the whole point of the technology?

Multi-cluster is not an accidental evolution. It was largely inevitable. Kubernetes has no hard tenancy concept (no, cobbling together a pile of namespaces and feature encrustations and calling it "tenancy" doesn't count). Instead people simulate tenancy with multiple clusters, because that isolates a particular API server's data completely.

While early attention and nerd glamour may have attached to the idea of workload consolidation, the true nexus of value these days is the API itself. This will probably remain true even while people are being burned by multi-cluster waste. There will be some movement to one of the various I-can't-believe-it's-not-multi-cluster solutions that are currently swirling around. Vendors of all sizes will be standing by to sell their take on the Kubernetes of Kubernetes.

I guess it would be helpful to understand what your expectations of the technology are as well as providing a few more details. What you are describing doesn't sound crazy to me assuming selecting a cluster to deploy to is a one time thing. If it was every time you wanted to deploy a change that would be cumbersome. I suppose the step you need to do could be automated, and there are multi cluster solutions available, but I'm not familiar with them because I don't use Kubernetes this way, so I can't speak to how well they address this particular issue. Are you expecting that cluster selection is automated the same way that Kubernetes automatically selects which node to assign a container?

Edit:

Some things that might be helpful to know:

- Is your company using cloud or DC? This might explain some of the decision making.

- Since this is a pilot, is it possible there are plans for future improvements regarding cluster selection?

I built out the Kubernetes stack for data processing / batch processing with Airflow at Everlane. Fully automated deploys, monitoring, etc. Mind you - we didn't build this from scratch / use native Helm charts, but used an integration partner that gave us a much better foundation. The data engineers were responsible for maintenance, deploys, etc., but with the sophistication we got it was very rare to have to actually do helm deploys.

It really depends on why your company is using multiple clusters. Are there any size constraints? Any organization rules for which application should go where?

Overall no it doesn't miss the point because once you select a cluster to deploy to Kubernetes will still handle the rest for the lifetime of the application. And I imagine the selection part can be easily automated down the line as well.

Selecting a ready-made cluster that you just have to check is marked healthy and has the resources your project needs?

Sounds like they made it really easy for you.

An auomation could break something if automaticalyy choose the wrong cluster. Are all these clusters "free to pick"?

What in your opinion is “the point?”

To me as an SRE, the steps you mentioned sound entirely reasonable.

In our org, the cluster operators manage which clusters are open for onboarding vs. closed to new projects but running for existing projects (based on various scaling limits, some of which are cloud-specific). We don't ask developers to choose what's healthy and what's not. Our internal tooling simply only offers "open" clusters.

If you are cloud based this does not sound correct. If you have the cluster autoscaler setup, then the cluster should grow to allow for the deployment based on the constraints in your deployment.

If you are on prem then this is reasonable, since you have limited physical hardware so the cluster can not automatically grow.

I feel your pain man.

A big part of the story is that k8s is a naked king.

It’s just really poorly designed, over-engineered, shitty devx piece of software.

But most of devs eat you alive if you tell them this harsh truth.

We, regular engineers must be patient while people coming to enlightenment on their own and someone with big money would start developing & promoting better alternative.

Kubernetes is horribly complicated, I would not recommend it to anyone.