I've seen a combination of manual, expert rules/code and machine learning involved at my company, but it's a mess. So many teams, so much politics. How does your company do it? What do you'll do well, and how did you get there? Pitfalls?
If you could design a system that works for engineering and operations teams, how would you?
Manually doesn't scale, but also the people who are closest to the customers (operations) usually have most decision-making context anyway.