So I'm curious... has anyone here actually succeeded in estimating a project properly, and gotten everything done as expected within that timeframe? Like say, they had 3 weeks to build a website, and the site was done in exactly 3 weeks with no crunch, no changed features, no team changes, etc?
Are there actually teams that manage to deliver exactly what they say they will in the time they estimate, with no issues coming from the business itself?
- Unplanned Work
- Planned Work That Didn't End Up Mattering
A prerequisite to No Unplanned Work is Knowing All The Work. To get from a hypothesis of the work, to a validated understanding of the work, requires there to be Value Delivery, i.e. you've already demonstrated you understand at least institutionally how to get an increment of value delivered into the hands of a customer, which is how you validate that the Work Ended Up Mattering.
(Consequently, any work that hasn't been validated by a customer, is actually only hypothetical work, as we have not validated that the Work Ended Up Mattering.)
So you want to establish an end-to-end "throughline" as rapidly as possible so you can start iterating and setting up your feedback loops which facilitate work discovery and value validation.
Here's the full maturity ladder that I like, trigger warning bigco language, my product management coach actually doesn't like this: https://www.dustingetz.com/#/page/devops%20maturity%20levels
Yes, it's rare but it takes good leadership from both the technical and non-technical sides of the business.
The technical leadership needs to push back on scope and encourage the folks that need problems solved to be descriptive of their problems rather than prescriptive of the solutions they think they need (but at the same time not dismissive of those ideas).
The reason for the scope pushback is that ideas for problems to solve are likely to have some flaws in the hypothesis given until there's something out there in the real world to learn from. So it's the job of tech leadership to work with other parts of the business to figure out how to fail fast and burn as little engineering hours as possible in the quest to become less ignorant faster.
This has the added benefit of making things easier to estimate, I don't think it's controversial to say that smaller scope means smaller task breakdown leading to more accurate estimates. Teams will usually fail to deliver on estimates until they iteratively go through this process and reduce scope until they can consistently deliver.
When that happens you start to build more trust with other parts of the business, and maybe over time, when you do dare to throw out a ballpark scope for what can be achieved in the month/quarter, you're not that far out.
To be clear though, the trust that you'd be asking for as tech leadership should be a two way street, for the most part it's to be appreciated that the parts of the business that some in tech routinely dismiss are often keeping the lights on, and you need to figure out a cadence that allows your teams time to breathe and refactor in such a manner that still delivers product changes.
First of all you cannot estimate what you don't know. And you don't know what you haven't done yet.
Unknown unknowns is what Fred Brooks explained in detail in his Mythical Man Month.
Then there is the psychological aspect which is what is called optimistic bias. Which is that humans are optimistic beings and will always believe they are better than they really are at some things.
The planning fallacy is based upon this.
All of this is why we try to iterate and validate on feedback, instead of doing fixed timed development. That is simply because we have no idea about the future and guessing on it is more often than not a waste of time.