Why do software companies still insist on Estimation?

Why do software engineers insist on being paid when they never deliver any working software? Imagine a world in which programmers were only paid once their code hit production. My hypothesis is that estimation skills would improve at a geometric rate.

Said differently: good programmers are not afraid of making estimates. This classic article says it all better than I could: https://www.kitchensoap.com/2012/10/25/on-being-a-senior-eng...

From The Unwritten Laws of Engineering, quoted in the article:

Promises, schedules, and estimates are necessary and important instruments in a well-ordered business. Many engineers fail to realize this, or habitually try to dodge the irksome responsibility for making commitments. You must make promises based upon your own estimates for the part of the job for which you are responsible, together with estimates obtained from contributing departments for their parts. No one should be allowed to avoid the issue by the old formula, “I can’t give a promise because it depends upon so many uncertain factors."

Also from the article:

Avoiding responsibility for estimates is another way of saying, “I’m not ready to be relied upon for building critical pieces of infrastructure.”

When you hire a contractor to redo your kitchen, do you ask how long it may take?

When you order food online, do you expect to know when it will arrive?

When you go to your mechanic for a repair, do you want to know when you may get your car back?

When you go to a lawyer to verify a contract, do you need to know when they will be done?

Why should building a software solution be any different?

Let's talk about the reality of software development. Some projects are easy to estimate, especially in domains a developer has tackled before. As an example, consider a developer who has written five static pages who's asked to estimate how long it'll take them to write a sixth. Think of these projects as a well-defined path through the meadow to success. However, some projects are closer to crossing glacier for the first time; you never know when that fluffy bit of snow on the ground is hiding a crevasse of challenges.

While it may be true, saying "it'll be done when it's done" isn't useful. All tasks involve some aspect of uncertainty. Coming up with an at least half-way decent estimate requires a team to break down the problems they're facing into smaller chunks that can be estimated. This is a useful exercise when it comes to prioritizing projects, budgeting, and go/no-go decisions.

There are many ways in which estimation as an exercise can be corrupted. Some people, particularly those who haven't written software themselves, treat all estimates as accurate. In their mind, that "I guess 3 months" estimate a developer gave quickly turns into "I can 100% deliver this to you fully tested, with all of the additions you added along the way in 3 months". This over-indexing often leads to the situation where a developer is disciplined for not meeting expectations, even though they tried their hardest to do so. Some people don't like the estimates they are presented with, so they seek out a lower estimate. They are then shocked when the project takes longer.

I actually agree with this take, but I think the opposite can work. Create delivery dates based on runway / business priorities, re-evaluate what you think you can do in that amount of time. Every few weeks try to determine what you might need to cut to meet that date or see if the business can push back or delay that date.

I don't think starting from I think X feature will take Y days ever works. By contrast, we need to launch by Z date is technically a valid statement, it doesn't need to be estimated.

I think for this to work the business actually needs to be willing to admit that either more people need to be hired, features need to get cut or dates need to be pushed back.

Estimating what?

If we're talking about effort, timelines, etc, then it does work. You are right that it generally doesn't work on the surface. However, even if the estimates are wrong, the benefit is that the artificial dates and such provide a mental push. In some cases dates are important, to meet a business deadline, beat a competitor, etc. Breaking that up and pushing it down to smaller stories means more feedback which provides more of that 11th hour push more often.

And then of course things like costs and staffing are crucial to any business.

There are estimates and estimates.

As a rough ballpark figure (1 day, 1 week, 1 month, 3 months, 6 months, 1 year, 2 years, 3 years, 5 years) estimates are doable and reasonable.

It's also reasonbable and doable to let scope be the factor rather than time. "Tell us the most important thing and we'll do that and deliver the results every couple of weeks, tell us when what you have is good enough".

You can also do fairly accurate estimates when you're basically just doing the same thing as you've already done many times or when the customer doesn't have to give much input. Most of the construction industry is like this because it's older, goverend by physics and building codes and we have a few standard models for most things. Most of the software industry isn't like that.

However for something new, where the customer doesn't really know what he wants in much detail accurate estimates aren't really possible. In this case anyone who purports to give you an accurate estimate is either lying or using extensive padding.

Customers, and non techinical people in general, have a hard time understanding what things are difficult / important from a complexity perspecitve. Often seemingly "simple" (to them) changes are actually very complex, and sometimes the other way.

A trick I learned a long time ago, that is very helpful in old structured code style (e.g. C, mostly-class-free Python, some subsets of C++ -- but rarely Java) is to estimate the number of non-comment lines for each task, rather than the time needed.

This turns out to be much easier to reason about, discuss with colleagues, calibrate one's intuition etc; If one person thinks it will take 100 lines at the end, and another thinks it will take 1200 lines at the end - it provides a more concrete discussion when trying to resolve the difference than "10 days vs. 3 months".

And the number of bug-free structured-programming-style lines a programmer produces per day is surprisingly low variance, when measured at the end of a project. (And ... the mean is often in the 1-10 range for good projects and good programmers for places that require high quality like medical devices or aviation). If that sounds too low to you - consider a project 6 years in the making (~2000 days) worked on by 50 programmers. That's 100,000 programmer days. If it has 1M non-comment lines, it's only 10 per days on average.

Somehow, I've never been able to apply this to Java code or classful Python code, and have never met anyone who was - I suspect it's because these projects are dominated by boilerplate and huge bureaucracies, which weren't prevalent in the old days when this trick was devised.

(Which is not to say that object oriented programming and classful Python and Java are bad; Just that they seem to be resistant to this form of estimation)

Estimates or scientifically wild ** guesses (SWAGs) can be an incredible feedback loop to peers, a team, or an organization.

For peers, far apart estimates might indicate differences in understanding of the problem/solution.

For a team, if they are wildly diverging or low-certainty it can indicate the right next step is experimentation or exploration, rather than committing on delivering something to production.

For an organization, they can provide a barometer for investment decisions, such as what the appetite to solve a given problem space is, or how to evaluate sunken costs.

I particularly like a progressive framework on estimates, i.e. 10% certainty when a project is just taking shape, 20% when scope gets tightened up, 50%+ after spikes / rfc drafting / design etc.

Often they are used for rote accountability, which can be detrimental and/or naive - they should be a stake in the ground to have discussions around risks, assumptions, and trade-offs!

I’m not sure what the alternative is, in any field, unless your work is so boutique and specialized that nobody else does it (auteur filmmakers, maybe). I used to work in a design firm where estimates were even more meaningless than in software. We all hated estimates but we all knew nothing would get done without them.

There's a certain class of engineer that wants the freedom to build whatever, take however long it takes, and have the broader organization react. "You can market the feature whenever I'm done" is, unfortunately, not going to lead to success for the org as a whole.

Because you can't go to money people with "it'll take as long as it takes". It may be true, but that doesn't fit in Microsoft Project, and as long as there is another reputable group out there providing estimates, even junk ones, people will expect them.

I mean, what's the alternative? Even if it's nowhere near perfect, it's far, far better than no information at all. Surely you'll concede that there's an objective difference between something that will take one person 1–5 days, vs. a team 6–12 months?

Also, the process of estimation very often reveals difficulties or unknowns that nobody was even thinking about previously. That's the whole purpose of estimation happening via unanimous planning poker -- the discussion as to whether a feature is a 3 or a 10 is incredibly valuable in getting on the same page as to what the feature even is.

Accurately estimating work is possible and I have done it plenty of times. Precisely estimating work is nearly impossible. I'm of the opinion that everyone who has given up on estimates has some combination of not enough practice and inefficient communication of their estimates.

If I ask an average web developer if they can create a basic landing page in 3 months, they will always say 100% yes. Now shrink it to 1 month: 100%. 3 weeks: 90%. 1 week: 50%.

Now as the person paying for the work, you can make a choice. Do you plan to start advertising on the odds of a coin flip or do you wait until one of the 100% points?

Estimation can be a useful tool for a team to do to ensure that everyone has a shared understanding of a task.

Two drastically different estimations can highlight that there may be a lack of understanding or a high difference in skill / experience on the team. Knowing that allows empowers the team to break a task down smaller, make the task more clear, or spend some time educating team members.

I don't think those estimates should be shared outside the team.

Where I work they ask me how much time a certain task will take, but they still go with whatever time the PM/Manager thinks it will actually take, then they complain when things get late.

I once had a huge discussion with the Architect because I said something would take an year to finish and the Architect was claiming it would take 2 months. He would not understand that the remaining 10% of the task would take 90% of the time (yes, it was partially complete and I had been following its development from the start, although never involved in it directly). In the end they went with his estimation but it took 2 years to finish the thing.

Still, I think estimation is important, once you actually listen to the estimations and understand it really depends on who is doing it.

This take lacks nuance, better to ask why software estimation is hard. To name a few reasons:

1. Business pressures to 'accelerate' timelines without understanding the tradeoffs

2. Frequent developer turnover / ramp-up time

3. Developer + engineering manager level of competence, mostly due to people going where the money is and it being difficult to assess skill level

4. 4-year degrees teaching very few applicable real-world skills and bootcamps teaching even less

Expecting a blank check, eh?

Steve Mconnell dispelled many of the misconceptions around project planning and estimation around twenty years ago. Worth a read.

Because they're not talking to only Developers but others who think in traditional project terms. Timelines, or at least tentative timelines. Or at least a range. Just something that someone ELSE can plan around. Like Marketing teams who need to know when something will likely work so they can do their own piece of work.

How would rest of the organization work if every new feature and bugfix would never have expected delivery date and everything was done "when it's ready"?

I understand your point and estimating is never easy, but software engineering alone won't keep the company a float

I dunno, using previous projects as a similar baseline seems to work out very well for estimating within ~10% of future projects.

Although not all future projects are of a similar scope with similar people as a previous project but when it applies it seems to work well.

OP: If you were spending your hard-earned money to pay a team of engineers to build your vision, would you care about how long it might take and how much it might cost?