The reason is simple. The ability to define and specify scope/requirements is a really hard problem. Stakeholders change all the time, they change priorities all the time, scope items themselves can be too vague or requirements poorly drafted or the requirements completely change during the lifecycle etc etc.
Deadlines are missed mostly because they are not realistic. It doesn't mean we shouldn't have deadlines but we should use them as a ballpark/placeholder to measure output.
PS: I have only met deadlines in my projects when I was the stakeholder myself :). Every other time, it is technically a miss or "extension". Fun fact: I have worked on many 4 week deadline projects that went for months, may times. I have worked on 6 months project that went for years (looking at you major investment banks :)). Not just because of me but because of incorrect expectations, scoping and assumptions from the stakeholders. Our job is to obviously setup the right expectations and lock down scope as much as possible but in real world, it is frikin hard.
Here’s what the late projects look like:
No project kickoff with the business and architecture. High visibility. Deadlines coming from people outside my hierarchy, with no explanation around who did it or why. Massive, massive scope increases. Estimates turned into “commitments” by my manager. Stress and pressure put down onto to team. The c-levels peering in. The team stressing out about the pressure. Managers creating more stress while I’m trying to promote can-do comrade and keep the morale up, even though I stress-drink at night.
We missed our deadline today too. The business is in an arm wrestle to operate more like them, which we would call waterfall.
But like my therapist said, who used to work in IT: “A deadline is missed. A cloud passes by. A squirrel climbes up a tree.”
I need to keep with my chill self, know for myself personally that I bust ass, and gladly accept that corporate compensation.
If someone on my team misses a deadline it means that I wasn't able to predict the level of effort properly. It means I changed the spec too much and caused chaos. It means I didn't communicate with the engineer well enough to chart out their progress and anticipate the delivery date. It means I wasn't keeping a close enough eye. It means I didn't parallelize the task properly.
Before that, in previous companies, the same thing happened. Lots of projects and no missed deadlines.
I believe it’s due to a simple thing: expectations.
Project expectations must not be around specific implementations or features, but around problems solved.
That gives the project team room to learn and adapt to changing conditions which is especially valuable when you are doing something for the first time.
For the customer, that is having his problem solved, it doesn’t matter if a button is white or blue or if it is actually a button. It matters that a specific problem gets solved when it needs solving.
Everything in addition to that is waste.
Wordplay for profit: Have "due dates" and "milestones" instead of "deadlines". "Due dates" imply on the day after that something new has arrived and needs to be supported.
We've found most late projects were actually late before they started. The slippage was just a symptom of underlying issues within the team, local environment or external ecosystem.
Either way, the reasons are as follows:
1. The timeframe provided for the project has no correlation to reality, and either under or overestimates the time needed.
2. The client/customer needs to do something... then puts it off a few days/weeks/months.
3. Feature/scope creep kicks in.
That's just the work related ones. Personal projects get sidetracked all the time.
When I was in a small team at a larger org, we always missed deadlines.