The software development world has been sucking at estimating cost and time of projects for over 50 years now. Not only are we wrong so often, but there is a lot of suspicion about how much “buffer” we’re adding.
To this day, team leads and project managers wonder why estimation (and work-logging) is the cause of so much contention in our industry.
How a car accident led to a revelation
A week ago, I was in a car accident.
I’m fine, thanks for asking.
I took the car to a body shop, where a youngish gentleman came to the carpark to look over the car and give me a quote to fix the damage.
I’m always finding ways to learn from people – no matter how whacky – so I asked this guy, “How do you estimate this accurately?”
He told me how he breaks quotes into Parts and Labour components. The Parts are easy – a simple lookup in a database. The Labour, however, is much more tricky (and similar to software development estimation woes). He told me that it takes experience and expertise. Then he dropped the bombshell – only ONE person in the team does it. What!?
One person?? I have been talking about Wideband-Delphi estimation techniques with so many people over so many years that I took for granted that estimations are best done with a wisdom-of-crowds approach. When I heard one person was doing all of these, I had to ask:
How often do you get the estimation right?
The guy’s face went pale and an awkward silence ensued. Without words, lots had been said – so I left it alone and slowly backed up towards my (damaged but still running) getaway vehicle!
Clearly, our auto-shop friends have the same worries with estimations as we do in the software development world. Not really a revelation – when you think about it – both services are “job shops”, both services are complex and intricate, and both services can be analysed to the very last dollar!
How often are we accurate when we estimate?
But the uneasiness kept me wondering…
I had to know whether this one person estimation technique was better than our multi-person approach.
So I got to digging. I got access to 3200 data-points across multiple teams, organisations, projects, and even countries! I analysed each tuple with just one question:
Does Original Estimate = Final Actual?
This is where things got real.
I couldn’t believe it.
I spent about 10 seconds processing it, while my brain tried to convince me how wrong I had been with wideband-dephi, how I had misguided multiple software teams, and how I was a total fake. The stat?
Estimate = Actual for 45% of all user stories!
Forty-five percent. FORTY FIVE! No way, no way we are THAT good at estimating software. Given that more than fifty percent of software projects fail, how on earth can this stat be true?
Something very suspicious was going on.
Given we were using Fibonacci number estimates – with larger and larger gaps between available estimated hours – surely most Actual times would fall between these estimates (e.g. 9.7 hours logged because 8 is too small an estimate and 13 is the next option).
The mystery of exact time work-logging
I needed to get to the bottom of this.
So at 2am in the morning, I fired up my laptop and started some research. A peculiar concept of work-logging kept popping up: Parkinson’s Law.
Work expands so as to fill the time available for its completion.
The amount of time that one has to perform a task is the amount of time it will take to complete that task.
Okay, so this guy’s basically saying that we do add a lot of “padding” when it comes to estimations and work logging – when we could easily get the work done in less time.
The academic in me didn’t believe his premise, so I jumped onto Google Scholar and looked for evidence to suggest Parkinson was not on the herb.
I found it (not the on-the-herb part)!
Multiple studies have been commissioned to (dis)prove Parkinson. In all instances, the same conclusions were drawn out – people will fill up any given time to complete a given task.
What happens is that, in any given deadline situation (think back to school/university assignments), your brain consciously ticks away the time. At the start of the given period, it lets you concentrate on details, take things slow and produce high-quality output. As time moves on, the quality is compromised for efficiency and speed. By the end of the period, you do more but with little regard for quality. The two trends counteract each other:
So what we do as humans when we are planning (a.k.a. ‘Estimating’) is we look for the point of intersection in the above graph – and set our estimation in line with the expected averages of the two dimensions.
The research kept showing that whether you have to speed up or slow down, you tend to meet the deadline exactly.
Aha! You. Meet. The. Deadline.
Now 45% was making a lot more sense. In fact by about 3:45am, my brain was starting to ease, the heart rate was coming back to normal, and my earlier sense of total failure and boneheaded-ness (yup, I really just used that word) was starting to subside. Time for a nap.
The problem is not our Estimations
So after a brief nap, I hit the office with my next question: how do we fix this problem?
My first inkling was that a % reduction for each estimation would mean that we can get the same amount of work done quicker (and yes, cheaper). Clearly, my MBA has made me a bit of a tool. But after a few more seconds, I realised something…
There’s actually no (estimation) problem to fix.
The problem is not in the estimation.
It’s in the environment that pushes estimates to be more than that.
Estimations are not, will not, and cannot be future actuals. It’s simply not possible to know how long something will take to build. But in a culture that turns estimations into a premonition of the future, we get a lot of suspicion and second-guessing.
That’s the real problem.
The problem is rooted in the embedded psyche of an organisation or team, where cultural traits give rise to behaviours such as scrutiny of estimations and using them as key performance indicators. These behaviours are the real problem – and the fix is a long, painful process of looking at organisational health and building a culture of respect and understanding.
In fact, that’s exactly what I had to do through my deep-diving on estimation. I had come full-circle.
- I started with the premise that estimation is good for planning and forecasting, but can never be used for measurement of performance.
- I then had my OMG moment and did a whole lot of research, only to find that we’re getting it exactly right 45% of the time – which I found awfully suspicious.
- I then understood how work-logging happened in our (human) brains
- Finally I realised that the accuracy (being dead-right 45% of the time) is actually great!
The key here is to not link the concepts of performance and estimations. It’s a really easy trick to fall into, but SO not worth it.
Should the mystery of super-accuracy worry you?
No, it shouldn’t.
There are many entrenched reasons that estimations get the scrutiny that it does (even in the auto body shop world). By far the biggest contributor to the underlying suspicion we have with work-logging and estimation is the team culture. In an open, honest, and functional team there is a place for estimations, and a place for performance, but they are not linked.
They (and a million other factors) build a story of overall team performance.
Now that is a pretty long read, so thank you so much if you’ve made it here ’til the very end. For those of you guys who are more into videos, you know I always got you covered! Enjoy!