Ever heard of Planning Fallacy ? Planning fallacy is defined as:
A phenomenon in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed.
We tend to estimate with an assumption that there would be no issues during implementation. Dunning and Kruger mentions that we suck at measuring our own abilities and competence. However, irrespective of the fallacy, one thing that most of us will agree is that estimations are hard and tricky and often they are a point of debate between the developers and engineering /program /product managers. In my opinion, estimating software is a very underrated skill.
I am putting down some pitfalls and practices wrt software estimations. I will also talk about various situations that the developers face with respect to estimations before the feature is started as well as when they are implementing it.
I think it would be a useful read for developers and product owners alike. Let’s dive in.
Why are estimates necessary ?
Although the premise of no estimates which says “If we accept the premise that our estimates are going to be wrong anyway, any time we spend on them is wasted” looks woke and cool, but in my opinion, there are a lot of benefits in estimating things.
- It is a very key input to devise strategy. Estimates are useful in determining what work can the team/org undertake in a given planning cycle as the number of engineers are limited. These are really high level estimations (t shirt sizing) which help management in particular understand whether certain work can be considered part of the planning cycle or not.
- Product owners are looking for predictability and some sense of the time/effort it will take to deliver a feature. They are also juggling with the impact vs effort estimations.
Product owners are also part of this estimation cycle dependent on developers estimation. They will estimate the number of resources needed for the feature and whether help is required e.g. do you need additional SWEs, PMs, etc. They will communicate the estimation higher up on the ladder.
- If other teams are dependent on this feature, they need to communicate these estimates to them.
- It helps in identifying various subtle aspects and ambiguities in requirements.
How should I go about estimating ?
As per my understanding there is no common process of estimations as it is highly dependent on business, culture, team expertise, size etc. Here are some tips to make reasonable estimations.
Before the feature implementation starts
Understand the requirements
Understanding of the requirements in detail is one of the key requirements before giving an estimate. During the planning cycle go properly through the product requirements, mocks and any other documents which explains the requirements. If nothing is present ask for one or create something yourself so that you have something in writing which you can refer to later. Ask questions if any and get clarity from the product owners.
Don’t think only in terms of code
A feature implementation does not just mean writing code. Every team should have a DoD — “Definition of done” at the feature and a bug level. This is very useful in estimations. Some people think that sending out a changelist is what the estimate is for but that is totally incorrect. There are a lot of other things which need to be taken care of. Some things which are on top of my mind are:
- Testing strategy
- Documentation
- Deployment
- Accessibility
- QA cycle
- UX reviews
- Bugs triaging
Each of them can take a significant amount of time depending on the kind of feature you are building. Do take these in account before giving an estimate.
Break the tasks
Breaking a feature into logical tasks helps us in identifying the various aspects of the task at hand. It helps us to see the feature as more approachable and doable. It will be easy to create a mental picture of which tasks look familiar and which are unknown.
- For tasks which look familiar or similar to what you have done earlier, you can take historical context into account and add reasonable estimates for them.
- For unknown tasks, do some research, take help of teammates and seniors so that you get an idea of what you are tackling. Consider putting down time boxed tasks for the research aspect as well.
Don’t go to the other extreme and waste time breaking down things at the lowest level of granularity possible.
Discuss, Prioritize and Negotiate
Once you are done breaking down the tasks, discuss these tasks within the team and prioritize them. Categorize the tasks in ‘must have’ and ‘nice to have’ categories. If the estimate is high from the product owners, help product owners understand why it would take so long. Prioritizing and categorizing helps a lot in moving the levers of quality, time and scope of feature.
Notes for Product owners:
- Have a notion of “efficiency ratio” which can be used to map developer estimates to calendar dates. Dev effort usually excludes things like meetings, on call, sick days, routine code reviews etc. Most teams simply divide the dev estimates by 0.7 or 0.8 to arrive at real dates.
Communicate when in doubt
Whenever, you feel that due to any reason you might be missing the estimates. Be proactive, discuss the issue that you are facing with your team members or your manager. Make sure that they understand that the issue you are facing can lead to slippage on estimates.
General Situations and Dilemmas wrt Estimations
Before the feature is started
My manager is asking for an estimation of a new feature. I have worked on a similar feature and know how much time it took earlier.
You can give a high level estimate but make sure that you understand the requirement properly. Don’t get pressured into giving an estimate right away. Learn to ask for time to derive the estimate :).
My manager is asking for an estimation of a new feature. Although I am familiar with some stuff in the feature but it has unknown unknowns.
You don’t need to have an answer on how you would deal with the unknowns. However, you should have an idea on how complex they can be. Do some time boxed research to know the unknown. To your manager, it is perfectly fine to say I don’t know at the moment, but I can spend a few hours/days on it to get you a reasonable estimate.
My manager is asking for an estimation of a new feature. The work involved is completely new to me.
It is better to give an estimate of a high level design doc in such a case, rather than giving an estimate of the whole feature. Once you have a high level design doc, then you will be able to break the tasks and give a reasonable estimate. The question that immediately comes to a curious mind is how do you give an estimate for a design doc. Well, from my perspective I think the design doc estimate depends on how big the feature or how complex it is looking upfront. The whole idea is to time box the writing of a high level design doc to bring some clarity.
There is a very important feature with a hard deadline. I was asked about the estimates and I don’t think I will be able to finish on time.
Discuss, Prioritize and Negotiate is the way to go.
There is a very important project with a hard deadline assigned to me. I was not asked about the estimates. I don’t think it will be done in time.
Dev working on the project are the ones who will implement the feature. They should be involved in estimation of the feature. Raise concern to your manager.
I gave an estimate and my Manager/TL disagrees with the estimate. I said X, they say it should take less than X or I said X, they say it would take more than X.
This is a common situation coming out of either
1. Ambiguity in the understanding of the requirement. In this case discuss and understand the requirements. Make sure that everyone is on the same page.
2. Ambiguity in the understanding about the complexity/scope of work. Instead of going into bargaining mode, help product owners understand why you think it will take the time specified by you. You can discuss, prioritize and negotiate.
3. Lack of trust in between the developer and their Manager/TL. This is a bigger issue than the above two and kinds of out of scope of this blog :).
During implementation
There are some unforeseen issues like random build, environment issues which are taking a lot of extra time. It is possible I might not meet the timelines.
Don’t get stuck silently, Ask for help. Take help from teammates. If it is taking more than a day, make sure that your manager knows that you are stuck.
While implementing it, I realized that there is a design change which is needed. This change will help in meeting the requirements but will result in missing the estimate given by me.
Discuss the design change immediately with your TL. If it is a must, weigh in options of going with a suboptimal design and it’s consequences. Estimate the time it will take to go with the new approach. Communicate to the stakeholders and take a decision. Help them understand the consequences of not going with the design change. Make a side note to understand why it got missed in the initial design.
While implementation, I realized that there is a component which can be made reusable and will be useful to other teams as well. Making it reusable will take some extra time though.
Generalizing too early is a fallacy. Understand the “rule of three” in [software] reuse. It is three times as difficult to build reusable components as single use components, and a reusable component should be tried out in three different applications before it will be sufficiently general to accept into a reuse library.
Few good reads
1. YAGNI (“You Aren’t Gonna Need It”)
Discuss with your TL. If it needs to be done then follow, communicate when in doubt.
I am stuck on an external dependency. This is making me worried about the timelines.
- Code reviews are not happening on time.
- Some comments on code reviews are not getting resolved and need more discussions.
- Some work is pending on an external team.
With respect to code reviews, every team should have some rules around SLA and what are the steps that can be taken if the SLAs are not met.
Don’t hesitate in asking help from your manager when stuck.
There is some unrelated work that came in. I am spending time on it. I didn’t factor this while estimating.
Let your manager know when you get assigned a new task. If the task is more important than the feature that you are working on, then the calendar date for finishing the features should be pushed. Or if it is the other way round then they can help in finding someone else who can work on the assigned issue.
There is some unrelated work that came in or some tasks in the project are taking time. There is a weekend in between. I might work on weekends to finish the project on time.
Working on weekends once or twice is fine. But please do not make it a habit.
