Last week I connected with a friend and colleague and we talked about all the exciting things his team was up to, as well as the challenges they were facing. He mentioned that their business was in a rapid expansion phase, which means new hires, quick decisions, and lots of investment into what's working well.

One consequence of this growth is pressure external to his development team to deliver specific things by specific dates. Typically, these dates can come from sales, executive promises, or just arbitrary deadline decisions from management to encourage people to "get work done".

From a software engineer's perspective, these careless deadlines can strike a tone of mistrust of the sort that makes you feel like you can't be trusted to get the work done in a timely manner unless a fire is lit under you by your superior. These deadlines alone can undermine relationships and create hostility among colleagues who are otherwise working towards the same goal.

Sometimes, however, a target date is calculated from a set of estimations provided by the development team itself. If you break your Product Backlog Items into small enough sizes to estimate them, you can try to project a date by which a certain subset of them will be delivered, maybe. Certainly it's better for the software team to be deciding delivery dates when at all possible. However, software engineering, like many complex disciplines, is fraught with unknown unknowns. Even these dates can missed, and occasionally by a large margin.

On the flip side of external dates that erode trust of the software team, internal dates that are missed erode trust of the user. After carefully listening to my colleague's description about the challenges of deadlines and estimating work, I asked him if he had heard of deterministic estimation versus probabilistic estimation. He had not, and so I described to him a forecasting model with a scenario, much like the weather forecasting model we are used to:

If Bill the software engineering manager says to Ted in sales, "We will be done with your PBI by Friday" and then doesn't deliver on Friday, Bill has eroded trust with Ted and the sales department. However, if Bill says, "There is an 85% chance we will be done with your PBI by Friday based on historical data" then Ted might be disappointed on Friday by a miss, but understands this feature is an outlier and can still expect most of his PBIs to deliver on or before their forecasted date.

As if prompted by my recent conversation, Louis-Philippe Carignan has written on this exact topic yesterday in a blog post titled Agile Forecasting Techniques for the Next Decade. He writes from experience using SLEs (Service Level Expectations from the Kanban Guide for Scrum) + PBI age to track daily work in the Scrum, as well as using Monte Carlo simulations at the release and product level to project the likelihood of finishing a certain number of PBIs within different days. This strikes me as a vast improvement to the way my teams have estimated in the past.

I used to advise mentees to estimate task hours and track actual hours in order to learn how to estimate better. It does work, but this advice still champions deterministic estimation. Humans are pretty terrible at estimating, even with plenty of practice. I agree with Carignan that it's time to look to a future where we lean on probabilistic estimation over the deterministic estimation that has failed us so many times in the past.

How does your team estimate? If your estimation technique has burned you in the past, how so? What do you think of the concept of probabilistic estimation?

photo credit: albyantoniazzi MINIMETEO for iPad via photopin (license)