Estimates, Time and Toxicity

or... How to build software reliably without “Crunch”

I have been building software professionally for twenty years, and the one utterly constant truth I have discovered is this;

Making Time a key factor in the lives of Coders, and specifically in the process of estimating and planning work, is a recipe for any number of separate and parallel disasters.

(Oliver Godby, October 2020)

Now, I know what you are thinking… You are thinking that I am one of those opinionated mavericks that believes Software Engineering should have complete primacy over the business and that being subject to a wider business priority, such as the needs of a major client or the whimsy of the market you are operating in, is some kind of affront to the purity of the code cut by the highly trained artisans that deign to cut it. I can see why you might think that, but bear with me, I promise you that there are benefits to come that you might appreciate, and that I am not so mad after all.

Human beings experience Time as a scarce and fleeting resource; it is the only thing(*) that we cannot ever make more of, and somewhere in the back of our minds we all know that we have an arbitrary and limited amount of it ahead of us. The mystery that is the available pool of time we each have available to us in any number of contexts is at the very root of why the planning and estimation of the creation process in software (and perhaps other pursuits of which I cannot speak with any authority in the way I can about software) is irrevocably poisoned by attempting to use it as a frame of reference before the work has begun.

Sit with this for a moment; imagine a World where no one draws a notional line in the sand and then attempts to work backwards from it in order to deliver code, applications or systems by the time the clock strikes the appointed hour.

Okay, let us dig into this a little further; have you heard of the Productivity Triangle or the Creative Triangle? If I am honest I am not sure it is called either of those names officially, but I have heard it referred to by both in the past. Here it is:

This age-old idea is that in terms of building or creating anything, and I do mean anything that needs to be fashioned in some way for a bespoke purpose, you can only pick two sides of the triangle and must sacrifice the third in order to achieve your aim. For example, if you want something to be built from scratch quickly and to a high level of quality it can never, ever, be cheap; in fact it will cost you a lot more than it might if you did not insist on speed and quality. If you want something to be high quality, but also less costly than the market rate, then you must accept that it will not be built or achieved quickly.

There is no great wisdom here, this is a fairly low-level cliché that most people can understand intuitively, but there is a deeper truth waiting to be perceived. If you can simply remove the importance of Time from the equation it becomes possible to operate at a level of high quality and predictable cost all the time. Your developers are no longer caught up in discussions about what corners to cut in order to deliver “on time”, as Time no longer has any relevance to them. There is no need to “crunch” to reach a deadline because the work will take the Time that it takes given the other constraints and that is simply the way it is. Put another way; you can get more money, you can hire better people to get a higher level of quality, but if Time is on the table you are operating in a frame of reference where one of your key decision-making factors is scarce and there is no way to change that. Not only is it scarce, but it never stops, so if you create a point in the future by which things must be done you are begging to fail from the moment that you do, because Time is already running out.

Even in Agile methodologies, where the entire approach is centred around results (even if those results need not all arrive on the same day and in the same way) Time insinuates itself into the planning, management and reporting process if you allow it to, and once it is a part of the discourse there is an unbreakable psychological connection between an expression of the amount of Time any given task might require and an expectation that delivery will be possible in that timeframe. What I am saying is, if your estimation process boils down to “X days for Y feature”, no amount of generous reinforcement of the idea that delivery could be early or late will alter the instinctive way in which particularly non-technical people will pin their hopes and expectations to an estimate.

Surely not, I hear you cry! The word “estimate” means an approximation(**), surely people can understand that? Well when you ask a car mechanic for an estimate to get your car back on the road, even though they may warn you that it might be more in the end, it becomes your expectation of what you are going to pay. Expectation is the key word here. Because of the way we use the word “estimate”, in relation to a job of work, what leaves the professional’s mouth as an estimate becomes an expectation in the mind of the stakeholder.

The solution, is remarkably simple. I am going to turn to Ian Carroll, a prominent Agile Practitioner and thought-leader in the space, as he has said it so perfectly:

“The first rule of sizing user stories is, never ever (directly or indirectly) size stories in any unit of time. The second rule is to always use relative sizing and avoid at all costs absolute sizing.”

(How to do Agile Story Point Estimation by Ian Carroll)

The moment you take Time out of the equation and focus on the overall size of a piece of work, feature or task, and “size it” instead of “estimate it”, basing your thinking only on complexity and risk, and only relative to other pieces of work, you are able to ignore the tick, tick, tick of the clock and all of the negative associations that Time brings into the room with it.

Now, some of you are starting to wonder how anyone could ever hope to run a business on this basis, right? I mean at some point you are going to need to be able to promise your investors or your bank manager or an important customer that features A, B and C will be ready and released by some point in the future and if your development team cannot project a workload into the future accurately then more often than not delivery will be late and people will be let down, money will be wasted and your business will fail. Right?

Ok, so it does not have to be this way. There are two strategies that will allow you to make plans and be effective without trying to get your coders to wrestle with Chronos.

If we are talking about a brand new company with a newly assembled team and an as-yet unrealised product, the first six to twelve months need to be planned not on how long things will take, but how much money you can afford to spend (or burn, as people locked into the VC model like to say).

If you have enough money to run your development team for twelve months without any revenue or any further investment, then realistically you have nine months in which to deliver a saleable product (let us leave discussion of the mythical “Minimum Viable Product” for another day) and have enough time to sell it to enough people to either carry on by virtue of cashflow alone or to attract further investment based on momentum and traction.

Do not be tempted to set a deadline and then work backwards trying to figure out what can be achieved in the Time allowed. Look at your budget and your people and ask them “What can you get done with this pile of money?”. Do not even mention Time. Make a realistic plan to get to something that you can sell before you run out of money by focusing on delivered, production-ready features, and do your planning in a Time-vacuum from the start so that you can gather data about the cycle-time and productivity of your development team based on the number of features that they can deliver for a certain budget. There is a catch here, and this is really important. Even though I am going to suggest, in a while, that you should look back at the number of points they are able to reliably deliver each iteration (some might say sprint), DO NOT attempt to work out at this point the money cost of a story or complexity point from their estimation process. There is no harm in considering the money cost of features, as long as you can consider features that overall were roughly the same complexity as one another in the eyes of the team, but honestly I would not bother. Focus on outputs and outcomes, there a greater rewards to be had if that is where you put the emphasis.

Okay, so you have got your initial sale offering and the budget to carry on operating. You start to sell and you are lucky enough to attract some investment off the back of the traction that you are showing. Time to build more features, and more value, and level-up. The problem you may face is that now the market, your investors and your customers, are hungry for the next feature release and they want to believe that you can tell them when it is coming.

Look back at all the iterations that got you to that initial saleable place, and use them to work out the velocity of your team, effectively the number of points that they can achieve in a single iteration. We will call it v.

Now go to the team and ask them to size the work that will be required over the coming weeks to release the next big, juicy feature on your roadmap. Do not say anything about Time. They say it will be P points. Divide P by v, and that is how many IDEAL iterations it will take to get to your next feature, and we will call that number x. Build in some safety and then look at the calendar. At the moment there has not been a single line of new code written — do not pick a deadline. Look carefully at the calendar and decide if it is more likely to be the end of the quarter or the start of the next quarter (or month or year, you know based on the size of x) when x Time has elapsed. Sanity check your general timeframe with the Delivery Manager or Scrum Master or whatever you are calling the person who runs your process, and that is the slightly vague timeline you will give to investors and customers, e.g. “We’re going to have [insert feature name here] by the end of Quarter 3”.

Set the team off running and monitor the completed stories / tickets / jobs and keep in touch with whether or not the Delivery Manager feels that velocity is being maintained. If it is being maintained by the end of iteration three or four (assuming six, two week iterations in a quarter, sticking with the example as established), start to talk about “the end of September”. When you get to the end of iteration five you and the Delivery Manager will know for sure whether or not the work is going to be finished by the end of September. Plan accordingly; you will have enough information, without ever having to talk to your developers about Time at all, to be able to commit to a delivery date that is as safe as it can ever be, one that could only fail to be met due to mass casualties amongst the development team. If something unforeseen comes up it will likely happen early, before you’ve committed to even the vaguest of timeframes. Once there is certainty and consistency in the velocity again you will be able to follow the same pattern, the delivery date will simply be further out.

You (and by you I mean you the company, and by that I mean all of you; everyone will play a part not just the developers) will deliver on time. Frankly you will probably deliver early, internally, thereby giving the development team and the whole company plenty of opportunity to dog-food the new features, iron out any really embarassing bugs (there will ALWAYS be bugs) and release publicly on schedule.

Your investors will start to believe that you have supernatural powers.

Your customers will rave about how they did not have to wait for the promised bounty.

Your development team will be the happiest bunch of developers you have ever met. They will have delivered well made, reliable, solid code, which is what they live for, and no one will have pulled an all-nighter, no marriages will hang in the balance, you will not have dropped five hundred pounds or dollars on pizza in the last week. After two or three days of washup and project retrospectives they will all be raring to go with regard to the next challenge.

Most importantly, your development team will get better and better at judging the relative and non-Time-based “size” of the things that you ask them to achieve and you will be able to commit to delivery dates(***) sooner in the cycle. As your company grows and thrives your initial cadre of developers will teach the next wave of people that you hire how to size the work that they are given, and how to love the development nirvana they find themselves in. You will still lose the odd person, you may even make the odd bad hire, but overall you will create a “dev culture” that will attract the best people and when they discover that the hype is not just hype they will stay longer and be more loyal — this is a cast iron lock.

I know that I am making it sound more simple than it really is. You need smart, motivated people who are prepared to trust you that you really do not want them to think about Time. You need discipline and nerve to begin with in order to trust that it is really going to work and that you are going to get reliability back in return for your bravery. You need Delivery professionals who will take it seriously and preserve the conditions needed to make this approach sing. You need a business that actually makes something people want, because you need to make money and be successful, otherwise the best “dev culture” in the World cannot help you.

Like Orpheus leading Eurydice out of Hades, you must NEVER look back. If you look back once, if you lose the faith and give in to the temptation of an arbitrary deadline or asking a development lead to tell you whether or not Feature A will be finished by Date B it will crumble and fall. That danger will always be there. If you want this to work you have to stay the course, but I have seen it work, I have been there when it flourished and I am here to tell you that it is worth it.

(* I realise that there are a few other things we cannot make, like matter and energy, but I mean most people in the normal course of life… Look it is a well known mode of speech to talk about not getting the two hours back that you lost watching Scary Movie 3 or somesuch. You know what I mean!)

(** I once heard someone say that “an estimate is merely a guess in a clean shirt”, but we all know that size matters.)

(*** Please stop calling them deadlines and start calling them delivery dates. If you do happen to miss one, no one is going to be dead.)

Oliver Godby (LinkedIn) is a veteran Software Engineer and System Architect who has been writing code for over thirty years and making money doing it for twenty, on three continents. He currently heads up Platform Engineering for Cervest, where he is trying to use his powers (and Clojure) to help Cervest do its part in saving the World.

Title Photo by Aron Visuals on Unsplash