Forecasting a Roadmap with Flow Metrics

Predicting the future is hard, predicting further into the future is really hard. In my role I’m often asked to forecast large projects spanning multiple teams to give senior stakeholders a high level representation of what we are likely to be working on at any given time. This is essential for ensuring budgeting and hiring decisions which need to be considered further ahead than a single Sprint.

We must understand that long term forecasts can (and should) change. Not just because we learn more and work has been completed, but because we should be willing to pivot onto more valuable work should the opportunity arises. But that doesn’t mean that we should neglect forecasting at this higher level entirely.

Scrum Teams represent longer term themes of work as a Product Goal, this is a future state of the product which we are aiming towards over the course of multiple Sprints. We’re not naive enough to believe we can define or plan it all up front, but it provides a longer term direction which we can plan against. Its reasonable to assume that an organisation may have multiple goals one after the other, however (as per the Scrum Guide) we should clearly be only working towards one at a time, see more on this below.

Lets model these Product Goals as a collection of work items which need to be completed for that goal to be realised. This being Agile we should continously inspect and adapt these work items to ensure that they are all necessary and of the highest value. To borrow some common terminology we’ll refer to these as Epics and User Stories although this may change from team to team. If you’re not using Scrum then don’t close your browser quite yet, you can think of these as projects, initiatives, or whatever terminology exists in your organisation.

We want to be able to predict the rough delivery dates for these Epics several months out. As I described above, this is not an unreasonable request, in my industry we have lots of people involved in the legalities of getting work done in the financial sector and it’s only to be expected that they would like to know when their help will be required, even more so if it requires work from teams in other companies. We also want to avoid that dreadful feeling when work seems to keep “sliding right” with more and more delays on original forecasts. Why is it that work never seems to be overestimated?

With this in mind lets apply our Kanban knowledge and discuss some lessons we can take from managing the flow of work through teams which can be taken up a level to help us forecast when these Product Goals are likely to be completed.

I hypothosised that a Product Goal was just another kind of work, albeit a significantly larger one than the work items the teams deliver day to day. I summised that if I was to track the Cycle Times of these large work items, identify a Service Level Expectation and use that for forecasting to a higher level of confidence than the estimates we had been using.

This seemed good in principal, however there were a few problems:

• It takes a long time to gather a decent sample size when each piece of work takes a month or more.
• Forecasting a roadmap is not a matter of sticking 85% SLEs one after the other. This is Monte Carlo territory and so should be prepared for a little scripting.
• If we decided that our SLE was a month we’d ask the team to reject work which they believed would take longer than that time. However engineers are eternally optimistic and a month is a long time. More often than not teams would believe they could complete the work.

I was quite lucky that while I was doing this Prateek was working on the new Applying Scaled Portfolio Kanban course for ProKanban and his new book Scaling Simplified. We got chatting and he gave me a lot to think about.

After a little bit of rework to follow Prateek’s advice I decided that the best approach was to right size the Product Goals by the number of User Stories they contained. So if our regular User Story SLE was that each item should be doable in 14 days or less our Product Goal would be rightsized to include 10 User Stories or less. We would ask teams refining the work to estimate how many pieces of work were required to complete the Product Goal, obviously this would just be an estimate but a 50% margin of error on 10 work items is significantly safer than on 50!

With my Product Goals estimated in terms of stories I could then run a variation on a regular Monte Carlo to forecast when each Product Goal would be completed.

The main changes which were required were to record completion dates for the individual Product Goals rather than only when the entire list of items completed. The other change was to allow Product Goals to be worked on simultaneously (a hundred Scrum Masters just took a great intake of breath, but please bear with me for the time being).

This let me run a slightly different Monte Carlo simulation:

So far so good, when communicating with stakeholders and customers we can give our 85% date and explain that we have a high level of confidence that work will be completed on or before this date. When discussing with other teams inside the business we can give a range from the 25% to the 85% to give the a range of dates when their support will be required. Obviously as the number of Completed Stories increases and the date moves forward the uncertainty (and hopefully the remaining work) decreases and those dates become closer and closer together.

There’s something else I’d like to highlight. It’s easier if I run a new simulation with equally sized Product Goals.

Do you see it?

There are 31 days between the 25% column and the 85% of Goal A, 43 days between the corresponding dates of Goal B, 52 for Goal C, and 60 for Goal D. In other words despite the amount of work required for Goal A being exactly the same as Goal B the range of likely dates (the dates between the 25% and 85% threshold) is significantly larger for the goals which are further away.

So why is this?

The answer is actually very intuitive. We are about to start on Goal A immediately and there are 10 pieces of work between us an the Product Goal being realised. However there are 40 items between us and the completion for Goal D. Because there is more work, there is more uncertainty and the range of realistic dates is broader.

However, there is good news here. In the example above I’m creating a forecast at quite long range. I can be 85% confident that Goal D will be delivered on or before the 24th of April 2024 (I’m sorry if you’re reading this in 2025). Now, lets fast forward a to the end of 2023, we’re now finished with goal A and B and have started on Goal C. At this point there may only be 18 outstanding pieces of work. Now there are only 18 items left and we will be able to make a much more accurate forecast, most likely some point in mid-March. Each time we complete preceeding work the uncertainty decreases and those long range forecasts become a little more accurate.

The final point I want to touch one was the comment about multiple Product Goals on the go at a time. Any Scrum Practitioner will tell you that we have a single future vision for the product and the team focus on moving towards it. So why introduce that WIP column? To answer that lets look at what happens when you take our 4 goal example and start changing things around.

This table shows you what happens to the 85% dates when you have four Product Goals and single focus on 1, or multi-task 2, 3, or 4 simultaneously. This is especially obvious because all the goals in this example are the same size however the sample principal applies when you’re working on differently sized items.

In short when you split your focus between two items and progress both it takes significantly longer to deliver both. The ordering of the work becomes less relevant and the size of the work takes over as the largest factor for how long it takes. This means you lose the benefit of prioritising your work in the first place!

This scales up until with four items and a WIP of 4 you’re likely to finish all at pretty much the same time. The same anomaly can result in smaller items lower down the backlog being completed before their larger cousins further up, even though they were prioritised lower.

When you graph these you can see the range of likely completion dates work getting shorter and wider the further out you go.

I don’t think there’s anything too surprising here. However, lets think about the impact of this. Our Product Owner works extremely hard to identify the work which would yield most value. These are ranked, so Goal A is more valuable than Goal B, than C, and so on. However, when we start focusing on two or more Product Goals (or at least two large pieces of work made up of multiple Stories) we are actively delaying the most important work in favour of lower value work. This could result in less value being delivered, lost revenue, and disappointed customers. Yet another reminder of why we should work towards a single vison of where we want to get to and then inspect and adapt our progress towards it reguarly.

That wraps up what I wanted to discuss around Product Goals. In summary I would recommend estimating how many stories make up the goals and right sizing these to limit the scope being vastly inaccurate. Use a customised Monte Carlo to track when each goal is likely to complete. Be aware that work further out will have far less confidence than work in the short term and if you want to ensure you deliver the most valuable work first then avoid multitasking larger projects.

Author - Adam Griffiths