Peter Thomas – Award-winning Business Intelligence and Cultural Transformation Expert

This post is based on some comments that I made on an article by Dorothy Miller at IT Business Edge. The title of Dorothy’s piece is Measuring the Return on Investment for Business Intelligence.
 
 
Introduction

In a nutshell, good BI is all about users taking better business decisions. There are other benefits, some of which I allude to in this article, but nothing outweighs the potentially massive impact of taking better decisions.

While you would think that this is a very positive scenario for BI projects, there is a fly in the ointment. BI doesn’t let users take better business decisions, only good BI does that. Which begs the question: how do you tell how good you BI is and whether it is providing the promised benefits? This difficult area is the subject of this short article.
 
 
Measuring BI payback directly

All of the various different ways of measuring return on investment, from the simplest to the most sophisticated, compare the amount of money you spend to the amount of money you gain. Establishing the first parameter for this equation should be relatively simple, determining the second one is anything but straightforward.

In its purest form, the observation that taking better decisions should result in increased profits is entirely sound. If a BI project is targeting areas at the core of a company’s business (and why would you be investing in BI if this is not the case) then there should be a positive impact on profitability. This might be through increased sales or other income (assuming that these are at a profitable price), through increased margins on sales, reduced expenses, or a combination of all three. Of course the ways that good BI can contribute to each of these areas are manifold, but each will come down essentially to one of these three things.

The problem is that many other externalities can impact each of these areas. Some industries are cyclical, the impact of competitors will vary over time, demand for products will be different at different times, the business mix will not stay constant, markets can appear and disappear very quickly sometimes. Also, often the impact of better business decisions will not be realised until some future date; this year’s decisions may impact next year’s profits, or a strategic investment decision may not bear fruit for many years. While good BI will help you to make sense of these trends, at the same time it is difficult to disentangle the impact of BI on results from them.

However, sometimes it is possible to discern the direct impact of BI and I will offer a few examples:

1. Improvements in what BI is measuring

Suppose an element of your BI system tracks the number of sales leads that an organisation has against the number of orders placed. Being able to slice and dice such numbers by territory, product, marketing campaign, channel, customer segment, customer and so on is valuable in managing the sales process. However the trending capabilities of BI will also allow you to compare before and after. If the BI tool is truly being effective, then it should be possible to show that proportion of orders to leads is greater after its implementation than before it. In fact there should be a steady upward curve post-implementation as the system beds in and also as users find more creative ways to employ the information it gathers. If this is not the case, then something is wrong.

2. What if analyses

Supposing that the BI system aims to better manage the portfolio of a business; i.e. the balance of products and/or services that it sells. Again the historical aspect of BI can help us to consider its impact. Consider the simplified example of an organisation that has just three products: A, B and C, with each making up a third of sales. A good BI system will be able to calculate profitability for each product. Inevitably, shifts in market conditions will cause these profitabilities to change over time. Our example company might react to an increase in the profitability of product A in such a way that its portfolio becomes A (50%), B (25%), C (25%) – the assumption here is that the information available in their BI system is the main catalyst for change. What is important again is that the BI system holds historical information. This should allow a what if calculation to be performed; in this case, what if the mix of products had been left at 33%/33%/33%? The difference between calculated profitability under this scenario and actual profitability is one measure of the impact of good BI.

[It is interesting to note that this approach can also be used to support the case for BI projects. One technique that I have successfully employed is to look at an organisation's results over a five-year period, use the first three years to develop a simple hypothesis for decision making (e.g. we should sell a product if it meets a set of criteria and not otherwise) and then use the last two to validate this. This type of study says something like: if we had had better information in the past, we would have taken different decisions with the following - hopefully positive - results. To make this a little clearer, the study I carried out showed that greater availability of even very basic BI would have led to profit margins doubling. I will cover this approach in more detail in a forthcoming article.]

3. Measurable productivity increases

Aside from enabling better decisions to be taken, good BI can save people time and effort, thereby increasing productivity. It is possible to measure such things. By both general survey and a specific analysis of tasks carried out, I was able to demonstrate that pre-BI implementation it could take 5-7 days to assemble all the information needed for a key account review. Post-BI implementation, this became a matter of minutes. Care should be taken in scaling-up these figures to overall time saved, but such studies can give a good indication of increased productivity and, if accompanied by some conservative cost estimates, can be presented in terms of actual monetary impact.

4. Direct comparison of profitability

In some companies and markets it may be possible to directly measure changes in profitability relating to the use of BI. Alternatively, it may be possible to adjust results to remove the impact of some external issues (e.g. where there is publicly available information about overall trends in the market). Sometimes the relationship between the implementation of good BI and corporate results will be so striking, that there will be no argument about their correlation. This is a situation that I have experienced myself.
 
 
Measuring BI payback via proxies

Notwithstanding the above points, the most likely scenario is that it will be challenging to discern the precise impact of a BI project. Given this, instead of giving up, it is important to consider any available proxies. Some of these are discussed in this section, all rest upon the assumption that business people are rational and will only use systems that add value, making their work easier or improving decision-making.

Many of these proxies are self-evident, so I won’t offer too much commentary on them.

5. User adoption

How many people use the system and do more people want to use it? How do these numbers change over time? What is penetration like in different areas of the organisation?

6. Actual usage

Of course this relates to the previous point as well, but directly measuring usage and even using your BI tool to analyse how this changes over time is important. If there is a correlation between how much a part of the organisation uses the system and how good its results are, so much the better.

7. User retention

Of the people who are given access to the system and trained in its use, how many go on to become regular users? How does this change over time?

8. Demand for enhancements / extension to the system

If you have people wanting the system to do more, then they must be happy with how it is working in general terms.

9. Feedback from surveys

It always helps to get feedback on what you are doing. Excerpts from one such survey appear here.

10. Do business users mention the system in meetings?

When presenting figures to senior management, is the source quoted as a matter of course (hopefully to establish that the figures are reliable)?
 
 
Summary

This article has argued that establishing the benefits of BI can be difficult, but that it is by no means impossible. There are a range of techniques available to either directly or indirectly assess its impact. Of course there are probably other creative ways to do this that other organisations are employing and which I have not mentioned.

Business Intelligence practitioners should pay especial attention to this area, it is an opportunity to demonstrate that the yields from BI projects can be substantial. In fact, it is my opinion that in many industries no other type of IT project will have greater payback than BI. It should be a priority for those who tout the benefits of BI to show that there is real substance behind these claims. My experience suggests that it is definitely worth taking the time and effort to prove this convincingly.
 

	Tweet this article on twitter.com

Bookmark this article with:
Technorati	| del.icio.us	| digg	| Reddit	| NewsVine