Limitations of Business Intelligence

My predictive analytics model didn't foresee this outcome, therefore it can't be happening. With apologies to the makers of 2012.

Introduction

“So how come Business Intelligence didn’t predict the World Economic Crisis?”

I have seen countless variants of the above question posted all over the Internet. Mostly it is posed on community forums and can often be a case of someone playing Devil’s Advocate, or simply wanting to stir up a conversation. However I came across reference to this question recently in the supposedly more sober columns of The British Computer Society (now very modishly re branded as BCS – The Chartered Institute for IT). According to the font of all human knowledge, the BCS is:

“a professional body and a learned society that represents those working in Information Technology. Established in 1957, it is the largest United Kingdom-based professional body for computing”

The specific article was entitled Data quality issues ‘to blame for financial crises’ (I’m not sure whether the BCS is saying that data quality issues are responsible for more than one financial crisis, or whether there is a typo in the last word). The use of quotation marks is also apt as the BCS seem to be reliant for the content of this article on both the opinions of the owner of a on-line community and a piece of commercial research finding that:

“more than 75 per cent of top financial services firms are to increase the amount of money they allocate to combating data quality and consistency issues”

and

“a further 44 per cent said clarity of data would be their ‘key focus'”

How this adds up to the conclusion appearing in the title is perhaps something of a mystery. The process is not exactly a shining example of how to turn source data into actionable information.
 
 
Lessons from Lehmans

Theirs not to reason why,  Theirs but to do & die,  Into the valley of Death  Rode the six hundred

It is arguable (though maybe not on the evidence presented in the BCS article) that poor data quality may have contributed to the demise of say Lehman Brothers. However the following line of argument is a bit of a reach:

  1. Poor data quality [arguably] contributed to the failure of Lehman Brothers
  2. Lehman Brothers’ failure was a trigger for a broader collapse of the world economy
  3. Therefore Lehman’s collapse was solely to blame for the crisis
  4. Thus (as per the BCS): Data quality issues [are] ‘to blame for financial crises’ [sic.]

There are a number of problems with this logic. To address just one, the failure of Lehmans did not cause the recession, it precipitated problems that were much larger, had been building up for years and which would have been triggered by something sooner or later (all balloons either deflate or pop eventually, even if not pierced by a needle).

By way of analogy, thinking that the assassination of Archduke Ferdinand was the sole reason for the outbreak of The Great War would be an over-simplification of history; greater forces were at work. Does a dropped match [proximate cause] lead to a massive forest fire, or are the preceding months of drought [distal cause] more to blame, with the fire an accident waiting to happen?

To most observers the distal causes of the recession were separate bubbles that had built up in a variety of asset classes (e.g. residential property) that were either going to deflate slowly, or go bang! Leverage created by certain classes of financial instruments made a bang more likely, but these instruments themselves did not create the initial problems either.

Extending our earlier analogy, if the asset bubbles were a lack of rain, then maybe the use of financial instruments – such as collateralised debt obligations – was a drying wind. In this scenario, Lehman Brothers was the dropped match, nothing more. If it wasn’t them, it would have been another event. So for causes of the World Economic crisis, we need to look more broadly.
 
 
Cui culpa?

First published in September 1843 to take part in 'a severe contest between intelligence, which presses forward, and an unworthy, timid ignorance obstructing our progress' [nice use of the Oxford / Harvard comma BTW]

Before I explore whether BI should have performed better in predicting the most severe recession since the 1930s, it is perhaps worth asking a more pertinent question, namely, “so how come macroeconomics didn’t predict the World Economic Crisis?” Again according to the font:

macroeconomics is a branch of economics that deals with the performance, structure, behavior and decision-making of the entire economy, be that a national, regional, or the global economy

so surely it should have had something to say in advance about this subject. However at least according to The Economist (who one would assume should know something about the area):

[Certain leading economists] argue that [other] economists missed the origins of the crisis; failed to appreciate its worst symptoms; and cannot now agree about the cure. In other words, economists misread the economy on the way up, misread it on the way down and now mistake the right way out.

On the way up, macroeconomists were not wholly complacent. Many of them thought the housing bubble would pop or the dollar would fall. But they did not expect the financial system to break. Even after the seizure in interbank markets in August 2007, macroeconomists misread the danger. Most were quite sanguine about the prospect of Lehman Brothers going bust in September 2008.

Source: The Economist – 16th July 2009

[Note: a subscription to the magazine is required to view this article]

In a later article in the same journal, Robert Lucas, Professor of Economics at the University of Chicago, rebutted the above critique, stating:

One thing we are not going to have, now or ever, is a set of models that forecasts sudden falls in the value of financial assets, like the declines that followed the failure of Lehman Brothers in September. This is nothing new. It has been known for more than 40 years and is one of the main implications of Eugene Fama’s “efficient-market hypothesis”, which states that the price of a financial asset reflects all relevant, generally available information. If an economist had a formula that could reliably forecast crises a week in advance, say, then that formula would become part of generally available information and prices would fall a week earlier.

Source: The Economist – 6th August 2009

[Note: a subscription to the magazine is required to view this article]

So if economists had at best a mixed track record in predicting the crisis (and can’t seem to agree amongst themselves about the merits of different ways of analysing economies), then it seems to me that Business Intelligence has its work cut out for it. As I put it in an earlier article, The scope of IT’s responsibility when businesses go bad:

My general take is that if the people who were committing organisations to collateralised debt obligations and other even more esoteric asset-backed securities were unable (or unwilling) to understand precisely the nature of the exposure that they were taking on, then how could this be reflected in BI systems. Good BI systems reflect business realities and risk is one of those realities. However if risk is as ill-understood as it appears to have been in many financial organisations, then it is difficult to see how BI (or indeed it’s sister area of business analytics) could have shed light where the layers of cobwebs were so dense.

As an aside, the above-referenced article argues that IT professionals should not try to distance themselves too much from business problems. My basic thesis being that if IT is shy about taking any responsibility in bad times, it should not be surprised when its contributions are under-valued in good ones. However this way lies a more philosophical discussion.

My opinion on why questions about whether or not business intelligence predicted the recession continue to be asked is that they relate to BI being oversold. Oversold in a way that I believe is unhealthy and actually discredits the many benefits of the field.
 
 
Crystal Ball Gazing

One of these things is not like the others,  One of these things just doesn't belong,  Can you tell which thing is not like the others  By the time I finish my song?

The above slide is taken from my current deck. My challenge to the audience is to pick the odd-one-out from the list. Assuming that you buy into my Rubik’s Cube analogy for business intelligence, hopefully this is not an overly onerous task.

Business Intelligence is not a crystal ball, Predictive Analytics is not a crystal ball either. They are extremely useful tools – indeed I have argued many times before that BI projects can have the largest payback of any IT project – but they are not universal panaceas.

The Old Lady of Threadneedle Street is clearly not a witch
An inflation prediction from The Bank of England
Illustrating the fairly obvious fact that uncertainty increases in proportion to time from now.

Business Intelligence will never warn you of every eventuality – if something is wholly unexpected, how can you design tools to predict it? Statistical models will never give you precise answers to what will happen in the future – a range of outcomes, together with probabilities associated with each is the best you can hope for (see above). Predictive Analytics will not make you prescient, instead it can provide you with useful guidance, so long as you remember it is an prediction, not fact.

It is amazing the things that people find to do in their spare time, isn't it?

However, in most circumstances, the fact that your Swiss Army knife doesn’t have the highly-desirable “tool for removing stones from horses hooves” does not preclude it from fulfilling its more quotidian functions well. The fact that your car can’t do 0-60 mph (0-95 kph, or 0-26 ms-1 if you insist) in less than 4 seconds, does not mean that it is incapable of getting you around town perfectly happily. Tools should be fit-for-purpose, not all-purpose.

Unfortunately, sometimes business intelligence can be presented as capable of achieving the impossible; this is only going to lead to disillusionment with the area and to the real benefits not being seized. Also it is increasingly common for vendors and consultancies to claim that amazing results can be obtained with BI quickly, effortlessly and (most intoxicatingly) with minimum corporate pain. My view is that these claims are essentially bogus. Like most things in life, what you get out of business intelligence is highly connected with what you put it.

If you want some pretty pictures showing some easy to derive figures, then progress in days rather than months is entirely feasible. But if you want useful insights into your organisation’s performance that can lead to informed decision making, then time is required to work out what makes the company tick, how to best measure this to drive action and – a part that is often missed – to provide the necessary business and technical training to allow users to get the best out of tools. Here my experience is that there are few meaningful short-cuts.
 
 
Crystallising BI benefits

Adopting a more positive tone, if done well, then I believe that business intelligence can do a lot of great things for organisations. A brief selection of these includes:

  1. Dissect corporate performance in ways that enable underlying drivers to be made more plain (our drop-off in profitability is due to pricing pressures in Subsidiary A and poor retention of mid-sized accounts in Territory B, compounded by a fall in the rate of new business acquisition in Industry Segment C).
  2. Amalgamate data from disparate sources, allowing connections to be made between different, but related, areas (high turnover of staff in our customer services centre has coincided with both increased lead times for shipments and greater incidence of customer complaints)
  3. Give insights as to how customers are behaving and how they react to corporate initiatives (our smaller customers appear to be favouring bundled services, which include Feature W, however there was increased uptake of unbundled Service Z following on from our recently published video extolling its virtues)
  4. Measure the efficacy of business initiatives (was the launch of Product X successful? did our drive to improve service levels lead to better business retention?)
  5. Transparently monitor business unit achievement (Countries P, Q and R are all meeting their sales and profitability targets, howvever Country Q is achieving this with 2 fewer staff per $1m revenue)
  6. Provide indications (not guarantees) of future trends (sales of Service K are down 10% on this time last year and fell on a seasonally-adjusted basis for four of the last six months)
  7. Isolate hard-to-find relations (the biggest correlation with repeat business is the speed with which reported problems are addressed, not the number of problems that occur)

It is worth pointing out that a lot of the above is internally focussed, about the organisation itself and only tangentially related to the external environment in which it is operating. Some companies are successfully blending their internal BI with external market information, either derived from specialist companies, or sometimes from industry associations. However few companies are incorporating macroeconomic trends into their BI systems. Maybe that’s because of the confusion endemic in Economics that was referenced above.

However there is another reason why BI is not really in the business of predicting overall economic trends. In the preceding paragraphs, I have stressed that it takes lot of effort to get BI working well for a company. To have the same degree of benefit for a nation’s economy, you would have to aggregate across thousands of companies and deal with the same sort of inconsistency in data definitions and calculation methodologies that are hard enough to fight within an organisation; but orders of magnitude worse.

Nationwide (let alone global) BI would be a Herculean (and essentially impossible) task. Instead simplifying assumptions have to be made, and such assumptions do not generally lead to high-quality BI implementations; which are typically highly-tuned to the characteristics of individual organisations.
 
 
Leverage

"Give me a place to stand, and I shall move the earth"

There are of course organisations whose general profitability exceptionally depends on broad economic trends. These include the much maligned banks of varying flavours. The unique problem that many of these face is of leverage. While a 1% fall in economic activity might have a 1% impact on the revenues of a manufacturing company (in fact seldom is the relationship so simple), it might have a catastrophic impact on a bank, depending on how their portfolio is structured.

To look at the simplest form of option, which pays out the differential between the market price and a floor of £50. If conditions in the economy drive the share price from £55 to £50, the regular shareholder has lost 9% of their investment; the option holder has lost 100%. So while both the shareholder and option-holder will have an equal chance of experiencing such a price-fall, the impact on them will be radically different (in this case by 91%). Like BI, derivatives are a very useful tool, however they also need to be used appropriately.
 
 
Closing thoughts

You will notice an absent of fortune-telling from the above list of BI benefits. As indispensable as I believe good BI is to organisations of all shapes and sizes, if fortune-telling is your desire then my advice is to forswear BI and wait until this lady is next in town…

...though of course you may not be able to foretell when this will be
 


 
For readers who are interested in this area, I recommend Neil Raden’s artcile: Wherefore Analytics on Wall Street? An Homage to Hy Minsky.
 

 

9 thoughts on “Limitations of Business Intelligence

  1. Excellent article (as always) Peter,

    Ockham’s Razor, which translates directly as “entities must not be multiplied beyond necessity,” far more commonly as “the simplest answer is usually the correct answer,” and even more colloquially as “keep it simple, stupid,” illustrates how our general disdain of complexity often leads to our general preference for simplicity.

    For this reason, proximate causes are often mistaken for distal causes. The recent economic crisis has been blamed on many of the individual variables within what is a very complex equation.

    Business intelligence is indeed no crystal ball, and statistical probabilities are not factual certainties. However, short-term financial success has a wonderful way of clouding our perception of risk.

    Similar to how an understanding of mathematics, which can help us calculate the odds of losing a game of chance, is both conveniently forgotten while we are emotionally swept up in a winning streak and of no solace after a big loss or a sustained losing streak, business intelligence can be a forgotten ally in good times and an easy enemy in bad times.

    Business Intelligence is not Business Clairvoyance — but business is far less risky with it than without.

    Cheers,

    Jim

    • Good points Jim,

      To me BI is far more valuable with a proper understanding of what it both can and cannot do.

      Humans have been shown to be really bad at estimating certain probabilities. I guess that lotteries have not been around for long enough to exert sufficient evolutionary pressure for this to improve :-).

      Peter

  2. BI is built on the premise of timely reporting. Many BI solutions are run from stale data warehouses. Simple improvements in process can enable Real-Time Oracle BI that processes BI 24×7. Unlock the immediate business value buy creating a Report Server that houses BI, warehousing and Operational Reporting.

    Stream-line your intelligence :-) without fortune tellers. Deal with real-time facts to make important decisions. Findout how… http://bit.ly/aYPVNx.

    John

    • John,

      I don’t mind you linking back to quest.com – but it would be helpful if you cited your affiliation with the company – Sales Director right?

      Otherwise it comes across a little clumsy.

      Peter

  3. BI can be viewed from two limiting viewpoints. A maximum viewpoint and the other minimum one. From the maximum viewpoint BI is touted as a panacea for identification of principal business drivers and for taking optimum business decision. A minimum view point would sound more modest and would only claim that BI is essentially an interactive tool for viewing a business situation from many angles and at any user defined level of aggregation of the real business data. How good, how deep and how accurate view you would get depends on the quality of data and your ability to model the data appropriately.
    But it would be extremely naive to expect that BI would necessarily identify the causal factor(s) for any business event / fact / observed pattern. BI is always looking at the fact. This means it is backward looking. From the past we can only discover correalation, a coincident occurrence of two or more facts over many time points. That does not say anything about the causal direction.
    But what a proper BI can do and no BI tool does it today, is to help a BI user to update her prior view or understanding of the causal mechansim with the help of oberved fact. Today, the BI landscape is dominated by a data analysis paradigm that is totally unsuited to the data environment of corporate business world. First of-all mass of data avaialble for analysis today is huge and ordinary correlation analysis has very limited utility in such an environment. For example every BI tool will generate trend analysis of various business metrics – profitability, prductivity etc. A rising graph may be interpreted as vinidication of the startegy follwed. This is the worst kind of causal analysis one can think of. But as far as I know every BI tool vendor would gloat about their capability to create such graphs across many dimensions and drill down capability. Does the drill down capability necessarily help you to discover any causal mechanism. It may only help you to understand what actually has happened in a particular space-time coordinate.
    As of now all BI tools are glorified, albeit very effective, reporting tool and nothing else. It does not add to any intelligence whatsoever. We have to wait for such a tool to come in future.

    • Ashok,

      Thanks for the comment – my general thesis is that it is not about the tool, but what you use it for. Maybe having spent quite a bit of my career in Insurance, where the entire game is making estimations about the future based on the past, I have a slightly different perspective.

      Until someone develops a time machine, there is no alternative to looking backwards, all information (not just numeric, but any information whatsoever) is historical. Using the past as a guide to the future is the best we can hope to achieve.

      Peter

  4. […] In my introduction, I began by issuing my customary caveat about the danger of too blindly following any recipe for success. I then provided some background about my first major achievement in data warehousing and went on to present the general framework for success in BI/DW programmes that I developed as a result of this. In concluding the first part of the speech, I attempted to delineate the main benefits of BI and also touched on some of its limitations. […]

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s