While spring cleaning at home at the weekend, I came across a DVD of an interview I did for Informatica back in March 2005. This is still accessible on the Informatica web-site and appears in my video library, but I thought that I had lost my copy of the original.
Having made this discovery, I added it to my selection of videos on YouTube.com.
Disclosure – Part I: In the work I refer to above, I leveraged Infomatica’s toolset (PowerCentre) alongside software from Oracle (RDBMS and PL/SQL), IBM Cognos (PowerPlay and ReportStudio) and Microsoft (.NET). I have used tools from other vendors in other projects. While there is clearly a promotional sub-text to the video, it is not a product endorsement and I believe that my comments are generally applicable to any business intelligence / data warehousing project.
Disclosure – Part II: I have already had it pointed out to me – by @ocdqblog and others – that the braces (suspenders if you are from the US; suspenders having quite a different connotation in the UK) were perhaps something of a fashion faux pas. My American partner has long since despaired of my British approach to “co-ordination” of patterns. You may be glad to know that I no longer own the offending item.
One of the benefits of the WordPress.com platform is that you can get some indication as to which other parts the the web are directing traffic your way. It was via this facility that I came across an article on Microsoft‘s site linking back to my piece Measuring the benefits of Business Intelligence. The title, sub-title and authorship of the Microsoft post is as follows:
How to Measure BI Value
A thorough assessment will help you demonstrate the effectiveness of your BI investments. We offer 8 factors to consider.
As always, my aim in writing this column is to remain vendor-neutral, however the Microsoft piece is not specifically pushing their BI products (though clearly further information about them is only a click away), but rather offering some general commentary.
Again it is interesting to note the penetration of social media (such as this blog) into mainstream technology business.
On 29th April 2010, I will be speaking at this event, along with Sarah Burnett (Sarah’s blog, @sarahburnett) and Madan Sheina. The seminar is entitled Business Intelligence – Effective Strategies for Successful Deployment and you can find further details here.
Ovum provides clients with independent and objective analysis that enables them to make better business and technology decisions.
I have been a regular visitor to Merv Adrian’s excellent blog since just after its inception and have got to know Merv virtually via twitter (@merv) and other channels. I recently read his article : Oracle Ups EPM Ante, which covered Oracle’s latest progress in integrating its various in-house and acquired technologies in the Enterprise Performance Management and Business Intelligence arenas.
The article is clearly written and helpful, I recommend you take a look if these areas impinge upon you. One section caught my attention (my emphasis):
Finally, Oracle has long had a sizable base in government, and its new Hyperion Public Sector Planning and Budgeting app suite continues the integration theme, tapping its ERP apps (both Oracle E-Business Suite [EBS] and PeopleSoft ERP) for bidirectional feeds.
My current responsibilities include EPM, BI and the third Oracle ERP product, JD Edwards. I don’t work in the public sector, but was nevertheless interested in the concept of how and whether JDE fitted into the above scenario. I posted a comment and within a few hours Merv replied, having spoken to his senior Oracle contacts. The reply was from a vendor-neutral source, but based on information “straight from the horse’s mouth”. It is illuminating to ponder how I could have got a credible answer to this type of question any quicker.
To recap, my interactions with Merv are via the professional social media Holy Trinity of blogs, twitter.com and LinkedIn.com. The above is just one small example of how industry experts can leverage social media to get their message across, increase their network of influence and deliver very rapid value. I can only see these types of interactions increasing in the future. Sometimes social media can be over-hyped, but in the world of industry analysis it seems to be a marriage made in heaven.
Analyst and consultant Merv Adrian founded IT Market Strategy after three decades in the IT industry. During his tenure as Senior Vice President at Forrester Research, he was responsible for all of Forrester’s technology research, covered the software industry and launched Forrester’s well-regarded practice in Analyst Relations. Earlier, as Vice President at Giga Information Group, Merv focused on facilitating collaborative research and served as executive editor of the monthly Research Digest and weekly GigaFlash.
Prior to becoming an analyst, Merv was Senior Director, Strategic Marketing at Sybase, where he also held director positions in data warehouse marketing and analyst relations. Prior to Sybase, Merv served as a marketing manager at Information Builders, where he founded and edited a technical journal and a marketing quarterly, subsequently becoming involved in corporate and product marketing and launching a formal AR role.
My blog attempts to stay vendor-neutral, but much of Bruno’s article is also in the same vein; aside from the banner appearing at the top of course. It is noteworthy how many of the big players are realising that engaging with the on-line community in a sotto voce manner is probably worth much more than a fortissimo sales pitch. This approach was also notable in another output from the BI stable at Microsoft; Nic Smith’s “History of Business Intelligence” , which I reviewed in March 2009. However, aside from these comments I’ll focus more on what Bruno says than on who he works for; and what he says is interesting.
His main thesis is that good BI can “sharpen competitive skills […] turning competitive insights into new ways to do business”. I think that it is intriguing how some organisations, ideally having already got their internal BI working well, are now looking to squeeze even further value out of their BI platform by incorporating more outward-looking information; information relating to their markets, their customers and their competitors. This was the tenth BI trend I predicted in another article from March 2009. However, I can’t really claim to be all that prescient as this development seems pretty common-sensical to me.
Setting the bar higher
Competition between companies is generally seen as a positive thing – one reason that there is so much focus on anti-trust laws at present. Competition makes the companies involved in it (or at least those that survive) healthier, their products more attuned to customer needs, their services more apt. It also tends to deliver better value and choice to customers and thus in aggregate drives overall economic well-being (though of course it can also generate losers).
In one of my my earliest blog articles, Business Intelligence and Transparency, I argued that good BI could also drive healthy internal competition by making the performance of different teams and individuals more accessible and comparable (not least to the teams and individuals themselves). My suggestion was that this would in turn drive a focus on relative performance, rather than settling for absolute performance. The latter can lead to complacency, the former ensures that the bar is always reset a little higher. Although this might seem potentially divisive at first, my experience of it in operation was that it led to a very positive corporate culture.
Although organisations in competition with each other are unlikely to share benchmarks in the same way as sub-sections of a single organisation, it is often possible to glean information from customers, industry associations, market research companies, or even the published accounts of other firms. Blended with internal data, this type of information can form a powerful combination; though accuracy is something that needs to be born in mind even more than with data that is subject to internal governance.
A new source of competitive advantage
Bruno’s suggestion is that the way that companies leverage commonly available information (say Governmental statistics) and combine this with their own numbers is in itself a source of competitive advantage. I think that there is something important here. One of the plaudits laid at the feet of retail behemonth Wal Mart is that it is great at leveraging the masses of data collected in its stores and using this in creative ways; ways that some of its competition cannot master to the same degree.
In recent decades a lot of organisations have attempted to define their core competencies and then stick to these. Maybe a competency in generating meaningful information from both internal and external sources and then – crucially – using this to drive different behaviours, is something that no self-respecting company should be without in the 2010s.
You can follow Bruno on twitter.com at @brunoaziza
Strong business sponsorship is generally cited as a major success factor for IT projects. From one perspective this is essentially a truism, but looking at the phrase from a different angle perhaps reveals something of interest – indeed perhaps it highlights a reason for some IT projects failing. Let’s look at a definition to start with:
sponsor/spónsər/ n. & v. • n.2 a a person or organization that promotes or supports an artistic or sporting activity etc. (O.E.D.)
There are other definitions, but maybe surprisingly the one I show above is probably the closest to the meaning of “business sponsorship”. The very first entry in my Oxford English Dictionary for this word is one that brings back memories:
sponsor/spónsər/ n. & v. • n.1 a person who supports an activity done for charity by pledging money in advance. (O.E.D.)
This takes me back to school (a long time ago) when every year we had a sponsored 20 mile (32 km) walk around the streets of London, each time for a different charity. In an age before such events became mainstream, I believe we held some record for the amount of money raised. It is surprising how many hills you can fit into 20 miles, even in London, and I can well remember how tired I was after doing this as an eleven-year-old.
I can also recall wandering from house-to-house in my neighbourhood, knocking on doors with my sponsorship form to ask for pledges. As a rather naive child I never really understood why some people were occasionally a little disgruntled to have me appear on their doorstep at 9am on a Sunday. Of course, post walk, I had to do the same rounds again to collect the money. I escapes me how much I raised, several hundred pounds I think, but I also remember some people raising a lot more than that.
Both of the above definitions have the connotation of a kindly benefactor indulging a pet cause, be that the arts, or a small schoolboy. There is also the sense that the sponsor is vicariously involved, no one is asking them to play a recital, or to walk 20 miles. Perhaps here we begin to detect the seed of a problem.
When I read IT people on various on-line forums speaking about ensuring business sponsorship, or gaining business buy in, I get the strong impression of an idea originated in IT which is seeking support. Some of the recent discussions on LinkedIn.com, which formed the basis for my earlier article: Who should be accountable for data quality? are a case in point. Several contributors have made comments along the lines of “IT needs to educate the business about the importance of data quality” – as well as being rather patronising, I think that this perspective on business life is rather wrong-headed.
In my mind it takes me back to an IT colleague (at which company I will not mention) saying “of course we [i.e. IT people] are so much smarter than them [i.e. non-IT people]”. To this day I am unsure whether he was joking or not. In my experience, IT people are just like non-IT people, some are smart, some are not, most are somewhere in between – I suspect the distribution is pretty similar in both cases.
So when people talk about business sponsorship, maybe this is code for convincing the paymasters that some of IT’s ideas are worth spending money on. Maybe it is the same as a penniless 18th century musician seeking the indulgence of a feudal monarch. IT may have all of the tunes, but he who pays the piper…
On the other hand, if IT and non-IT were well-aligned then maybe it would be more of a case of the business seeking IT sponsorship; i.e. of business folk originating ideas and IT working out how to implement them. Of course I tend to be an advocate of a partnership approach. I read recently on a LinkedIn.com thread about some IT departments being active and others passive. I would recommend IT being active, but not in the sense of pursuing its own agenda, or feeling (as perhaps my IT colleague did) that it knows best.
Maybe instead of seeking business sponsorship – which sounds rather like what you would do after IT had already figured out what to do and why – it would make sense to seek business engagement much earlier in the piece – this would hopefully lead to jointly crafted approaches that have business support baked-in from the outset. Surely this is preferable to the corporate equivalent of going door-to-door soliciting money, no matter how noble the cause might appear the the IT person who originated it.
I expanded on this theme to include it in my presentation slide deck and have been playing about with various PowerPoint to video technologies with the following result (note: the video has no sound):
Apologies for the poor fidelity, maybe I should paid for a conversion tool, rather than using a free one.
PS The function in question above is √x log x, which is of importance in Number Theory; specifically in Koch’s approximation that (assuming the Riemann hypothesis):
On a less mathematical note, you can see me in some rather higher quality videos here.
By way of [very necessary] explanation, this post is a response to an idea started on the blog of Curt Monash (@CurtMonash), doyen of software industry analysts. You can read the full article here. This is intended as an early April Fools celebration.
A summary:
[…] the Rules of the No-Fooling Meme are:
Rule 1: Post on your blog 1 or more surprisingly true things about you,* plus their explanations. I’m starting off with 10, but it’s OK to be a lot less wordy than I’m being. I suggest the following format:
A noteworthy capsule sentence. (Example: “I was not of mortal woman born.”)
A perfectly reasonable explanation. (Example: “I was untimely ripped from my mother’s womb. In modern parlance, she had a C-section.”)
Rule 2: Link back to this post. That explains what you’re doing.
Rule 3: Drop a link to your post into the comment thread. That will let people who check here know that you’ve contributed too.
Rule 4: Ping 1 or more other people encouraging them to join in the meme with posts of their own.
*If you want to relax the “about you” part, that’s fine too.
I won’t be as dramatic as Curt, nor will I drop any names (they have been changed to protect the guilty). I also think that my list is closer to a “things you didn’t know about me” than Curt’s original intention, but hopefully it is in the spirit of his original post. I have relaxed the “about me” part for one fact as well, but claim extenuating circumstances.
My “no-fooling” facts are, in (broadly) reverse chronological order:
I have recently corrected a Physics paper in Science – and please bear in mind that I was a Mathematician not a Physicist; I’m not linking to the paper as the error was Science’s fault not the scientists’ and the lead author was very nice about it.
My partner is shortly going to be working with one of last year’s Nobel Laureates at one of the world’s premier research institues – I’m proud, so sue me!
My partner, my eldest son and I have all attended (or are attending) the same University – though separated by over 20 years.
The same University awarded me 120% in my MSc. Number Theory exam – the irony of this appeals to me to this day; I was taught Number Theory by a Fields Medalist; by way of contrast, I got a gamma minus in second year Applied Mathematics.
Not only did I used to own a fan-site for a computer game character, I co-administered a universal bulletin board (yes I am that old) dedicated to the same character – even more amazingly, there were female members!
As far as I can tell, my code is still part of the core of software that is used rather widely in the UK and elsewhere – though I suspect that a high percentage of it has succumbed to evolutionary pressures.
I have recorded an eagle playing golf – despite not being very good at it and not playing at all now.
I have played cricket against the national teams of both Zimbabwe (in less traumatic times) and the Netherlands – Under 15s and Under 19s respectively; I have also played both with and against an England cricketer and against a West Indies cricketer (who also got me out), but I said that I wasn’t going to name drop.
[Unlike Curt] I only competed in one chess tournament – I came fourth, but only after being threatened with expulsion over an argument to do with whether I had let go of a bishop for a nanosecond; I think I was 11 at the time.
At least allegedly, one of my antecedents was one of the last hangmen in England – I’m not sure how you would go about substantiating this fact as they were meant to be sworn to secrecy; equally I’m not sure that I would want to substantiate it.
And a bonus fact (which could also be seen as oneupmanship vis à vis Curt):
One of the articles that I wrote for the UK climbing press has had substantially more unique views than any of my business-related articles on here (save for the home page itself) – sad, but true, if you don’t believe me, the proof is here.
“So how come Business Intelligence didn’t predict the World Economic Crisis?”
I have seen countless variants of the above question posted all over the Internet. Mostly it is posed on community forums and can often be a case of someone playing Devil’s Advocate, or simply wanting to stir up a conversation. However I came across reference to this question recently in the supposedly more sober columns of The British Computer Society (now very modishly re branded as BCS – The Chartered Institute for IT). According to the font of all human knowledge, the BCS is:
“a professional body and a learned society that represents those working in Information Technology. Established in 1957, it is the largest United Kingdom-based professional body for computing”
The specific article was entitled Data quality issues ‘to blame for financial crises’ (I’m not sure whether the BCS is saying that data quality issues are responsible for more than one financial crisis, or whether there is a typo in the last word). The use of quotation marks is also apt as the BCS seem to be reliant for the content of this article on both the opinions of the owner of a on-line community and a piece of commercial research finding that:
“more than 75 per cent of top financial services firms are to increase the amount of money they allocate to combating data quality and consistency issues”
and
“a further 44 per cent said clarity of data would be their ‘key focus'”
How this adds up to the conclusion appearing in the title is perhaps something of a mystery. The process is not exactly a shining example of how to turn source data into actionable information.
Lessons from Lehmans
It is arguable (though maybe not on the evidence presented in the BCS article) that poor data quality may have contributed to the demise of say Lehman Brothers. However the following line of argument is a bit of a reach:
Poor data quality [arguably] contributed to the failure of Lehman Brothers
Lehman Brothers’ failure was a trigger for a broader collapse of the world economy
Therefore Lehman’s collapse was solely to blame for the crisis
Thus (as per the BCS): Data quality issues [are] ‘to blame for financial crises’ [sic.]
There are a number of problems with this logic. To address just one, the failure of Lehmans did not cause the recession, it precipitated problems that were much larger, had been building up for years and which would have been triggered by something sooner or later (all balloons either deflate or pop eventually, even if not pierced by a needle).
By way of analogy, thinking that the assassination of Archduke Ferdinand was the sole reason for the outbreak of The Great War would be an over-simplification of history; greater forces were at work. Does a dropped match [proximate cause] lead to a massive forest fire, or are the preceding months of drought [distal cause] more to blame, with the fire an accident waiting to happen?
To most observers the distal causes of the recession were separate bubbles that had built up in a variety of asset classes (e.g. residential property) that were either going to deflate slowly, or go bang! Leverage created by certain classes of financial instruments made a bang more likely, but these instruments themselves did not create the initial problems either.
Extending our earlier analogy, if the asset bubbles were a lack of rain, then maybe the use of financial instruments – such as collateralised debt obligations – was a drying wind. In this scenario, Lehman Brothers was the dropped match, nothing more. If it wasn’t them, it would have been another event. So for causes of the World Economic crisis, we need to look more broadly.
Cui culpa?
Before I explore whether BI should have performed better in predicting the most severe recession since the 1930s, it is perhaps worth asking a more pertinent question, namely, “so how come macroeconomics didn’t predict the World Economic Crisis?” Again according to the font:
macroeconomics is a branch of economics that deals with the performance, structure, behavior and decision-making of the entire economy, be that a national, regional, or the global economy
so surely it should have had something to say in advance about this subject. However at least according to The Economist (who one would assume should know something about the area):
[Certain leading economists] argue that [other] economists missed the origins of the crisis; failed to appreciate its worst symptoms; and cannot now agree about the cure. In other words, economists misread the economy on the way up, misread it on the way down and now mistake the right way out.
On the way up, macroeconomists were not wholly complacent. Many of them thought the housing bubble would pop or the dollar would fall. But they did not expect the financial system to break. Even after the seizure in interbank markets in August 2007, macroeconomists misread the danger. Most were quite sanguine about the prospect of Lehman Brothers going bust in September 2008.
[Note: a subscription to the magazine is required to view this article]
In a later article in the same journal, Robert Lucas, Professor of Economics at the University of Chicago, rebutted the above critique, stating:
One thing we are not going to have, now or ever, is a set of models that forecasts sudden falls in the value of financial assets, like the declines that followed the failure of Lehman Brothers in September. This is nothing new. It has been known for more than 40 years and is one of the main implications of Eugene Fama’s “efficient-market hypothesis”, which states that the price of a financial asset reflects all relevant, generally available information. If an economist had a formula that could reliably forecast crises a week in advance, say, then that formula would become part of generally available information and prices would fall a week earlier.
[Note: a subscription to the magazine is required to view this article]
So if economists had at best a mixed track record in predicting the crisis (and can’t seem to agree amongst themselves about the merits of different ways of analysing economies), then it seems to me that Business Intelligence has its work cut out for it. As I put it in an earlier article, The scope of IT’s responsibility when businesses go bad:
My general take is that if the people who were committing organisations to collateralised debt obligations and other even more esoteric asset-backed securities were unable (or unwilling) to understand precisely the nature of the exposure that they were taking on, then how could this be reflected in BI systems. Good BI systems reflect business realities and risk is one of those realities. However if risk is as ill-understood as it appears to have been in many financial organisations, then it is difficult to see how BI (or indeed it’s sister area of business analytics) could have shed light where the layers of cobwebs were so dense.
As an aside, the above-referenced article argues that IT professionals should not try to distance themselves too much from business problems. My basic thesis being that if IT is shy about taking any responsibility in bad times, it should not be surprised when its contributions are under-valued in good ones. However this way lies a more philosophical discussion.
My opinion on why questions about whether or not business intelligence predicted the recession continue to be asked is that they relate to BI being oversold. Oversold in a way that I believe is unhealthy and actually discredits the many benefits of the field.
Crystal Ball Gazing
The above slide is taken from my current deck. My challenge to the audience is to pick the odd-one-out from the list. Assuming that you buy into my Rubik’s Cube analogy for business intelligence, hopefully this is not an overly onerous task.
Business Intelligence is not a crystal ball, Predictive Analytics is not a crystal ball either. They are extremely useful tools – indeed I have argued many times before that BI projects can have the largest payback of any IT project – but they are not universal panaceas.
An inflation prediction from The Bank of England Illustrating the fairly obvious fact that uncertainty increases in proportion to time from now.
Business Intelligence will never warn you of every eventuality – if something is wholly unexpected, how can you design tools to predict it? Statistical models will never give you precise answers to what will happen in the future – a range of outcomes, together with probabilities associated with each is the best you can hope for (see above). Predictive Analytics will not make you prescient, instead it can provide you with useful guidance, so long as you remember it is an prediction, not fact.
However, in most circumstances, the fact that your Swiss Army knife doesn’t have the highly-desirable “tool for removing stones from horses hooves” does not preclude it from fulfilling its more quotidian functions well. The fact that your car can’t do 0-60 mph (0-95 kph, or 0-26 ms-1 if you insist) in less than 4 seconds, does not mean that it is incapable of getting you around town perfectly happily. Tools should be fit-for-purpose, not all-purpose.
Unfortunately, sometimes business intelligence can be presented as capable of achieving the impossible; this is only going to lead to disillusionment with the area and to the real benefits not being seized. Also it is increasingly common for vendors and consultancies to claim that amazing results can be obtained with BI quickly, effortlessly and (most intoxicatingly) with minimum corporate pain. My view is that these claims are essentially bogus. Like most things in life, what you get out of business intelligence is highly connected with what you put it.
If you want some pretty pictures showing some easy to derive figures, then progress in days rather than months is entirely feasible. But if you want useful insights into your organisation’s performance that can lead to informed decision making, then time is required to work out what makes the company tick, how to best measure this to drive action and – a part that is often missed – to provide the necessary business and technical training to allow users to get the best out of tools. Here my experience is that there are few meaningful short-cuts.
Crystallising BI benefits
Adopting a more positive tone, if done well, then I believe that business intelligence can do a lot of great things for organisations. A brief selection of these includes:
Dissect corporate performance in ways that enable underlying drivers to be made more plain (our drop-off in profitability is due to pricing pressures in Subsidiary A and poor retention of mid-sized accounts in Territory B, compounded by a fall in the rate of new business acquisition in Industry Segment C).
Amalgamate data from disparate sources, allowing connections to be made between different, but related, areas (high turnover of staff in our customer services centre has coincided with both increased lead times for shipments and greater incidence of customer complaints)
Give insights as to how customers are behaving and how they react to corporate initiatives (our smaller customers appear to be favouring bundled services, which include Feature W, however there was increased uptake of unbundled Service Z following on from our recently published video extolling its virtues)
Measure the efficacy of business initiatives (was the launch of Product X successful? did our drive to improve service levels lead to better business retention?)
Transparently monitor business unit achievement (Countries P, Q and R are all meeting their sales and profitability targets, howvever Country Q is achieving this with 2 fewer staff per $1m revenue)
Provide indications (not guarantees) of future trends (sales of Service K are down 10% on this time last year and fell on a seasonally-adjusted basis for four of the last six months)
Isolate hard-to-find relations (the biggest correlation with repeat business is the speed with which reported problems are addressed, not the number of problems that occur)
It is worth pointing out that a lot of the above is internally focussed, about the organisation itself and only tangentially related to the external environment in which it is operating. Some companies are successfully blending their internal BI with external market information, either derived from specialist companies, or sometimes from industry associations. However few companies are incorporating macroeconomic trends into their BI systems. Maybe that’s because of the confusion endemic in Economics that was referenced above.
However there is another reason why BI is not really in the business of predicting overall economic trends. In the preceding paragraphs, I have stressed that it takes lot of effort to get BI working well for a company. To have the same degree of benefit for a nation’s economy, you would have to aggregate across thousands of companies and deal with the same sort of inconsistency in data definitions and calculation methodologies that are hard enough to fight within an organisation; but orders of magnitude worse.
Nationwide (let alone global) BI would be a Herculean (and essentially impossible) task. Instead simplifying assumptions have to be made, and such assumptions do not generally lead to high-quality BI implementations; which are typically highly-tuned to the characteristics of individual organisations.
Leverage
There are of course organisations whose general profitability exceptionally depends on broad economic trends. These include the much maligned banks of varying flavours. The unique problem that many of these face is of leverage. While a 1% fall in economic activity might have a 1% impact on the revenues of a manufacturing company (in fact seldom is the relationship so simple), it might have a catastrophic impact on a bank, depending on how their portfolio is structured.
To look at the simplest form of option, which pays out the differential between the market price and a floor of £50. If conditions in the economy drive the share price from £55 to £50, the regular shareholder has lost 9% of their investment; the option holder has lost 100%. So while both the shareholder and option-holder will have an equal chance of experiencing such a price-fall, the impact on them will be radically different (in this case by 91%). Like BI, derivatives are a very useful tool, however they also need to be used appropriately.
Closing thoughts
You will notice an absent of fortune-telling from the above list of BI benefits. As indispensable as I believe good BI is to organisations of all shapes and sizes, if fortune-telling is your desire then my advice is to forswear BI and wait until this lady is next in town…