Standard note: You need to be a member of both LinkedIn.com and the group mentioned to view the discussions.
The case for a CBIO
I won’t republish all of John’s initial post, but for those who cannot access the thread these are the essential points that he raised:
There is an ever-increasing need for more and better information in organisations
Increasingly Business Intelligence is seen as a major source of competitive advantage
A CBIO would bring focus and (more importantly) accountability to this area
The CBIO should report directly to the CEO, with strong relations with the rest of the executive team
The CBIO’s team would be a hybrid business / technical one (as I strongly believe the best BI teams should be)
This team should also be at the forefront of driving change, based on the metrics that it generates
Now obviously creating a senior role with a portfolio spanning BI and change is going to be music that falls sweetly on my ears. I did however attempt to be objective in my response, which I reproduce in full below:
As someone who is (primarily) a BI professional, then of course my response could be viewed as entirely self-serving. Nevertheless, I’ll offer my thoughts.
In the BI programmes that I have run, I have had reporting lines into people such as the CIO, CFO or sometimes a combined IT / Operations lead. However (and I think that this is a big however), I have always had programme accountability to the CEO and have always had the entire senior leadership team (business and service departments) as my stakeholders. Generally my direction has come more from these dotted lines than from the solid ones – as you would hope would be the case in any customer-centric IT area.
I have run lots of different IT projects over the years. Things such as: building accounting, purchasing and sales systems; configuring and implementing ERP systems; building front-end systems for underwriters, marketing and executive teams; and so on. Given this background, there is definitely something about BI that makes it different.
Any IT system must be aligned to its users’ needs, that much is obvious. However with BI it goes a long way beyond alignment. In a very real sense, BI systems need to be the business. They are not there to facilitate business transactions, they are there to monitor the heartbeat of the organisation, to help it navigate the best way forward, to get early warning of problems, to check the efficacy of strategies and provide key input to developing them.
In short a good BI system should be focussed on precisely the things that the senior leadership team is focussed on, and in particular what the CEO is focussed on. In order to achieve this you need to understand what makes the business tick and you need to move very close to it. This proximity, coupled with the fact that good BI should drip business value means that I have often felt closer to the overall business leadership team than the IT team.
Please don’t misunderstand my point here. I have been an IT person for 20 years and I am not saying that BI should not be fully integrated with the overall IT strategy – indeed in my book it should be central to it as a major function of all IT systems is to gather information (as well as to support transactions and facilitate interactions with customers). However, there is something of a sense in which BI straddles the IT and business arenas (arenas that I have long argued should be much less distinct from each other than they are in many organisations).
The potentially massive impact of BI, the fact that it speaks the language of business leaders, the need for it to be aligned with driving cultural change and that the fact that the skills required for success in BI are slightly different for those necessary in normal IT projects all argue that something like a CBIO position is maybe not such a bad idea.
Indeed I have begun to see quite a few BI roles that are part of change directorates, or the office of the CEO or CFO. There are also some stand-alone BI roles out there, reporting directing to the board. Clearly there will always be a strong interaction with IT, but perhaps you have detected an emerging trend.
I suppose a shorter version of the above would run something like: my de facto reporting line in BI programmes has always been into the CEO and senior management team, so why not recognise this by making it a de jura reporting line.
BI is a weird combination of being both a specialist and generalist area. Generalist in needing to play a major role in running all aspects of the business, specialist in the techniques and technologies that are key to achieving this.
Over to the jury
Maybe the idea of a CBIO is one whose time has come. I would be interested in people’s views on this.
As might be inferred from my last post, certain sporting matters have been on my mind of late. However, as is becoming rather a theme on this blog, these have also generated some business-related thoughts.
Introduction
On Friday evening, the Australian cricket team finished the second day of the second Test Match on a score of 152 runs for the loss of 8 (out of 10) first innings wickets. This was still 269 runs behind the England team‘s total of 425.
In scanning what I realise must have been a hastily assembled end-of-day report on the web-site of one of the UK’s leading quality newspapers, a couple are glaring errors stood out. First, the Australian number 4 batsman Michael Hussey was described as having “played-on” to a delivery from England’s shy-and-retiring Andrew Flintoff. Second, the journalist wrote that Australia’s number six batsman, Marcus North, had been “clean-bowled” by James Anderson.
I appreciate that not all readers of this blog will be cricket aficionados and also that the mysteries of this most complex of games are unlikely to be made plain by a few brief words from me. However, “played on” means that the ball has hit the batsman’s bat and deflected to break his wicket (or her wicket – as I feel I should mention as a staunch supporter of the all-conquering England Women’s team, a group that I ended up meeting at a motorway service station just recently).
By contrast, “clean-bowled” means that the ball broke the batsman’s wicket without hitting anything else. If you are interested in learning more about the arcane rules of cricket (and let’s face it, how could you not be interested) then I suggest taking a quick look here. The reason for me bothering to go into this level of detail is that, having watched the two dismissals live myself, I immediately thought that the journalist was wrong in both cases.
It may be argued that the camera sometimes lies, but the cricinfo.com caption (whence these images are drawn) hardly ever does. The following two photographs show what actually happened:
Michael Hussey leaves one and is bowled, England v Australia, 2nd Test, Lord's, 2nd day, July 17, 2009
Marcus North drags James Anderson into his stumps, England v Australia, 2nd Test, Lord's, 2nd day, July 17, 2009
As hopefully many readers will be able to ascertain, Hussey raised his bat aloft, a defensive technique employed to avoid edging the ball to surrounding fielders, but misjudged its direction. It would be hard to “play on” from a position such as he adopted. The ball arced in towards him and clipped the top of his wicket. So, in fact he was the one who was “clean-bowled”; a dismissal that was qualified by him having not attempted to play a stroke.
North on the other hand had been at the wicket for some time and had already faced 13 balls without scoring. Perhaps in frustration at this, he played an overly-ambitious attacking shot (one not a million miles from a baseball swing), the ball hit the under-edge of his horizontal bat and deflected down into his wicket. So it was North, not Hussey, who “played on” on this occasion.
So, aside from saying that Hussey had been adjudged out “handled the ball” and North dismissed “obstructed the field” (two of the ten ways in which a batsman’s innings can end – see here for a full explanation), the journalist in question could not have been more wrong.
As I said, the piece was no doubt composed quickly in order to “go to press” shortly after play had stopped for the day. Maybe these are minor slips, but surely the core competency of a sports journalist is to record what happened accurately. If they can bring insights and colour to their writing, so much the better, but at a minimum they should be able to provide a correct description of events.
Everyone makes mistakes. Most of my blog articles contain at least one typographical or grammatical error. Some of them may include errors of fact, though I do my best to avoid these. Where I offer my opinions, it is possible that some of these may be erroneous, or that they may not apply in different situations. However, we tend to expect professionals in certain fields to be held to a higher standard.
For a molecular biologist, the difference between a 0.20 micro-molar solution and a 0.19 one may be massive. For a team of experimental physicists, unbelievably small quantities may mean the difference between confirming the existence of the Higgs Boson and just some background noise.
In business, it would be unfortunate (to say the least) if auditors overlooked major assets or liabilities. One would expect that law-enforcement agents did not perjure themselves in court. Equally politicians should never dissemble, prevaricate or mislead. OK, maybe I am a little off track with the last one. But surely it is not unreasonable to expect that a cricket journalist should accurately record how a batsman got out.
Twitter and Truth
I made something of a leap from these sporting events to the more tragic news of Michael Jackson’s recent demise. I recall first “hearing” rumours of this on twitter.com. At this point, no news sites had much to say about the matter. As the evening progressed, the self-styled celebrity gossip site TMZ was the first to announce Jackson’s death. Other news outlets either said “Jackson taken to hospital” or (perhaps hedging their bets) “US web-site reports Jackson dead”.
By this time the twitterverse was experiencing a cosmic storm of tweets about the “fact” of Jackson’s passing. A comparably large number of comments lamented how slow “old media” was to acknowledge this “fact”. Eventually of course the dinosaurs of traditional news and reporting lumbered to the same conclusion as the more agile mammals of Twitter.
In this case social media was proved to be both quick and accurate, so why am I now going to offer a defence of the world’s news organisations? Well I’ll start with a passage from one of my all-time favourite satires, Yes Minister, together with its sequel Yes Prime Minister.
In the following brief excerpt Sir Geoffrey Hastings (the head of MI5, the British domestic intelligence service) is speaking to The Right Honourable James Hacker (the British Prime Minister). Their topic of conversation is the recently revealed news that a senior British Civil Servant had in fact been a Russian spy:
Hastings:
Things might get out. We don’t want any more irresponsible ill-informed press speculation.
Hacker:
Even if it’s accurate?
Hastings:
Especially if it’s accurate. There is nothing worse than accurate irresponsible ill-informed press speculation.
Was the twitter noise about Jackson’s death simply accurate ill-informed speculation? It is difficult to ask this question as, sadly, the tweets (and TMZ) proved to be correct. However, before we garland new media with too many wreaths, it is perhaps salutary to recall that there was a second rumour of a celebrity death circulating in the febrile atmosphere of Twitter on that day. As far as I am aware, Pittsburgh’s finest – Jeff Goldblum – is alive and well as we speak. Rumours of his death (in an accident on a New Zealand movie set) proved to be greatly exaggerated.
The difference between a reputable news outlet and hordes of twitterers is that the former has a reputation to defend. While the average tweep will simply shrug their shoulders at RTing what they later learn is inaccurate information, misrepresenting the facts is a cardinal sin for the best news organisations. Indeed reputation is the main thing that news outlets have going for them. This inevitably includes annoying and time-consuming things such as checking facts and validating sources before you publish.
With due respect to Mr Jackson, an even more tragic set of events also sparked some similar discussions; the aftermath of the Iranian election. The Economist published an interesting artilce comparing old and new media responses to this entitiled: Twitter 1, CNN 0. Their final comments on this area were:
[…]the much-ballyhooed Twitter swiftly degraded into pointlessness. By deluging threads like Iranelection with cries of support for the protesters, Americans and Britons rendered the site almost useless as a source of information—something that Iran’s government had tried and failed to do. Even at its best the site gave a partial, one-sided view of events. Both Twitter and YouTube are hobbled as sources of news by their clumsy search engines.
Much more impressive were the desk-bound bloggers. Nico Pitney of the Huffington Post, Andrew Sullivan of the Atlantic and Robert Mackey of the New York Times waded into a morass of information and pulled out the most useful bits. Their websites turned into a mish-mash of tweets, psephological studies, videos and links to newspaper and television reports. It was not pretty, and some of it turned out to be inaccurate. But it was by far the most comprehensive coverage available in English. The winner of the Iranian protests was neither old media nor new media, but a hybrid of the two.
Aside from the IT person in me noticing the opportunity to increase the value of Twitter via improved text analytics (see my earlier article, Literary calculus?), these types of issues raise concerns in my mind. To balance this slightly negative perspective it is worth noting that both accurate and informed tweets have preceded several business events, notably the recent closure of BI start-up LucidEra.
Also main stream media seem to have swallowed the line that Google has developed its own operating system in Chrome OS (rather than lashing the pre-existing Linux kernel on to its browser); maybe it just makes a better story. Blogs and Twitter were far more incisive in their commentary about this development.
Considering the pros and cons, on balance the author remains something of a doubting Thomas (by name as well as nature) about placing too much reliance on Twitter for news; at least as yet.
Accuracy an Business Intelligence
Some business thoughts leaked into the final paragraph of the Introduction above, but I am interested more in the concept of accuracy as it pertains to one of my core areas of competence – business intelligence. Here there are different views expressed. Some authorities feel that the most important thing in BI is to be quick with information that is good-enough; the time taken to achieve undue precision being the enemy of crisp decision-making. Others insist that small changes can tip finely-balanced decisions one way or another and so precision is paramount. In a way that is undoubtedly familiar to regular readers, I straddle these two opinions. With my dislike for hard-and-fast recipes for success, I feel that circumstances should generally dictate the approach.
There are of course different types of accuracy. There is that which insists that business information reflects actual business events (often more a case for work in front-end business systems rather than BI). There is also that which dictates that BI systems reconcile to the penny to perhaps less functional, but pre-existing scorecards (e.g. the financial results of an organisation).
A number of things can impact accuracy, including, but not limited to: how data has been entered into systems; how that data is transformed by interfaces; differences between terminology and calculation methods in different data sources; misunderstandings by IT people about the meaning of business data; errors in the extract transform and load logic that builds BI solutions; and sometimes even the decisions about how information is portrayed in BI tools themselves. I cover some of these in my previous piece Using BI to drive improvements in data quality.
However, one thing that I think differentiates enterprise BI from departmental BI (or indeed predictive models or other types of analytics), is a greater emphasis on accuracy. If enterprise BI is to aspire to becoming the single version of the truth for an organisation, then much more emphasis needs to be placed on accuracy. For information that is intended to be the yardstick by which a business is measured, good enough may fall short of the mark. This is particularly the case where a series of good enough solutions are merged together; the whole may be even less than the sum of its parts.
A focus on accuracy in BI also achieves something else. It stresses an aspiration to excellence in the BI team. Such aspirations tend to be positive for groups of people in business, just as they are for sporting teams. Not everyone who dreams of winning an Olympic gold medal will do so, but trying to make such dreams a reality generally leads to improved performance. If the central goal of BI is to improve corporate performance, then raising the bar for the BI team’s own performance is a great place to start and aiming for accuracy is a great way to move forward.
A final thought: England went on to beat Australia by precisely 115 runs in the second Test at Lord’s; the final result coming today at precisely 12:42 pm British Summer Time. The accuracy of England’s bowling was a major factor. Maybe there is something to learn here.
The last few weeks have been a pretty quiet for me on-line with little activity in either the twitterverse or blogosphere. The flip-side of this is that I have been very busy in a number of other areas, both professional and personal (rumours that my low-profile coincides precisely with the current Ashes series are of course greatly exaggerated).
While my “free” time seems unlikely to increase dramatically in the near future, it is my hope that I will be able to return to penning a few blog articles. Speaking of which, that is precisely what I am about to turn my attention to now.
Peter
I am not 100% sure how this photo managed to creep into the post.
I have been following DecisionStat’s excellent series of interviews with leading figures in the IT industry who have a focus on Business Intelligence, Analytics and Data Management. So I was delighted when I received the invitation to be interviewed by Ajay myself.
This turned into a wide-ranging discussion on a number of areas including the perception of science in society, but most of the content refers to Business Intelligence, analytics, cloud computing, data quality and related areas. You can read the interview in full by clicking on the image or text below.
Ajay Ohri established DecisionStats in 2007 to focus on a number of areas pertinent to business an technology. These include: India, The Internet, Analytics, Company Analysis and Interviews. Ajay is also principal of SwanPLC, who are in the business of helping customers with advanced analytical solutions including recommendations of products and services.
Back in April, I posted My “all-time” most-read 5 articles and mentioned that I would update the list from time to time. At the half-year point of 2009, it seemed appropriate to revisit this area.
I have done two things with the new statistics. First, given the number of articles that I have published, I have expanded the list to 20 articles. Second, to give a different perspective, I have added a run-down of those articles that have received the most views per day.
Of course the first list is more likely to contain older articles (which have had more time to accumulate views), whereas the second list is more likely to include new articles (given that most articles have peak viewing figures soon after they are posted). Despite the vagaries of both approaches, it is probably safe to say that if an article appears on both lists it has been pretty popular. I have highlighted 12 such posts in yellow below.
In future, I may consider calculating how well an article has performed against the typical ageing profile. This would address the shortcomings of both of the current tables and therby offer a more definitive benchmark. However this will be dependent on WordPress.com making it a bit easier to download information about page views.
As before I have focussed just on articles, so views of pages about my career or other background information have been omitted from the following.
I have been a featured blogger on SmartDataCollective.com almost as long as I have been a blogger. SDC.com is Social Media Today’s community site, focussed on all aspects of Business Intelligence, Data Warehousing and Analytics, with a pinch of social media thrown in to the mix.
Brian Roger, the SDC.com editor, was recently kind enough to interview me about my career in BI, the challenges I have faced and what has helped to overcome these. This interview is now available to listen to as part of their Podcast series – click on the image below to visit their site.
I would be interested in feedback about any aspect of this piece, which I am grateful to Brian for arranging.
Social Media Today LLC helps global organizations create purpose-built B2B social communities designed to achieve specific, measurable corporate goals by engaging exactly the customers and prospects they most want to reach. Social Media Today helps large companies leverage the enormous power of social media to build deeper relationships with potential customers and other constituencies that influence the development of new business. They have found that their primary metrics of success are levels of engagement and business leads. One thousand people who come regularly and might buy an SAP, Oracle or Teradata system some day is better than a million people who definitely won’t.
Social Media Today LLC, is a battle-tested, nimble team of former journalists, online managers, and advertising professionals who have come together to make a new kind of media company. With their backgrounds, and passions for, business-to-business and public policy conversations, they have decided to focus their efforts in this area. To facilitate the types of convresations that they would like to see Social Media Today is assembling the world’s best bloggers and providing them with an independent “playground” to include their posts, to comment and rate posts, and to connect with each other. On their flagship site, SocialMediaToday.com, they have brought together many of the most intriguing and original bloggers on media and marketing, covering all aspects of what makes up the connective tissue of social media from a global perspective.
I have tried a number of approaches to people RTing my tweets on twitter. None of these seems wholly satisfactory and so I thought that I would ask people’s opinions:
If you think that there is an option missing from this list, please add it as a comment below.
I recently came across a record of my first ever foray into the world of Business Intelligence, which dates back to 1997. This was when I was still predominantly an ERP person and was considering options for getting information out of such systems. The piece in question is a brief memo to my boss (the CFO of the organisation I was working at then) about what I described as the OLAP market.
This was a time of innocence, when Google had not even been incorporated, when no one yet owned an iPod and when, if you tried to talk to someone about social media, they would have assumed that you meant friendly journalists. All this is attested to by the fact that this was a paper memo that was printed and circulated in the internal mail – remember that sort of thing?
Given that the document has just had its twelfth birthday, I don’t think I am breeching any confidences in publishing it, though I have removed the names of the recipients for obvious reasons.
INTERNAL MEMORANDUM
To:
European CFO
cc:
Various interested parties
From:
Peter Thomas
Date:
16th June 1997
Subject:
What is OLAP?
On-Line Analytical Processing (OLAP) is a category of software technology that enables analysts, managers and executives to gain insight into data. This is achieved by providing fast, consistent, interactive access to a wide variety of possible views of information. This has generally been transformed from raw data to reflect the real dimensionality of the enterprise.
There are around 30 vendors claiming to offer OLAP products. A helpful report by Business Intelligence[1] (an independent research company) estimates the market share of these . As many of these companies sell other products, the following cannot be viewed as 100% accurate. However the figures do provide some interesting reading.
Vendor
1996
1995
Market Position
Share (%)
Market Position
Share (%)
Oracle
1
19.0%
1
20.0%
Hyperion Software
2
18.0%
2
19.0%
Comshare
3
12.0%
3
16.0%
Cognos
4
9.0%
4
5.0%
Arbor Software
5
4.8%
7
2.9%
Holistic Systems
6
4.3%
6
4.7%
Pilot Software
7
4.0%
5
4.8%
MircoStrategy
8
3.5%
9
2.1%
Planning Sciences
9
2.6%
8
2.3%
Information Advantage
10
1.8%
10
1.4%
In this group, some companies (Hyperion, Comshare, Holistic, Pilot Software and Planning Services) provide either complete products or extensive toolkits. In contrast some vendors (such as Arbor and – outside the top ten – Applix) only sell specialist multi-dimensional databases. Others (e.g. Cognos and – outside the top ten – BusinessObjects and Brio Technology), offer client based OLAP tools which are basically sophisticated report writers. The final group (including MicroStrategy and Information Advantage) offer a mixed relational / dimensional approach called Relational OLAP or ROLAP.
If we restrict ourselves to the “one-stop solution” vendors in the above list, it is helpful to consider the relative financial position of the top three.
Standard note: You need to be a member of both LinkedIn.com and the group mentioned to view the discussions.
Introduction
Here are a couple of sections from the original poster’s starting comments:
I’ve been thinking: is one version of the truth attainable or is it a bit of snake oil? Is it a helpful concept that powerfully communicates a way out of spreadmart purgatory? Or does the idea of one version of the truth gloss over the fact that context or point of view are an inherent part of any statement about data, which effectively makes truth relative? I’m leaning toward the latter position.
[…]
There can only be one version of the truth if everyone speaks the same language and has a common point of view. I’m not sure this is attainable. To the extent that it is, it’s definitely not a technology exercise. It’s organizational change management. It’s about changing the culture of an organization and potentially breaking down longstanding barriers.
Please join the group if you would like to read the whole post and the subsequent discussions, which were very lively. Here I am only going to refer to these tangentially and instead focus on the concept of a single version of the truth itself.
Readers who are not interested in the ellipitcal section of this article and who would instead like to cut to the chase are invited to click here (warning there are still some ellipses in the latter sections).
A [very] brief and occasionally accurate history of truth
I have discovered a truly marvellous proof of the nature of truth, which this column is too narrow to contain.
— Pierre de Tomas (1637)
Instead of trying to rediscover M. Tomas’ proof, I’ll simply catalogue some of the disciplines that have been associated (rightly or wrongly) with trying to grapple with the area:
Various branches of Philosophy, including:
Metaphysics
Epistemology
Ethics
Logic
History
Religion (or more perhaps more generally spirituality)
Natural Science
Mathematics
and of course Polygraphism
Given my background in Pure Mathematics the reader might expect me to trumpet the claims of this discipline to be the sole arbiter of truth; I would reply yes and no. Mathematics does indeed deal in absolute truth, but only of the type: if we assume A and B, it then follows that C is true. This is known as the axiomatic approach. Mathematics makes no claim for the veracity of axioms themselves (though clearly many axioms would be regarded as self-evidently true to the non-professional). I will also manfully resist the temptation to refer to the wrecking ball that Kurt Gödel’s took to axiomatic systems in 1931.
I have also made reference (admittedly often rather obliquely) to various branches of science on this blog, so perhaps this is another place to search for truth. However the Physical sciences do not really deal in anything as absolute as truth. Instead they develop models that approximate observations, these are called scientific theories. A good theory will both explain aspects of currently observed phenomena and offer predictions for yet-to-be-observed behaviour (what use is a model if it doesn’t tell us things that we don’t already know?). In this way scientific theories are rather like Business Analytics.
Unlike mathematical theories, the scientific versions are rather resistant to proof. Somewhat unfairly, while a mountain of experiments that are consistent with a scientific theory do not prove it, it takes only one incompatible data point to disprove it. When such an inconvenient fact rears its head, the theory will need to be revised to accommodate the new data, or entirely discarded and replaced by a new theory. This is of course an iterative process and precisely how our scientific learning increases. Warning bells generally start to ring when a scientist starts to talk about their theory being true, as opposed to a useful tool. The same observation could be made of those who begin to view their Business Analytics models as being true, but that is perhaps a story for another time.
I am going to come back to Physical science (or more specifically Physics) a little later, but for now let’s agree that this area is not going to result in defining truth either. Some people would argue that truth is the preserve of one of the other subjects listed above, either Philosophy or Religion. I’m not going to get into a debate on the merits of either of these views, but I will state that perhaps the latter is more concerned with personal truth than supra-individual truth (otherwise why do so many religious people disagree with each other?).
Discussing religion on a blog is also a certain way to start a fire, so I’ll move quickly on. I’m a little more relaxed about criticising some aspects of Philosophy; to me this can all too easily descend into solipism (sometimes even quicker than artificial intelligence and cognitive science do). Although Philosophy could be described as the search for truth, I’m not convinced that this is the same as finding it. Maybe truth itself doesn’t really exist, so attempting to create a single version of it is doomed to failure. However, perhaps there is hope.
After the preceding divertimento, it is time to return to the more prosaic world of Business Intelligence. However there is first room for the promised reference to Physics. For me, the phrase “a single version of the truth” always has echoes of the search for a Grand Unified Theory (GUT). Analogous to our discussions about truth, there are some (minor) definitional issues with GUT as well.
Some hold that GUT applies to a unification of the electromagnetic, weak nuclear and strong nuclear forces at very high energy levels (the first two having already been paired in the electroweak force). Others that GUT refers to a merging of the particles and forces covered by the Standard Model of Quantum Mechanics (which works well for the very small) with General Relativity (which works well for the very big). People in the first camp might refer to this second unification as a ToE (Theory of Everything), but there is sometimes a limit to how much Douglas Adams’ esteemed work applies to reality.
For the purposes of this article, I’ll perform the standard scientific trick of a simplifying assumption and use GUT in the grander sense of the term.
Scientists have striven to find a GUT for decades, if not centuries, and several candidates have been proposed. GUT has proved to be something of a Holy Grail for Physicists. Work in this area, while not as yet having been successful (at least at the time of writing), has undeniably helped to shed a light on many other areas where our understanding was previously rather dim.
This is where the connection with a single version of the truth comes in. Not so much that either concept is guaranteed to be achievable, but that a lot of good and useful things can be accomplished on a journey towards both of them. If, in a given organisation, the journey to a single version of the truth reaches its ultimate destination, then great. However if, in an another company, a single version of the truth remains eternally just over the next hill, or round the next corner, then this is hardly disastrous and maybe it is the journey itself (and the aspirations with which it is commenced on) that matters more than the destination.
Before I begin to sound too philosophical (cf. above) let me try to make this more concrete by going back to our starting point with some Mathematics and considering some Venn diagrams.
Ordo ab chao
In my experience the following is the type of situation that a good Business Intelligence programme should address:
The problems here are manifold:
Although the various report systems are shown as separate, the real situation is probably much worse. Each of the reporting and analysis systems will overlap, perhaps substantially, with one or more or the other ones. Indeed the overlapping may be so convoluted that it would be difficult to represent this in two dimensions and I am not going to try. This means that you can invariably ask the same question (how much have we sold this month) of different systems and get different answers. It may be difficult to tell which of these is correct, indeed none of them may be a true reflection of business reality.
There are a whole set of things that may be treated differently in the different ellipses. I’ll mention just two for now: date and currency. In one system a transaction may be recorded in a month when it is entered into the system. In another it may be allocated to the month when the event actually occurred (sometimes quite a while before it is entered). In a third perhaps the transaction is only dated once it has been authorised by a supervisor.
In a multi-currency environment reports may be in the transactional currency, rolled-up to the currency of the country in which they occurred, or perhaps aggregated across many countries in a number of “corporate” currencies. Which rate to use (rate on the day, average for the month, rolling average for the last year, a rate tied to some earlier business transaction etc.) may be different in different systems, equally the rate may well vary according to the date of the transaction (making the last set of comments about which date is used even more pertinent).
A whole set of other issues arise when you begin to consider things such as taxation (are figures nett or gross), discounts, commissions to other parties, phased transactions and financial estimates. Some reports may totally ignore these, others my take account of some but not others. A mist of misunderstanding is likely to arise.
Something that is not drawn on the above diagram is the flow of data between systems. Typically there will be a spaghetti-like flow of bits and bytes between the different areas. What is also not that uncommon is that there is both bifurcation and merging in these flows. For example, some sorts of transactions from Business Unit A may end up in the Marketing database, whereas others do not. Perhaps transactions carried out on behalf of another company in the group appear in Business Unit B’s reports, but must be excluded from the local P&L. The combinations are almost limitless.
Interfaces can also do interesting things to data, re-labelling it, correcting (or so their authors hope) errors in source data and generally twisting the input to form output that may be radically different. Also, when interfaces are anything other than real-time, they introduce a whole new arena in which dates can get muddled. For instance, what if a business transaction occurred in a front-end system on the last day of a year, but was not interfaced to a corporate database until the first day of the next one – which year does it get allocated to in the two places?
Finally, the above says nothing about the costs (staff and software) of maintaining a heterogeneous reporting landscape; or indeed the costs of wasted time arguing about which numbers are right, or attempting to perform tortuous (and ultimately fruitless) reconciliations.
Now the ideal situation is that we move to the following diagram:
This looks all very nice and tidy, but there are still two major problems.
A full realisation of this transformation may be prohibitively expensive, or time-consuming.
Having brought everything together into one place offers an opportunity to standardise terminology and to eliminate the confusion caused by redundancy. However, it doesn’t per se address the other points made from 2. onwards above.
The need to focus on what is possible in a reasonable time-frame and at a reasonable cost may lead to a more pragmatic approach where the number of reporting and analysis systems is reduced, but to a number greater than one. Good project management may indeed dictate a rolling programme of consolidation, with opportunities to review what has worked and what has not and to ascertain whether business value is indeed being generated by the programme.
Nevertheless, I would argue that it is beneficial to envisage a final state for the information architecture, even if there is a tacit acceptance that this may not be realised for years, if at all. Such a framework helps to guide work in a way that making it up as we go along does not. I cover this area in more detail in both Holistic vs Incremental approaches to BI and Tactical Meandering for those who are interested.
It is also inevitable that even in a single BI system data will need to be presented in different ways for different purposes. To take just one example, if you goal is to see how the make up of a book of business has varied over time, then it is eminently sensible to use a current exchange rate for all transactions; thereby removing any skewing of the figures caused by forex fluctuations. This is particularly the case when trying to assess the profitability of business where revenue occurs at a discrete point in the past, but costs may be spread out over time.
However, if it is necessary to look at how the organisation’s cash-flow is changing over time, then the impact of fluctuations in foreign exchange rates must be taken into account. Sadly if an American company wants to report how much revenue it has from its French subsidiary then the figures must reflect real-life euro / dollar rates (unrealised and realised foreign currency gains and losses notwithstanding).
What is important here is labelling. Ideally each report should show the assumptions under which it has been compiled at the top. This would include the exchange rate strategy used, the method by which transactions are allocated to dates, whether figures are nett or gross and which transactions (if any) have been excluded. Under this approach, while it is inevitable that the totals on some reports will not agree, at least the reports themselves will explain why this is the case.
So this is my take on a single version of the truth. It is both a) an aspirational description of the ideal situation and something that is worth striving for and b) a convenient marketing term – a sound-bite if you will – that presents a palatable way of describing a complex set of concepts. I tried to capture this essence in my reply to the LinkedIn.com thread, which was as follows:
To me, the (extremely hackneyed) phrase “a single version of the truth” means a few things:
One place to go to run reports and perform analysis (as opposed to several different, unreconciled, overlapping systems and local spreadsheets / Access DBs)
When something, say “growth” appears on a report, cube, or dashboard, it is always calculated the same way and means the same thing (e.g. if you have growth in dollar terms and growth excluding the impact of currency fluctuations, then these are two measures and should be clearly tagged as such).
More importantly, that the organisation buys into there being just one set of figures that will be used and self-polices attempts to subvert this with roll-your-own data.
Of course none of this equates to anything to do with truth in the normal sense of the word. However life is full of imprecise terminology, which nevertheless manages to convey meaning better than overly precise alternatives.
More’s Utopia was never intended to depict a realistic place or system of government. These facts have not stopped generations of thinkers and doers from aspiring to make the world a better place, while realising that the ultimate goal may remain out of reach. In my opinion neither should the unlikelihood of achieving a perfect single version of the truth deter Business Intelligence professionals from aspiring to this Utopian vision.
I have come pretty close to achieving a single version of the truth in a large, complex organisation. Pretty close is not 100%, but in Business Intelligence anything above 80% is certainly more than worth the effort.
Peter James Thomas 