Introduction
The phrase IT outsourcing tends to provoke strong reactions. People either embrace it as a universal panacea capable of addressing any business problem, or recoil in horror at the very sound of it. Just for a change, I am somewhere in the middle; to me it is another tool at the disposal of businesses which can either be used wisely or poorly (much like IT itself you might say). As always the difference between the two extremes comes down to how well the project is led. Regardless of this, there are some benefits and some disbenefits associated with IT outsourcing and this article will explore the case for applying outsourcing to business intelligence.
Benefits of general IT outsourcing
Before I plunge into the world of BI, it is perhaps worth revisiting the general reasons for IT outsourcing, some of the most regularly quoted are as follows:
1. Reduction in costs
The provider of outsourcing (I’m just going to say “the provider” from now on to save typing) can carry out the same tasks at a cheaper cost to the client organisation (while still presumably turning a profit). There can be a number of bases for this; the one that generally comes to mind is wage arbitrage between different economies. However, it could also be that the provider has economies of scale; for instance, less people being required to run the consolidated data centres of several companies, than is required to run each separately. Also the provider may have staff who are more productive than at the client.
2. Ability to scale-up and scale down resource
The nature of business is such that sometimes all hands are required on the IT deck and at others there is spare capacity (this is something I address in my two articles on Problems associated with the IT cycle and Mitigating problems with the IT cycle). Now IT departments are normally quite good at finding (hopefully) useful things for people to do, but the issue remains. The promise of an outsourcing arrangement is that the tap of resource can be adjusted to meet demand without having to either fire and rehire staff, or rely on bringing in expensive contract resource. It is often hoped that this feature of outsourcing will also help to speed IT products to market.
3. Making IT provision a contractual relationship
An arrangement with a provider, depending on how the contract is drafted, can make the provision of IT services subject to penalties and claw-backs when service levels drop below those that have been agreed. While there are clearly some sanctions that can be applied to underperformance by internal IT departments, the financial benefit to the organisation is likely to be less (unless your CIO is a multi-billionaire of course). Companies are used to these contractual relationships, they are often the lifeblood of business, and it is a more familiar way of dealing with issues for them.
4. Access to skills
The nature of IT is that it does tend to evolve, sometimes quickly, sometimes slowly. For organisations this means keeping their IT people’s skills up to date though courses, or continually looking to bring people with new skills into an organisation (such people generally not being the cheapest). The idea with an outsourcing arrangement is that these issues become the headache of the provider, not the client. This area can be particularly pertinent when there is a technology change or a significant upgrade; these are times at which the prospect of being shot of IT worries may seem very attractive. The effort and cost of, as it were, upgrading your in-house IT staff may seem prohibitive in these circumstances.
5. Focus on core competencies
This has been a business mantra for many years, why should a company engaged in a wholly separate area of human endeavour want to become experts in building and supporting complex IT systems, when they can get a specialist organisation to do this for them? This moves towards the idea of a lean, or even virtual, organisation.
6. Failure of in-house IT
It is sad to have to add this item, but it is often the implicit (and sometimes even the explicit) driver of a desire to outsource. CEOs, COOs or CFOs may be so fed up with the performance of their IT people that they feel that surely someone else could not be worse. There is an adage that you don’t outsource a problem, but this is often honoured more in breech than observance.
I am sure that there are other advantages, claimed or real, for IT outsourcing, but the above list at least covers many of the normal arguments. At this stage a fully-balanced article would probably present arguments against IT outsourcing. However, my objective here is not to provide a critique of IT outsourcing in general, but to see whether the above benefits apply to business intelligence. Because of this, and I should stress purely for the purposes of this article, I am going to accept that all of the above gains are both realisable and desirable for general IT. There will therefore you will find no comments here about arbitrage (of its very nature) resulting in differentials of pricing closing over time.
The only benefit that I am going to rule out is the final one; addressing failed IT departments. Applying outsourcing in these cases is only likely to make things worse, and probably more expensive. Far better in my opinion to work out why IT is failing (most typically due to poor leadership it has to be said, see also my article: Some reasons why IT projects fail) and draw up plans for addressing this. If outsourcing is a strong element of this, then so be it, but thinking that it will resolve this type of issue is probably naive in most circumstances.
So, as always seems to be the case in these types of articles, we have five potential benefits against which to assess outsourcing BI. Before I look at each in turn, I wanted to make some general observations.
Things that are different about BI
The main fly in the ointment with respect to outsourcing business intelligence is the fact that good BI is reliant upon four things (see also BI implementations are like icebergs):
| A. | An in-depth understanding of business requirements, developed by close collaboration with a wide range of business managers. In particular, what is necessary is understanding what questions the business wants to ask and why (see Scaling-up Performance Management and Developing an international BI strategy) |
| B. | An extensive appreciation of the data available in different business systems, its accuracy and how data in different places is related to each other. |
| C. | Developing creative ways of transforming the available data into the required information and presenting this in an easy-to-understand and use manner. |
| D. | A focus on change management that includes business-focussed marketing, training and follow-up to ensure that the work carried out in the first three areas results in actual business adoption and thereby the creation of value (see my collection of articles focussed on cultural transformation). |
With the possible exception of item C., which is more technical, the above are best carried out in a symbiotic relationship with the business. Ideally what develops is a true IT / business hybrid team, where, though people have clear roles, the differences between these blur into each other. In turn, building thus type of team is predicated on developing strong relationships between the IT and business members and establishing high levels of trust and respect.
Also with item C., this is not precisely a stand-alone activity. It is one best carried out collaboratively by technically-aware business analysts and business-aware data analysts, ETL programmers and OLAP designers. Once again, distinctions blur somewhat during this work and a different type of hybrid team appears.
I have tried to illustrate the way that these tasks and teams should overlap in the following diagram.
Clearly it is not impossible to achieve what I have described above in an outsourced environment, but it seems that it might be rather tougher to do this. One key point is that the type of skills that are necessary for success in BI are cross-over business / IT skills and these are generally less easy to buy off the shelf. Another is that the type of intellectual property that a BI team will build up (basically extensive knowledge of what makes the organisation tick) is precisely the sort that you would want to retain within an organisation.
I would suggest that if an organisation wants to outsource BI, then they should start that way. Once a BI team has gone through tasks A. to D. above then I can’t see how it would be cost-effective to subsequently outsource. The transfer of knowledge would take too long and be too costly.
To provide some context to this let me share some non-confidential details of a study I performed recently comparing the efficiency of a well-established BI team in a developed country with a less mature BI team in a lower-cost location. Rather than considering relative costs, I looked at relative productivity. A simple way to do this is to get quotes for carrying out a certain type of work from both teams (though I also applied some other techniques, which I won’t go into here). My main finding was that the ostensibly high cost team was more than twice as productive as the allegedly low-cost team. Just to be clear, if the “high-cost” team quoted $X for a piece of work, the “low-cost” team quoted over $2X,because they required much more resource and/or time to carry out the same work.
So, in what follows, I will assume that a decision is taken to outsource at the inception of a project. With this assumption and the previous background, let’s go back and look at the five benefits of outsourcing from the beginning.
Matching the benefits to BI
1. Reduction in costs
It will take external BI resource at least as long as internal BI resource to understand business requirements and available data. In fact internal staff probably have something of an advantage as they should already have an appreciation of what the organisation does and how IT systems support this. The external resource also has the disadvantage of it probably being more difficult for them to build business relationships, this can be exacerbated if there are personnel changes during the project; something that is perhaps more likely to happen with an external provider. If the provider is located in another country, then this raises even more challenges and inefficiencies (and leads to travel expense).
It will take an external BI team at least as long as an internal one to dig into the available data and how the various systems inter-relate. Again, having some familiarity with the existing systems’ landscape would be an advantage for an in-house team.
If an external team can get to the position where they understand the business needs and the available data really well in a reasonable period of time, then they could possibly have an advantage in the arena of transforming data into information. Something that may mitigate this however is that fact that most BI development is iterative and that a rolling set of prototypes needs to be reviewed closely with the business. This element introduces the same challenges as were apparent with defining business requirements above.
Similar arguments as were made about the business requirements phase apply to deployment and follow-up.
2. Ability to scale-up and scale down resource
While it may be possible (subject to contract) to scale-down resource with a provider (though perhaps tougher to get them back when you need them), scaling-up is just as hard as it is in-house at it means more staff at the provider going through the learning curve about the organisations business needs and data.
4. Access to skills
This is the crux of the matter. The skills in question are not Java programming (or even Cobol), they are business knowledge. ETL and OLAP skills are important, but only if they are applied by people who understand what they are doing and to what purpose. These skills are not just lying around in the market place; they are acquired through hard work and dedication.
3. Making IT provision a contractual relationship
Clearly this is a benefit of outsourcing. However, given that the contract is there for when things go awry, it is worth asking the question “are things more or less likely to go wrong with a provider?”
5. Focus on core competencies
While it is quite easy to argue that building e-commerce systems is not necessarily a core competency, good BI is about understanding what is necessary to best run the business. If that is not a core competency of any organisation, then I struggle to think of what would be.
Summary
My main argument is that BI is different to general IT projects (an assertion to which I will return in a forthcoming article). Having successfully run both, I am confident in this statement. I also think that you need different types of people with different skills in BI projects. These facts, plus the closeness of business / IT relationships which are necessary in the area mean that outsourcing is less likely to be effective. I am sure that an outsourcing arrangement can work well for some organisations in some circumstances, but I would argue strongly against it being best practise for most organisations most of the time.
After penning this article, a further problem with outsourcing business intelligence came to my mind; security. On part of most BI systems is a facility to analyse the organisation’s results. Ideally the BI system will have these figures in place very soon after the end of a financial closing. Such data is market sensitive and there may be concerns with trusting an external provider with both producing this and ensuring that it remains confidential until market announcements are made. I am not suggesting that providers are unethical, just that companies may not wish to take a chance in this area.
I should also credit a thread on the LinkedIn.com EPM – Business Intelligence group, which got me thinking about this area (as ever, you need to be a member of LinkedIn.com and the group to view this)
[…] and cons of outsourcing BI which will appear here at some future point; UPDATE, now published as Is outsourcing business intelligence a good idea?). Instead I want to talk about another type of outsourcing, one that ended up being a major success […]
Very good, detailed coverage of this most controversial subject. Thanks Peter.
Thank you Ferenc
Peter,
Well thought out piece with very good points. I have always felt BI does belong in-house so that “tribal” knowledge can develop and benefit the company.
David
Thank you for your kind comments David.
Hi Peter — As a BI consultant, aren’t you being a little hypocritical? On one hand, you say that for BI to be successful you need an in-depth understanding of business requirements and an extensive appreciation of the data in the business systems, and that the type of intellectual property that a BI team will build up is precisely the type of knowledge you want to retain in house.
On the other hand, being a BI consultant, aren’t you telling customers that they should be hiring you to help them build their solutions? Isn’t that “outsourcing” the expertise that you recommend companies build and keep in house? More broadly, many/most large scale BI implementations bring in a consulting team to design and build the solution for the customer. And, most BI experts recommend bringing in expert consulting. But, these consulting teams don’t have the in-depth knowledge about the customer’s business or their systems, and they leave the customer when the project is done. Doesn’t that violate several of the items that you say BI is reliant on?
So, if customers already are relying on outside expertise to have successful BI projects, then the main difference between what is already being done and what you’re referring to as “outsourcing” is simply where the servers reside.
Hi Ken,
I think maybe you have made a wrong assumption at the beginning of your comment. I may describe myself as knowing something about BI (especially the management of BI projects and the change they involve), but I’m not currently and have not previously been (at least at the time of posting) a BI consultant and am not touting for consulting work on this site. Instead I’m just offering my perspective based on nine years working in the field.
At the beginning of my first BI project, I did speak extensively to one of the big consultancies, but all they wanted to do was to go on about their insurance industry template and how great it was. They seemed to have a plan for cutting code before they had asked what we wanted so I junked them.
I did bring in an independent consultant who has worked extensively in BI. He was with us for the first six months of the project. His role was to help mentor the team through the early stages and to assist with vendor selection. He was very helpful, but we outgrew his services after this period. He was not involved in business analysis (except from the point of view of a few tips here and there) during his stay. Nevertheless, I would recommend new teams having such a mentor, and have referred to this publicly elsewhere (on this LinkedIn.com thread for example).
I also brought in an external consultant to help put together a training programme (based on input from the BI team and the business), though he did end up joining the company subsequently.
I also used some contract ETL programmers from time to time when things got busy. I think that completes my full disclosure.
So what you describe as the approach used for “most/many large scale BI implementations” does indeed violate what I say in the article. I would never recommend an organisation taking this approach. I think external help can be useful in helping to define a strategy, avoiding common pitfalls and building training programmes. I think you could also make a case for bringing in some one external to manage a BI project, so long as part of their role was to groom an internal successor.
Given this, I’m not sure where my alleged hypocrisy comes in – are you?
Peter
Hi,
i found the article of Sid Adelman on this subject very interesting. It gives a guidlines on what should be outsourced in BI/DW projects.
Please find the article at :
http://www.information-management.com/issues/20060501/1053405-1.html
Voila !
Abdel,
Thanks for the comment (which WordPress initially tagged as spam for some reason!). I read the article and it chimes very closely with my thoughts – as expressed above.
I’m glad that I am not alone in holding these opinions. Thanks for the link.
Peter
Hi Peter — Thanks for your clarification. To clarify on my end, I do think outsourcing BI is a valid option (and often the only real option) for a large portion of the market. Most companies don’t have the resources or the skills in house to build and manage a BI solution, so they end up with manually created Excel spreadsheets (“Excel Hell”) and a lot of guesswork. This is why I believe that Software-as-a-Service BI is a very positive trend for the market.
I find it hypocritical when consultants say “don’t use SaaS BI. You don’t want to outsource BI — you should hire us instead”, because as I mention in my original comment, that’s outsourcing too.
If a company doesn’t have the resources to do the things that are most important for BI’s success internally (understanding the business requirements, understanding the data and how this relates to requirements, designing and executing change programmes to make sure that the best use is made of BI) then I’m not sure what a bunch of consultants or an outsourcing provider is going to do better for them. If there is no one in an organisation who either has or is able to acquire the necessary knowledge, then the company has bigger problems than not having a good BI system.
SaaS BI can work independently of the above (depending on how it is set up) and complement in-house work, but I can’t see how it would supplant it. Do you know more about your clients’ business needs than they do?
Peter
According to my experience most of the companies already has some departamental small BI systems in their companies which solve particular tasks of particular departments and this is a big issues for top management to find someone within the organization who can and have knowledge to start building the corparate BI system. In this situation top management in any case has to hire new people or outsource this services.
To hire good experts in BI most of the time is more difficult rather than to outsouces BI team. That’s why big companies usually outsources BI.
In my opinion, if the company’s business users has stong unterstanding what they want – outsourcing BI is the fastest way to make the implementation.
If no, it will be too expensive for the company to outsource BI.
Vlefanov,
Thanks for the comment. As someone who has successfully built a BI system covering nearly 20 countries, I’ll stick with what I know has worked for me as being cheaper and more effective.
Peter
[…] and although the article is obviously focussed on a specific issue, the first few sections of Is outsourcing business intelligence a good idea? pulls together many of these themes and may be a useful place to […]
[…] There are many skills that will be necessary in such a group. These match the four main pillars of a BI implementation (I cover these in more depth in several places on the blog, including BI implementations are like icebergs and the middle section of Is outsourcing business intelligence a good idea?): […]
[…] If the BI system delivered does not live up to expectations then there are two questions to be answered. In what way does it not meet expectations? and Why did it take until implementation to determine this? It could be that the functionality of the BI tool does not meet what is necessary, but most of these have a wide range of functionality and are at least reasonably intuitive to use. More likely the issue is in the information presented in the tool (which is not judged to be useful) or in an inadequate approach to implementation. The way to address both of these potential problems from the very start of the project is to follow the four-pillared approach that I recommend in many places on this blog; notably in one of the middle sections of Is outsourcing business intelligence a good idea?. […]
[…] Is outsourcing business intelligence a good idea? […]
Hi Peter,
I am BI professional and working in the field of MSBI specifically SSAS. I am based in India and looking for some companies for business partnership. I am looking for some companies those can outsource their BI projects.
Hi Aniruddha,
Thank you for taking the time to comment. Did you read the article by any chance?
Peter
Hi Peter, Thanks for the article. A lot of good arguments as to why BI is not really suited for outsourcing.
Any thoughts on how to reverse an outsourced BI department when one realises the real value? Are new projects the best way to start rebuilding or operations of the existing BI solutions?
Hi,
Thanks for taking the time to comment. If I have it right you are asking about bringing an outsourced BI function back in house – do I have that right?
I guess new projects vs business as usual might be somewhat related to how the outsourced systems are perceived. I’m guessing that if they were great, then keeping them outsourced would make sense as it had delivered, so perhaps the question would not arise. Am I right in assuming that this path has been less than successful?
Peter
Peter, Thank you for the great article.
My experience is somewhat different.
There is business and technical aspect of BI. Both aspects are important. Depending on what data sources are used, you can have a business /information analyst which delivers specifications which an external consultant can use to design a data warehouse and ETL. Those activities can en should be outsourced, I believe. There is no way an internal developer can match the skill level and experience of a consultant who have seen a lot of data warehouses in different companies. Learning “hard” technical skills cost time and money. This is an argument which managers understand.
Even iterative development which often comes at a later stadium when reports of OLAP-cubes have to be build, can be done by an external experienced BI-consultant. He has done it many times before, after all.
Tigor
Hi Tigor,
Thank you for reading and taking the time to comment.
As I say, what might be termed the programming resource is the only element that I would consider outsourcing. For mid-sized companies, this is likely to be of a size where the economies of scale are outweighed by the risks, lengthened communication lines and lack of flexibility (understandably) inherent in many outsourcing contracts.
Out of interest, what is your own role? It always hels to get disclosure on these matters, lest other readers infer a conflict of interest.
All the best
Peter
Thanks for the information and analysis about the BI and outsourcing
No problem
Peter
Brilliant article Peter. you have mentioned very good points regarding business intelligence. In my view, Business Intelligence is an arrangement of data management and recovery with the end goal of future projection of interest, sale, income, and so on. Better purpose expediting enhanced business performance is the fundamental profit of utilizing it yet it accompanies a conceivable issue of over-dissection or once in a while even wrong projection because of confusing some economic indicators or from a sudden surprising change good to go situation. Over dependence on business insights is not suggested.
[…] I cover this area in greater detail in Is outsourcing business intelligence a good idea? […]