Do any technologies grow up or do they only come of age?

The 2016 Big Data Maturity Survey (by AtScale)

I must of course start by offering my apologies to that doyen of data experts, Stephen King, for mangling his words to suit the purposes of this article [1].

The AtScale Big Data Maturity Survey for 2016 came to my attention through a connection (see Disclosure below). The survey covers “responses from more than 2,550 Big Data professionals, across more than 1,400 companies and 77 countries” and builds on their 2015 survey.

I won’t use the word clickbait [2], but most of the time documents like this lead you straight to a form where you can add your contact details to the organisation’s marketing database. Indeed you, somewhat inevitably, have to pay the piper to read the full survey. However AtScale are to be commended for at least presenting some of the high-level findings before asking you for the full entry price.

These headlines appear in an article on their blog. I won’t cut and paste the entire text, but a few points that stood out for me included:

  1. Close to 70% [of respondents] have been using Big Data for more than a year (vs. 59% last year)
     
  2. More than 53% of respondents are using Cloud for their Big Data deployment today and 14% of respondents have all their Big Data in the Cloud
     
  3. Business Intelligence is [the] #1 workload for Big Data with 75% of respondents planning on using BI on Big Data
     
  4. Accessibility, Security and Governance have become the fastest growing areas of concern year-over-year, with Governance growing most at 21%
     
  5. Organizations who have deployed Spark [3] in production are 85% more likely to achieve value

Bullet 3 is perhaps notable as Big Data is often positioned – perhaps erroneously – as supporting analytics as opposed to “traditional BI” [4]. On the contrary, it appears that a lot of people are employing it in very “traditional” ways. On reflection this is hardly surprising as many organisations have as yet failed to get the best out of the last wave of information-related technology [5], let alone the current one.

However, perhaps the two most significant trends are the shift from on-premises Big Data to Cloud Big Data and the increased importance attached to Data Governance. The latter was perhaps more of a neglected area in the earlier and more free-wheeling era of Big Data. The rise in concerns about Big Data Governance is probably the single greatest pointer towards the increasing maturity of the area.

It will be interesting to see what the AtScale survey of 2017 has to say in 12 months.
 


 
Disclosure:

The contact in question is Bruno Aziza (@brunoaziza), AtScale’s Chief Marketing Officer. While I have no other connections with AtScale, Bruno and I did make the following video back in 2011 when both of us were at other companies.


 
Notes

 
[1]
 
Excerpted from The Gunslinger.
 
[2]
 
Oops!
 
[3]
 
Apache Hadoop – which has become almost synonymous with Big Data – has two elements, the Hadoop Distributed File Store (HDFS, the piece which deals with storage) and MapReduce (which does processing of data). Apache Spark was developed to improve upon the speed of the MapReduce approach where the same data is accessed many times, as can happen in some queries and algorithms. This is achieved in part by holding some or all of the data to be accessed in memory. Spark works with HDFS and also other distributed file systems, such as Apache Cassandra.
 
[4]
 
How phrases from the past come around again!
 
[5]
 
Some elements of the technology have changed, but the vast majority of the issues I covered in “Why Business Intelligence projects fail” hold as true today as they did back in 2009 when I wrote this piece.

 

 

The Cloud Circle Forum – London

The Cloud Circle

Introduction

Earlier this week I attended the inaugural The Cloud Circle Forum in London. The Cloud Circle is the UK’s first independent Business and IT focused Cloud Computing Community. It is also the sister community of the Business Intelligence-focussed Obis Omni, an organisation with whom I have a longstanding (though I hasten to add, non-contractual) relationship (a list of Obis Omni seminars at which I have presented appears here, and you can also find some of my articles syndicated on their site).

There was a full programme with the morning being taken up by plenary presentations from Harrogate’s InTechnology (@InTechnology) and CloudOrigin (aka Cloud Computing evangelist Richard Hall – @CloudOrigin), followed by two Windows Azure case studies; one from EasyJet and one from Active Web Solutions (@AWSIpswich) for the Royal National Lifeboat Association – these were hosted by Microsoft themselves.

The afternoon programme saw delegates split into two work-streams, one focussed on strategy and management, the other on technology. Work-related pressures meant that I was unable to attend this part of the day, which was a shame as several bits of the morning speeches were helpful.

Who you gonna call?

Unfortunately, despite the fact that virtually every organisiation and individual I have mentioned so far has a Twitter account and the additional fact that there were hundreds of delegates at the forum, there was virtually no Twitter coverage. Maybe we can get too carried away with the all pervasiveness of social media sometimes. There are clearly some avenues of professional life where its influence is yet to be fully felt; even IT conferences!

I tweeted some commentary on the InTechnology presentation and the stream may be viewed here while it persists. However by the time that Richard Hall stood up to speak, a combination of a lack of reception (the auditorium was in the basement) and issues with mobile Twitter on my hand-held brought this activity to a halt.
 
 
The morning presentations

Note: I don’t want to steal the thunder from any of the speakers and so this article does not cover the content of their presentations in any detail. Instead my aim is to highlight a few points and provide a flavour of their talks.

 
InTechnology

The InTechnology talk was interesting in parts, in particular their focus on the savings to be achieved in cloud-based telephony alone. One of their speakers also suggested that the benefits of Cloud Computing were potentially reduced if an organisation worked with more than one vendor, which is clearly an aspect to consider.

Their presentations were topped and tailed by two segments of a Cloud Computing-related spoof of The Apprentice. Clearly some money had gone into this and the results were either hilarious or somewhat ill-advised depending on your personal taste. I have to admit to falling closer to the latter camp. While some delegates seemed to enjoy the fun of the fair, I felt the video distracted somewhat from InTechnology’s core message.
 
 
Cloud-origin

I billed Richard Hall as a Cloud Computing evangelist and certainly his tub-thumping upped the tempo. He made some interesting points, which included his assertion that the proprietors of cloud server farms were employing cutting edge technology that was not currently commercially available and might never be. The point here was that cloud providers were becoming true experts in the area with capabilities far beyond normal organisations. This segued with his prediction that there would be only four, or at most five, mega cloud vendors in the future.

Richard did have one slide focusing on the potential drawbacks (or current short-comings) of Cloud Computing, but you could tell his heart wasn’t really in it. One sensed that Richard never met a cloud he didn’t like, even referring to his only personal Road to Damascus during his talk. However one very valid point he made was that the legal agreements and licensing arrangements for Cloud Computing were significantly lagging the flexibility of the technology itself. This chimes with my own experience of the area.
 
 

Microsoft easyJet.com Active Web Solutions

The real-life case studies of cloud-based success were perhaps more telling than the earlier sessions. Bert Craven, Enterprise Architect at EasyJet, spoke about how his company had been moving selected elements of their IT assets to the cloud. Interestingly, while the original plan had been to keep some critical applications (the sort for which 99.9% availability is not good enough) in-house, one of these was now in the process of becoming cloud-based.

Richard Prodger, Technical Director of AWS, spoke about the work that his company had been doing with the RNLI – a charity that runs volunteer lifeboats around the coasts of Britain. The specific project was to provide fishermen with devices that would automatically alert the RNLI control centre if they fell overboard and then provide accurate positioning information enabling a faster rescue and thus one that would be more likely to result in success. Richard shared stories of several fishermen who were alive today thanks to the system. Here the cloud was not the original vehicle, but something that was subsequently employed to scale up the service.

Both case studies used Windows Azure as a component. I have not used this toolset, nor have I been briefed on it and so will refrain from any comment beyond stating that both Bert and Richard seemed happy with its capabilities; particularly in securely exposing internal systems to external web-users.
 
 
Some thoughts on what I heard and saw

When multiple presenters state that there is no agreed definition of the central subject matter of a seminar and then proceed to provide slightly different takes on this, you know that you are dealing with an emerging technology. That is not to deny the obvious potential of the area, but a degree of maturation is still necessary in this part of the industry before – in Richard Hall’s words – Cloud Computing becomes the future of IT.

There was more than one elephant in the room. First of which is bandwidth, which is relatively plentiful and relatively cheap in many parts of the world, but equally has neither trait in many others. This will be of concern to a lot of global organisations. Of course this is a problem that will undoubtedly go away in time, but it may dog true enterprise implementations of misison-critical Cloud Computing for some years yet.

Security remains a concern, it may well be true that the experts in Cloud Computing will be an order of magnitude more careful and competent about handling their customers’ data than many internal IT departments. However the point is that they are already handling the data of many customers and one error, or one act of malfeasance by an employee, could have a major impact. You may well be safer flying than driving your own car, but when a plane crashes, people tend to notice.

Future consolidation in Cloud Computing was mentioned by a number of speakers. Although this issue is not solely the preserve of cloud technology, it does raise some concerns about betting on the right horse. As has been seen in many areas of industry, the titans of today may be the minnows of tomorrow. When you are trusting an external organisation with your transactions, it helps to know for certain (or as close as you can get to it) that they will be around in five years’ time.

One of the central pitches of Cloud Computing is “let us look after the heavy lifting and your people can focus on more value-added activities”. While there are certainly economies to be seized in this area, the Cloud Computing industry may be doing itself a disservice by stating that customers can effectively stop worrying about functions moved to the cloud. In my mind a lot of care and attention will need to be put into managing relationships with cloud vendors and into integrating cloud-based systems with the rest of the internal IT landscape (or with other cloud-based systems). It may be that this type of work costs a lot less than the internal alternative, but it is nevertheless invidious to suggest that no work at all is required. This line of attack is reminiscent of some of the turn of the millennium sales pitches of ERP vendors, not all of which turned out to be well-founded.

In finishing this slightly downbeat section (and before a more optimistic coda), I’ll return to the commercial issues that Richard Hall referenced. He claimed – correctly in my opinion – that a major benefit of cloud-based solutions was not only that they scaled-up, but that they scaled-down. The “knob” could be adjusted in either direction according to an organisation’s needs. The problem here is that many parts of the Cloud Computing industry still seem wedded to multi-year fixed licensing deals with little commercial scope to scale either up or down without renegotiating the contract. What is technologically feasible may not be contractually pain-free. In the same vein more flexible termination clauses and guaranteed portability of data from one vendor to another need to be sorted out before Cloud Computing is fully embraced by many organisations.

On a more positive note, the above issues are maybe the typical growth pains of a nascent industry. No doubt solutions to them will be knocked into shape in the coming years. It is always tempting fate to predict the future with too much accuracy, but at this point it seems certain that Cloud Computing will play an increasingly important role in the IT landscapes of tomorrow. If nothing else this is attested to by the number of delegates attending Tuesday’s meeting.

The Cloud Circle are to be commended for getting out in front of this important issue and I hope that their work will better disseminate understanding of what is likely to become and important area and enable a wider range of organisations to begin to take advantage of it.