Now it gets personnel: Gurus and Geeks – the architecture of the Big Data universe

The universe of analysts in the world of Big Data. Where do you live?

Over the past few months I’ve been looking at what I believe to be a major meltdown of the Market Research polar caps. The growth of the industry, once assured, has turned slushy and meanwhile the growth of Big Data as a field of endeavour remains double-digit. If anything it has accelerated.  So how much effort will market researchers have to make if they wish to hitch their caboose to the big growth engine that’s running on the track next door.

It comes down to people and their skills and outlooks, so where I started my investigation was in the employment ads relating to Big Data. Ouch. The help wanted ads are dominated globally by vacancies for “data geeks” (and that’s the phrasing they choose to use) and the qualifications revolve around technical skills (typically SQL or more advanced) as well as basic statistics. Very few ads ask for Big data architects who can visualise and steer the mountains of digital data that every large firm is accumulating. I foresee a big trainwreck occurring unless a few more subject matter gurus – architects who can see the big picture – are employed in the Big data locomotive. Wanted, a few more Denzel Washingtons.

There’s another axis to the landscape, as I see it. This borrows heavily from the thinking of John Tukey, our statistical Godfather, who classifies stats into two zones: the Descriptive side (accurate reporting, concern about margins of error and significance etc) as well as the Explorational side where new patterns are being discerned, rare events are being predicted and fixation with decimal points can be quietly put aside. This is the realm of game theory, of neural networks, of unstructured data and just about all the tools that my colleagues in Market Research generally avoid. 

But while MR practitioners seldom live in the northern hemisphere of my diagram, above, not that many Big Data analysts, really, are working in that zone either. There will be strong demand, probably increasing demand for those people.

If Big Data analysts have a centre of gravity somewhere in the yellow square of my diagram, market researchers dwell, predominantly, over in the green zone. They’re good subject matter experts though not great explorers.

In respect of tomorrow’s business needs, I’m picking that most Big Data teams will require a mix of skills – people from each quadrant, or a number of generalist experts – those exceptional individuals (and I’ve met a few in both MR and BD) who dwell in the centre of the data universe – by turn gurus and technical experts, one minute retrieving old numbers and making them sing – and the next minute devising predictive models to illuminate tomorrow’s business decisions.  

Trends?  The world of business analysis will see shrinkage of MR as more and more data is retrieved from other sources. Meanwhile organisations will get quickly swamped with descriptive data and the Gods of tomorrow will be the Guru Explorers who can see the future and what they need – and are surrounded by the Explorer Geeks who can stoke the boilers and make the engine roar.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s