Tag Archives: Big Data

Here’s why I think Market Research companies are flying backwards.

Yes, technically we’re flying, but only just. Kitty Hawk thinking doesn’t help high-flying clients.

The very last meeting I attended when I was with my previous employer did not go particularly well. It was the weekly meeting of senior execs and on the agenda were things like staff training, client account management, and the usual HR concerns. I already knew that I was going to be leaving the organisation, so I wanted to end by contributing something which I felt was very important. “I want to run some staff training on SPSS.”

I didn’t expect the answer from my senior colleague. She asked simply: “why?”

I was quite taken aback, and I must admit I fumbled for an answer, mumbling things like; “extracting more value from the data we collect.” I can’t remember my exact words but they were along those lines. Anyway the workshop suggestion got kiboshed, and I left the organisation a few weeks later.

Three weeks ago one of the bigger clients of that same organisation announced that they were getting SPSS in-house, and they introduced me to a new member of the team, a particularly bright analyst who had previously worked with the banks.

I realised I had just witnessed a well expressed closure to my fumbling argument from 18 months earlier. The reason we need to smarten up our analytical skills as market researchers is because if we don’t,  we will simply get overshadowed by our clients.

In fact I see this all over the place. In-house analytical units within banks are using SPSS and SAS quite routinely, and most of the major banks have adopted text analysis software to help them swiftly code and analyse their vast streams of incoming verbal feedback. The text analysis centre-of-gravity in New Zealand is located client side, and not on the side of the market research industry. The same could be said of Big Data analytics. MR companies are scarcely in the picture.

Meanwhile, what is happening within market research companies? Well this last week I’ve been in conversation with two analysts looking for better more challenging roles than they have been given within their market research firms. One of them, an employee with one of New Zealand’s leading MR firms, (and one I largely admire – they are by every measure a world-class operator,) had for several years been asked to produce crosstabs and other descriptive outputs, and had on no occasion, ever, had his mental capabilities even remotely stretched. I’m talking about a graduate in statistics who has effectively been cudgeled to death by the rote boredom of low calorie market research thinking. I asked what he was equipped with, software-wise and he told me: “Microsoft Excel.”

This is simply not good enough both professionally or stragically. While globally the volume of marketing data analytics is growing by something like 45% per annum, the market research industry is relatively flat-lining or showing single digit growth at best. In other words most of the growth is happening over at the client’s place. And they aren’t even specialists in data.

If the market research industry wishes to gain relevancy, then it has got to offer something that clients can’t provide for themselves. It used to be the market researchers provided unrivalled capabilities in the design and execution and analysis of market research projects. The key word here is “unrivalled” but I’m afraid the leading research firms are being simply outstripped by their own clients.

The mystery to me is why the larger firms appear blind to this phenomenon. Perhaps in building their systems around undernourished, under-equipped DP departments, they have a wonderfully profitable business model. Pay monkey wages, and equip them with Excel. And for that matter, keep them at arms length from the client so they never get to see how their work even made a difference. The old production line model. Tighten the screws and send it along the line.

Or perhaps the big firms are simply comfortable in doing things the way that I’ve always done, or perhaps senior managers, having grown up in Kitty Hawk thinking lack the imagination or the will to fly into the stratosphere.

Either way, if I was a young analyst and looking at my career prospects my attention would be over on the client side, or on dedicated data analytics operators such as Infotools. That’s actually a painful thing for me to say, speaking as a life member of my market research professional body. But if the status quo prevails, then we are going to see not just the relative decline, but the absolute decline of our industry.

What can market research firms do to rectify this problem?  Here are 4 suggestions:

  1. Invest in decent analytical software. Just do it. A few thousand dollars – for a much better return than sending your exec to that overseas conference.
  2. Reignite the spirit of innovation becomes from your groovy team of analysts. Rather than focus merely on descriptive data, let them loose on the meta data – the stuff that explains the architecture of the public mood.
  3. Develop a value add proposition to take to clients. Look, all of us can describe the results of a survey, but we market researchers know how to make that data sing and drive decision-making.
  4. Employ specialists in big data, so that as market research companies we can integrate the thinking that comes from market surveys, and qualitative work, with the massive data sitting largely untouched in the world of the client.

In my view market research industry has been going off-course for the last 20 years. We are stuck at Kitty Hawk. We stopped shooting for the moon.




How to find the needle in a haystack of 30 billion lines of straw.

When I was in NYC two years ago the Occupy Wall St movement was in full swing.  I wandered to nearby Wall St (the protest was in a park several hundred metres away) and the famous street was being protected by dozens of police mounted on horseback. There were cordons everywhere and brokers rushed from building to building during their lunch hour in what felt palpably like a state of siege. A pretzel cart operated in the heart of the empty street – selling drinks and snacks – and I asked the vendor, an Indian man, how business was going for him. Trading on Wall St, he assured me, was slow.

Still if trading has come down from the record highs set in late 2008, the NYSE still turns over something like 1.5 billion transactions per day, or around 30 billion every month. And somewhere, lurking in those 30 billion lines of data is evidence of insider trading. The question is, how do you find it?  The needle is small – the haystack unimaginably big.

The answer is that the SEC is often informed by the investigative team of FiNRA – the Financial Industry Regulatory Authority spearheaded by a man who used to be in the DA’s office: a prosecutor you wouldn’t want on your tail. His name is Cameron Funkhouser, and he’s a big thickset man described by one colleague as an “I can smell a rat, kinda guy.”

But get this – the investigative team he’s put together to delineate all those billions of lines of data, looking for patterns of fraud and insider trading (do two apparent strangers make the same trades on the same days, repeatedly – could they be connected?) this team he’s put together is not chiefly characterised by data geeks and statistical nerds. Heading the unit, underneath Funkhouser’s command is a team spearheaded by:

  • Whistle-blower unit – Joe Ozag. (Ex terrorism detective.)
  • Anthony Callo – once prosecuted homicides at the DA’s office.
  • Laura Gansler – screenwriter and the “brains” of the group.

This team is characterised by diversity, out-of-the-box thinking, and an understanding not so much about statistics as about people: our drives, our motivations and our craftiness: in short; our narratives. Laura Gansler might be the least statistical member of the group – she is an accredited screenwriter and author – but she’s well versed in developing credible storylines and getting inside the head of crooked characters.

And this is the point about dealing with big data. For sure, your programmers and number grunts can dig around and reveal black and white statistical evidence of those occasional insider trading schemes – but the real value in this analysis comes from those who can furnish a credible story and, using logic, suggest where best to look for the telltale finger prints, bloodstains and data trails that mark the crime.

FINRA has proved remarkably effective at helping nail the bad guys, and Funkhouser speaks highly of his team.

The lesson for researchers and business people is that the best way to deal with massive data – 30 billion lines of the stuff every month – is to remember two things.

1)  Every bit of data reflects human activity. It isn’t about numbers – it is about people.

2) Get inside the mind of the bad guys – and you know where to look.

Incidentally the FINRA unit reflects, surprisingly closely, the fictitious Department S – the 1970s show that gave us Stewart Sullivan, conventional crime-fighting agent, Annabel Hurst, computer expert, and the redoubtable Jason King, cravat wearing international novelist of mystery.  Department S was born of a time when traditional crime (bank jobs and murders) were giving way to global operations. (The French Connection was another movie that wrestled with the same escalation of crime.) So it is interesting that the escalation of data into Big Data should meet with a similar response: put talent and creativity onto the case otherwise there will never be enough resource.



What a good, memorable story requires.


This last two weeks I’ve been reading a textbook called Storycraft, written by journalist Jack Hart and designed to help writers of non-fiction hone and enrich their skills: to turn true events into compelling stories. I was pleasantly surprised actually, because the book is damned good, but it reminded me how much I had learned in a previous life as a script writer and as a freelance journalist. For sure, there were new insights and tips that I will dial-up in future, but the most useful function of the book for me was to set out a formal checklist of things we ought to incorporate in a compelling story – especially one based on data. Here are a few must haves.

1) A clear tone of voice and standpoint. As teller of the story are you the problem solver who was given a challenge, or are you the skeptic who is trying to disprove something? Are you an insider or an independent outsider?  Be clear on this.

2) A clear story structure. Stories usually start calmly but quickly a crisis or decision-point occurs that threatens to change everything. The problem gets worse, and then gradually the heroes (in analytics perhaps, or those amazing customers and what they told us) wrest the flight controls off the dead pilot and set about bringing this aircraft down safely.  Most story structures rack up the tension and then engage in the process of solving the problems. Always, there are decision-moments along the pathway.

3) You need characters – especially good guys. Now in crunching Big Data, you’re reporting on numbers, right? Well, not quite. Those numbers represent people – so it can be useful to pull out one line of data, give the customer a nickname  – Honest Harry – and use him as a cipher to tell the big story. Here’s where Harry faces a choice – what will he do?  Personify the data. Don’t forget there are other characters in the story as well – including the analytics team.  

4) Setting.  A good story is underscored by the setting. High Noon took place in a lonesome, Godforsaken town miles from any help.  This framed Gary Cooper’s dilemma and added to the tension. CSI uses Las Vegas or NYC to good effect to create for each series a memorable backdrop against which their problem solving skills stand out in stark relief. When you’ve got 30 minutes to stand up and report on what your analytics have found – don’t forget to devote 3 or four minutes setting the scene.

5) Satisfactory denouement. The wrap up of the story had darn well better sing – not fizzle out. So in putting together your presentation or report think hard about this.  The plane is coming in to land, there’s ice on the runway and a small child (and a few nuns, there are always a few nuns) in the passenger cabin.  Structure the story so that when the ending occurs – the 747 ploughs through the snow on its only wheel before coming to rest right outside departure gate 9 – everything wraps up tidily. The hero gets home for thanksgiving. The little girl is saved. The nuns collective faith is restored. 

Now in writing these things I come over as pretty glib. Yet I’ve seldom done a presentation without thinking of these elements. At first I thought it was just a duty, if you have a story, tell it well.

But these days there’s a much bigger reason for storytelling skills to be employed in the boardroom. Big Data deals with 8 zillion narratives. You want this to be the one that the decision-makers remember.

Story technique has been with us for 2300 years – it’s time to brush up on it

2300 years ago Aristotle wrote “The Poetics” which contained his secrets of storytelling. Being human, these secrets haven’t really changed.

Many research presentations I’ve seen, including many of my own, have been bogged down by too many facts and figures. It is like reading a book which is so full of florid description that one begins to skip pages and start looking for the action.

Likewise stories can suffer from relentless action – the type we see in Peter Jackson movies where we get chase, fight, chase, fight, another chase, another fight followed by another chase – and the net result is just plain boredom. His King Kong movie is one of the few films I’d rate as un-watchable. It ignored the storytelling craft. It was all pageantry (look at our CGI techniques!) and no drama.

We do the same with research. We go heavy on descriptive results – without telling the true story. Or we go heavy on special effects (I do this too much: showing off analytic techniques) but forget to tell the story. Or we simply have a story but we don’t tell it with any craft. We muddle it up. The drama is in there somewhere but we didn’t quite extract it.

This seems crazy, because the craft of telling stories – the techniques and skills required – have been part of our pantheon of written human knowledge for 23 centuries. Storytelling goes back to the dawn of civilization  but the Greeks started thinking about the craft, and analysing it, and applying systematic rules to it since Aristotle considered the subject.

Why not? As Steven Pinker explains it, storytelling has universal elements across so many cultures to lead him to conclude that stories are part of how our brains are wired. They reflect how we think. We’re engineered to tell stories.  Stories are a means of processing complex visual, verbal and emotional information.

In the 20th Century much was written about story telling craft as writers considered modern day psychology and found, among other things, how well Shakespeare captured the human condition. You could pick apart Othello and find it stood up to a Jungian framework, or to modern theories of the human condition. Writers such as Lajos Egri who published the seminal guide for playwrights The Art of Dramatic Writing helped create the debate about what drives a good story: is it events and action, or is it character? He concluded that character was at the heart.

So here is a good definition of what makes a good non-fiction story, summed up by American Jon Franklin in his work: Writing for Story.

“A story consists of a sequence of actions that occur when a sympathetic character encounters a complicating situation that he or she confronts or solves.”

Sounds simple, and – actually – it is. But the next layer down is where the story craft gets more complicated:

  1. Giving the story some structure. Do we start at the beginning and build to a conclusion? Or do we start at a critical moment of decision – and then go back and fill in the back-story and offer the options that our lead character faces?
  2. Choosing from whose point of view we tell the story. (Do we tell it from the brand’s point of view?  Or the customers’ point of view?)
  3. Characterisation. Do we paint the Brand as a hero? Or is it a flawed everyman? Are those customers a roiling mass – a Spartacus uprising in the making? – or are they the savvy, price-seeking satisficers who are undoing the good work of marketing? Who are the goodies?

These are just some of the decisions we must make when we tell stories, and they require a lot of forethought and imagination. The process is far different from the usual art of starting a PPT deck with Q1 and working through to the results of the final Question. I wrote a teen-novel once, (The Whole of the Moon) which did quite well but I spent a month deciding whether to tell it first person or third person.

Well before then, working in TV, I learned in drama editing and writing just how important it is to find a congruency between action and character. The decisions made by the protagonist (he kills his attacker) need to be within the realm of possibility for that character. (Would Coca-Cola really do this??)

I also learned that good stories need some relief. Shakespeare would open each act with a couple of fools joking around: something to get the rowdy audience engaged before launching into a Lady MacBeth tirade.  In client presentations or conferences I try the same, and the light moments may seem like diversions, but they always have a point – they put across the enjoyment we’ve had in the project, or they give a bit of anecdotal evidence of the dramas and challenges we faced in the survey: the day the blizzard held up the fieldwork.  These diversions humanise the story, and connect the storyteller to the audience.

Writers can get into a groove and employ hundreds of these little lessons instinctively – but it is increasingly important that researchers and analysts now also learn some of these techniques. We have audiences who want to digest the main thrust of what the data is saying.

And as story tellers we don’t want them to walk out on us.

Even modern theatre employs the lessons from ancient Greece. After all, it is all about people and how they make decisions when complications get in the way of their objectives.

Now it gets personnel: Gurus and Geeks – the architecture of the Big Data universe

The universe of analysts in the world of Big Data. Where do you live?

Over the past few months I’ve been looking at what I believe to be a major meltdown of the Market Research polar caps. The growth of the industry, once assured, has turned slushy and meanwhile the growth of Big Data as a field of endeavour remains double-digit. If anything it has accelerated.  So how much effort will market researchers have to make if they wish to hitch their caboose to the big growth engine that’s running on the track next door.

It comes down to people and their skills and outlooks, so where I started my investigation was in the employment ads relating to Big Data. Ouch. The help wanted ads are dominated globally by vacancies for “data geeks” (and that’s the phrasing they choose to use) and the qualifications revolve around technical skills (typically SQL or more advanced) as well as basic statistics. Very few ads ask for Big data architects who can visualise and steer the mountains of digital data that every large firm is accumulating. I foresee a big trainwreck occurring unless a few more subject matter gurus – architects who can see the big picture – are employed in the Big data locomotive. Wanted, a few more Denzel Washingtons.

There’s another axis to the landscape, as I see it. This borrows heavily from the thinking of John Tukey, our statistical Godfather, who classifies stats into two zones: the Descriptive side (accurate reporting, concern about margins of error and significance etc) as well as the Explorational side where new patterns are being discerned, rare events are being predicted and fixation with decimal points can be quietly put aside. This is the realm of game theory, of neural networks, of unstructured data and just about all the tools that my colleagues in Market Research generally avoid. 

But while MR practitioners seldom live in the northern hemisphere of my diagram, above, not that many Big Data analysts, really, are working in that zone either. There will be strong demand, probably increasing demand for those people.

If Big Data analysts have a centre of gravity somewhere in the yellow square of my diagram, market researchers dwell, predominantly, over in the green zone. They’re good subject matter experts though not great explorers.

In respect of tomorrow’s business needs, I’m picking that most Big Data teams will require a mix of skills – people from each quadrant, or a number of generalist experts – those exceptional individuals (and I’ve met a few in both MR and BD) who dwell in the centre of the data universe – by turn gurus and technical experts, one minute retrieving old numbers and making them sing – and the next minute devising predictive models to illuminate tomorrow’s business decisions.  

Trends?  The world of business analysis will see shrinkage of MR as more and more data is retrieved from other sources. Meanwhile organisations will get quickly swamped with descriptive data and the Gods of tomorrow will be the Guru Explorers who can see the future and what they need – and are surrounded by the Explorer Geeks who can stoke the boilers and make the engine roar.