Category Archives: Better market research

Measuring the 2 million combo menu.

Upsize this. Downsize that. No wonder we have whiplash: a fast food menu presents us with something like 2 million possible combinations.

In an age of mass customisation we now have the tools to measure really complex customer choices.

Scenario one. A young guy walks into Burger King and he’s starving. He’s got $10 in his pocket and while he waits in the queue he stares up at the menu board. Does he go for the tender crisp chicken wrap with honey mustard? Or does he stick with his usual Double Cheese Whopper? Or should that be a single cheese but with an upsized Coke? Next please. He makes his order.

Scenario two. Life Insurance. The young mother sits at the dining table with her tablet opened to show the Life Direct website. She is comparison shopping – looking for the best life cover for herself and her husband. She enters the details into the online calculator and nominates how much cover they need (enough to cover the mortgage,) and six competing offers pop-up within 15 seconds. These are priced marginally differently per month. She has heard of some good things about some insurance companies but bad things about one of the six. And a few of the competing offers come with additional conditions and benefits. She weighs everything up and clicks one.

Scenario three. The couple have all but signed up for the new Ford. They tested other competing brands, but this is the hatchback they really like the best. “Of course,” says the salesman as he starts to fill in the sales order form; “we haven’t discussed your options yet. Do you prefer automatic, or manual? How about the sport model with the mag wheels? That’s only $1200 extra. Of course you have the two-door, but how about the four-door option? And it’s a bit extra for the metallic paint – though it tends to age a lot better than the straight enamel colours.”

“You mean it’s not standard?” she asks. The couple look at each other. Suddenly the decision seems complicated. She murmurs to her partner that maybe the Mazda with the free “on road package” is back in the running. The salesman begins to sweat. The deal is slipping through his fingers.

We are in the age of mass customisation

Three very common consumer scenarios. In an age of mass customisation, consumer choices don’t just come down to brand preference or advertising awareness – the two staples of consumer research – but rather come down to an astonishingly complex human algorithm which somehow handles what we might call menu-based choice architecture or MBC.

How complex? One estimation I have seen of the typical burger menu board from where during the 90 seconds in which you queue up you must make a choice, is that there are in excess of 2 million possible combinations and permutations for a typical $15 order. Double cheese. Extra fries. Upsized Coke. Hold the beetroot. Everything comes with a cost and a trade-off, governed by personal tastes, one’s own sense of value and the various conditional skews of this week’s promotions. Somehow humans manage to make a decision, though to be honest, you can see how habits develop. They become an efficient way of dealing with information overload.

A big challenge for market researchers

How do researchers even begin to test the attractiveness of 2 million options? The problem for market researchers is not just one of volume. The complication comes from the fact that the consumer choices are not strictly rational. When you think about it, paying an extra dollar for an additional slice of cheese in a double cheeseburger is bloody expensive. And if I’m constrained by the $10 note in my back pocket, then who can tell whether I would prefer an upsize Coke and small fries with my burger, or a Jumbo coke and no fries at all? Or no Coke at all, but extra fries? Or Coke and Fries but hold the extra cheese? Or…

Menu-based choice modelling is a relatively new field of market research that springs from its roots in conjoint analysis.
Conjoint, I learned recently to my delight, was a term first applied by my statistical hero John Tukey, who was fascinated by the question of how we can measure things that are measured not discretely, but considered jointly – hence con – joint.
Since the 1970s conjoint approaches have been applied with increasing confidence.

At first these were paper-based, and a typical example of this kind of study gave respondents a set of 16, 24, or 32 cards on which various combinations and permutations of product description were portrayed. Generally this might be enough to efficiently calculate the relative attractiveness of competing offers.

For example if five competing brands of peanut butter came in four different price points, low or medium saltiness, super crunchy, regular crunchy or smooth, then we have 5 x 4 x 2 x 3 = 120 potential combinations of product to test. Good design might reduce this to a fairly representative 32 physical cards each with a specific combo of price, brand and flavour etc.
The conjoint approach – with carefully calibrated options to test – enables us to determine which factors are most driving consumer decisions (price is often the main driver,) and the degree to which consumers consider the trade-offs between, say a preferred brand at a higher price, versus a lower priced offer from a brand we don’t trust so well.

Conjoint has the advantage of boiling each variable down to utility scores, and – since price was one of these variables – allowing us to put a price-tag on brand appeal, or flavour.

Even so, the paper-based system, because it still requires at least 32 cards to give us enough data on the peanut butter survey, still puts on respondents a high cognitive load.

By the 1980s conjoint studies were made possible on computers, and this approach enabled some clever variations to be tried. First, the computer approach eased the cognitive load by offering three or four cards at a time so that respondents could more easily choose their preferred option (or none of these) over 9 or 10 iterations. Another innovation: useful in some situations but not universally, is adaptive conjoint, which in cycling a respondent through a series of choice exercises, may quickly decide that this respondent never chooses Sanitarium brand, so it begins testing variations among the preferred ETA and Pam’s brands. It focuses on the useful part of the choice landscape.

These approaches have been increasingly honed and refined, and I have always admired the developers of Sawtooth for working and reworking their algorithms and methodologies to create increasingly predictive conjoint products. They are willing to challenge their own products. They moved away from favouriting Adaptive Conjoint a few years ago.

Up until 2010 the softwares offered respondents a choice between “competing cards.” This approach works best on mass produced products such as peanut butter, where there may be less than 200 combinations and permutations available. However in the last decade or so, and with the rise of e-commerce and the presence of bundled offers (think how phone, energy and broadband packages are now being bundled,) classic choose-one-card conjoint only goes some of the way to explaining what goes on in the consumer mind.

Enter MBC.

This is a Sawtooth product and is a particularly expensive software. I recently forked out New Zealand around $12,000 for a copy, and dismayingly, instead of receiving a sexy drag-and-drop statistical package, had the equivalent of a jumbo bucket of Lego bits offloaded onto my computer. It isn’t pretty. It requires HTML coding, and requires some real steep learning to get a handle on.

In my case, I’m just lucky that I have worked with conjoint for a few years now, and conceptually can see where this is coming from. But even so, I can think of no software that I have dealt with in 22 years that has required such a demanding user experience. Luckily by the time the respondent sees it, things aren’t so complicated.

Not pretty for the researcher – but emulates reality for the respondent

With the right HTML coding, the online survey presents – metaphorically – a replica of the Burger King menu board. There you can choose the basic flavour of burger (beef, chicken, fish, cheese,) and then top it up with your own preferred options. Choose the beef value meal if you want – and upsize those fries. As you select your options the price goes up or down so respondents can very accurately replicate the actual choice experience.

And that’s the point of MBC: to emulate as closely as possible real choice situations. We might discover that consumer choice architecture radically transforms itself depending on whether there is a price point of $10, or whether there is a price point of $12 or $15. Sawtooth experience shows that many of the trade-offs are not strictly rational. There may be an overriding decision about whether to order a whopper or double whopper, but regardless, the shopper may always prefer to have a Coke of a certain size. In other words there is no trade-off here, or there is: but it applies to the fries only.

In a typical example the respondent is shown 10 variations of menu boards, and each time given a price limit and asked to choose. Given that the questionnaire needs HTML coding, it will cost the client somewhat more to conduct an MBC survey compared to a regular out-of-the-box online survey. The investment should be well worth it.

Another consideration: given the 2 million permutations available, this is not something to be tested with small sample sizes. A thousand respondents may be required as a minimum, each doing ten choices to generate 10,000 data points. Picture these as trigs on data landscape.

Given enough of these reference points, the software then has enough information to fill in a complete picture of the consumer-choice landscape. You simply can’t do this when you break down the survey into constituent questions about price, taste, brand, options etc – that is, in considering these elements not jointly, but singly.

Now comes the good part. By assembling this data, you could more or less model the optimum menu board. If I was Burger King for example, I could probably massage the menu so that the average punter with $10 in their back pocket would spend not $9.20, but something closer to $9.80 a 9% lift in revenue willingly forked out by happy customers. If I wanted to try a special offer on the flame grilled chicken burger, I could see what impact this would have on revenue and on the demand for other options.

How accurate is MBC?

As with any conjoint approach, the accuracy is largely dependent on the questionnaire design, and on the survey design which can be tested before it goes into field – using random data and running it through the MBC simulator.

If I was doing a landscape map of New Zealand and had 10,000 trig points I could probably come up with a pretty accurate picture of the geological landscape. There would be enough points to suggest the Southern Alps, and the ruggedness of the King country for example. But it wouldn’t be totally granular or perfect, and I would be extrapolating a lot of the story.

So similarly, given that we are testing, say, 2 million combinations of menu with just 10,000 data points, don’t expect the modelling to be perfect. Sawtooth has tested MBC using actual data, (but held out for comparison,) and found the software chooses the right options with 80% accuracy or thereabouts. So for my money, the MBC accuracy is right up there with Neural Networks for accuracy, but much more useful than NNs for testing menu-based choices and explaining the interactions that are going on.

Currently I’m employing the software for a client who is looking at bundled products, in a particularly complex market, that’s why I bought the software. There is simply no other tool available to do the job. Without giving any client information away, I will within a few weeks disguise the data and make available a case study to show how MBC works, and how it can be used.
I mention this because I am a strong believer in the idea that market research professionals ought to share their experiences for mutual benefit. As a profession we are under mounting challenge from the big data mob. However, so long as they stay in the world of Excel, and so long as market researchers remain aggressive in the way we trial and purchase new software advances, we will continue to have a healthy profession. My first reaction to MBC is that it is an answer – at last – to a world where customers can now customise just about anything. You are welcome to contact me if you want to learn more.

Duncan Stuart
64-9 366-0620

The big thing we forget to measure

RISKIn our market research reports we generally make a concerted stab at the idea that our data is both precise and reliable. We gleefully report the sample size,  and we pinch our fingers together as we cite the maximum margin of error – which in many surveys is plus, or minus, 3.1%. Talk about pinpoint accuracy!

Yet we blithely ignore the fact that our clients work in a fuzzy universe where things go right, or horribly wrong. If you were the brand or marketing manager for Malaysian Airlines this year, I really wonder if your standard market research measures – brand awareness, consideration, advertising awareness etcetera – would have anything remotely to do with the fact that you have lost, very tragically, two airliners within the space of three months. Risk happens. Regardless of your marketing investment, passengers aren’t flying MH.

Or if you are the marketing manager for Coca-Cola in your country, do you think honestly that the subtle shifts of brand awareness, ad recall and consideration have as much effect as whether or not this summer proves to be wet and dismal, or an absolute scorcher?

We may not have a crystal ball when it comes to weather forecasting, but we do have decades of accurate climate data.  When I did this exercise a few years ago I gathered 30 years worth of data, popped it into Excel, then used a risk analysis tool to come up with a reasonable distribution curve based on that data. It looked something like the chart above.  Then I could set my temperature parameter – below x° – and based on that, could fairly reasonably calculate that my city had a 20% chance of having a dismal summer. The risk was high enough, I felt, that any marketing manager for any weather sensitive product or service should have a contingency plan in case the clouds rolled over and the temperatures dropped.

Why don’t we do this more often?  Why don’t we build in the various risks that accompany the work of our clients? If we did, then we could better help them to make decisions as circumstances arise.  We consider some risks – what if the competitor achieves greater brand awareness and consideration – yet we treat many of the other risks (weather,  currency or price fluctuations, whether or not supermarkets choose to stock us or otherwise, whether or not some kind of health scare will affect our category, etcetera,) as if these were off-limits and outside our scope.

Though these data are not outside our scope to all. The data may not come in via our surveys, but they are just as relevant.  Going back to the weather example: we did observational research at a local lunch bar, and found that on wet cool days the pattern of drinks consumption was quite different to that on hot sunny days. It wasn’t just a question of volume. On cool days people switched from CSD’s – slightly – towards juices, as well as choc-o-milk type drinks.

So if I was a soft drink marketer, and I had a market research firm supplying me with climate risk data, as well is an idea of how people behave when it is hot, average, or cool – then I might come up with a marketing plan for each of those circumstances. I would go into summer expecting it to be average, but as soon as the long-range weather forecast told me  that indeed the weather is going to be cool – I would think, well, I had a 20% expectation that this would happen. Time to wheel out Plan B. I’d be prepared.

The risk analysis tool that I use is called @Risk and it is frighteningly simple to use. It works as a plug-in to Excel, and takes roughly 10 minutes to learn.  Since using the software, my outlook toward what we do as market researchers has totally changed.

We are not in the survey business. We are in the business of assisting our clients to make informed, evidence-based decisions.

Sometimes the evidence comes from our survey work, bravo! But sometimes the evidence comes from the weather office, or from the war zone of the Ukraine.

Why we need to measure corporate forgivability

Image
One day before the explosion in 1984, Union Carbide were the friendly guys who brought you the Eveready brand and the Cat with 9 Lives. The NPS would have been pretty good. And pretty useless at measuring the forgivability, or lack of forgivability of the once proud Union Carbide name.

Most organisational and market research measures – the KPIs – are readings of static concepts rather than of processes.

The well trodden NPS measure is a measurement of hope, for example (how many of our customers would recommend us?) but it does little to indicate whether this score might be in danger of imploding.  It is like a measurement of speed, but without the other vital measure on your journey: how full is the tank?  The speedometer and fuel gauge  together measure a process and offer a predictive capability. “At this speed we’re using up too much gas to make it the the destination.”

Here’s another measure: Staff satisfaction. It may be high, it may be low, but the score itself says little of any predictive use.  Thus everybody might be deliriously happy at work (and I hope they are,) but we have no idea from the happiness Index whether they’ll stay happy once the take-over goes through, or when the job cuts are announced.  In fact the measure is pretty useless. It is a thermometer when really it might be more predictive to have some kind of staff barometer which hints at soft or deep changes immediately ahead.

Now brands are measured along similarly useless lines. They’re measured statically but not predictively. We measure the status of the brand, but that gives no feel for whether it is heading toward a cliff; or whether it would survive the fall, even if it did.

When we judge the people around us, we don’t just stop at “he’s a great guy” or “she’s 5 foot 11,” or “she’s a real leader,” we almost always add what I refer to as the moral dimension. “He’s a great guy…BUT I wouldn’t trust him with my money, or “She’s a real leader …AND did you see the way she showed so much respect, even to the people she had to make redundant.”  In other words we don’t just settle for today’s status update, we also tend to throw in the prognosis of how that person will act if they’re put in a conflicted, morally challenging position. We have a “Moral Vocab” with which we assess the likely behaviours of those we judge. Great guy – but morally iffy. Great leader, and puts people first no matter what.

The star term, I reckon, in this Moral Vocabulary is forgivability. It is a useful concept because it accepts that all people will fail at some point, or that all organisations will have their crisis and every brand will have its “New Coke”/”Ford Edsel” moment. They’re bound to. Forgivability measures how people will respond if and when that crisis occurs.

In terms of the resilience of the brand, or the company, the real test is not how you rate when everything goes according the plan – but how quickly you can bounce back if you falter.

To take an obvious case, Tiger Woods was the undisputed star of the golfing world (and still the highest paid athlete in 2013 despite not winning a major tournament.) A typical brand measure, up to the point of his personal and media scandal, would have given him stellar results on, say, an NPS scorecard, or on any other brand index I can think of.

But look how quickly that turned. The moment Woods fell to earth in December 2009 (or should I say, the moment he ran his SUV into a fire hydrant), everything changed. The tank of public goodwill suddenly showed “Less than half full” and sponsors started to walk away. Suddenly the values we valued in this amazing sportsman were reframed and seen in new light. Determination? Or simply arrogance?  Success? Or just runaway greed?  Perfection? Or just a sham facade? Everything that a brand measure might have rated as superb one day was shattered within 24 hours.

Four years later forgiveness has largely occurred. The gallery is generous once more when he makes a great shot, and descriptions of this golfer are laced more with qualifications about his pay check or his climb back to form, than they are about who he has a personal relationship with.

Organisations can rate well in terms of forgivability, or they can rate poorly.  It depends, as it did for Tiger Woods, on the seriousness of the sin and the forgiveability of the sinner.

In my view, Tiger Woods forgivability was undermined by the woeful stage-managed response by his sponsors. Remember those bleak Nike ads where a voice over (supposedly Tiger’s own dad) remonstrated mournfully with our hero? These attempted to package the redemption of Tiger Woods into the space of a 30 second TVC. The ads assumed that with a quick show of humility we’d be swift to forgive the golfing superstar.  Instead the ads gave us evidence that Woods  – he behaved not a man but as a marketing juggernaut – was attempting to media-manage his way out of his mess. It looked merely like insincere spin doctoring. Another sin! And so for weeks the Woods machine kept heaping more fuel onto the fire.

But what of your organisation? It may be sailing along – the speedomoter is reading high, the thermometer reading nice and warm, but what if it made a blunder? It will happen. How will your stakeholders or customers respond?

Apple, that golden child of the business media, has a string of business and product blunders a mile long. But was it forgivable?  Absolutely.  Why? Because the products are cool and because Steve Jobs never really deviated from his vision. The public understood his quest and knew that some failures will litter the pathway to success. No problem.

But post-Jobs, I think the forgivability factor is trending down. Steve’s quest is over and what we perceive is the hulking cash-cow of an organisation he built. The product may be designed in California, but the cash is domiciled wherever the company can avoid tax. Things like that start to reframe Apple not as “one man’s passion” but as just another bloody corporate.  In that light, every new launch looks less like Steve’s marvellous march of innovation, and more like the CFO’s latest plan to sucker the public. You can almost hear Mr Burns from the Simpsons. “A masterstroke Smithers! We’ll do what Microsoft used to do with Windows. Yet our fans will still think we’re the anti-Microsoft!”

Some sins are purely business as normal. Coke really did believe their New Coke formula was a better, more preferred option. They didn’t think things through.

But some sins are simply not forgivable. Union Carbide, that industrial fortress of a company that made Eveready Batteries, and Pesticides and Glad Wrap, was responsible for one of the worst industrial accidents in human history with the Bhopal disaster in India, back in December 1984.

Here was a company that was deliberately trading-off the cost of safety in order to boost profits from its poorly resourced pesticide plant located in a heavily populated area. As a result of an MIC gas explosion an estimated 40,000 individuals were either permanently disabled, maimed, or suffering from serious illness.

That was bad enough. But then after the disaster, Union Carbide tried overtly to avoid culpability or to pay any compensation to the families of the accidents thousands of victims. There was no mea culpa – instead the company fought a legal battle before finally being sued by the Indian Government for US$470 million, 5 years after the disaster. The guts of their defense was was that they weren’t responsible as a company for Bhopal – it was the fault of their employees in India. It was a massive squirm. The head of the company Warren Anderson was never brought to justice in India after the American fled India while on bail, and has since fought extradition from the USA. Today the company no longer owning its flagship brand (Eveready) and is part of the Dow Chemical company who have inherited the mess. In 2010 (25 years after the disaster) eight former executives of Union Carbide India Ltd. were finally found guilty of death by negligence. Dow, themselves burnished by the reputation of their own history with Napalm and Agent Orange are still assisting with the highly toxic Bhopal site cleanup.

Mistakes, blunders and sins can be made by any organisation. But how soon can these organisations recover – how soon can they be forgiven? In a dynamic world researchers need to measure these things. In my next blog I’m going to dissect the elements of forgivability. Get it wrong and your organisation will tread an unnecessarily risky path.