Monthly Archives: November 2014

A solution to the problems with ranking questions. My wine choice system revealed.

How did I choose this wine and win the compliments of the wine-waiter? My sad secret is revealed, and shows why ranking questions don't work.
How did I choose this wine and win the compliments of the wine-waiter? My sad secret is revealed, and shows why ranking questions don’t work.

I have never liked ranking questions. For a start, they seem to place an unusually high cognitive burden upon the respondents. After a lifetime of being asked to score things out of five, and to work efficiently through batteries of rating style questions, well, suddenly ranking questions emerge like a sudden speed bump. Any efficient respondent hits the thing going far too fast, and their suspension – or tolerance for the questionnaire – bottoms out. Clunk!

It isn’t just because there is confusion in the typical respondent about the difference between rating and ranking. That’s one cause of the problems. (Heaven knows, another one is the “meaning” of the numbers now that a “5” now means fifth ranked instead of an “excellent.”)

The second cause of the problem is that we often have too many items to rank. I call this the wine list effect. If you go into a restaurant and the waiter asks you whether you have selected a wine yet, you’ll know the feeling I generally experience. There’s a list of 30 white wines. You don’t know any of them, but you know three things.

  1. You prefer Chardonnay over Riesling.
  2. Anything over $60 a bottle is ridiculous.
  3. On the other hand, selecting the cheapest wine will simply make you look mean. Avoid that one.

My own wine-choosing rules are quite efficient, and present quite a good cognitive shortcut, as all heuristics are meant to do. The rules quickly eliminate a dozen of the 30 wines on offer. The remaining wines I am, of course, ill-equipped to choose from – I am indifferent to them. So at this point I quickly point to one of the remaining 18 candidates. It’s probably a pretty good drop, and if I am quick enough in my selection I maintain a look of appearing knowledgeable.

“Good choice Sir,” says the waiter. (And with those words he has just earned his tip for the evening!) Everybody wins.

Well, that’s my strategy revealed. That’s how most people rank most things in life. As soon as the shortlist grows longer than five or six items, we start applying a few blanket rules or heuristics. Our brains are not actually engineered to conduct a fully rational ranking process – one by one – towards seven or more competing products or services or attributes.

So what kind of data do you get when you ask a respondent in your questionnaire to rank nine service attributes, or 11 brands, or a dozen product configurations: rank them from best down to worst?

It is just too complicated!

And supposing you had 1000 respondents diligently ranking these items from one down to 12, (and we are assuming that they each fully understand that one equals good, and 12 equals the worst,) what sort of data do we get? Is the gap between the first and second attribute the same size as the gap between the second and third attribute etcetera? Can we tell whether the top three attributes are clear leaders? Or have they nudged just ahead of the pack?

Likewise, do we know whether an attribute has failed to impress – or whether in fact it has actively turned off the respondents.

Ranking questions fail to explain or illuminate what’s really going on in the consumer mind. You don’t know whether I rejected the Champagne because I don’t like Champagne, or because it was way over $60 per bottle, or whether I didn’t reject it all, but rather chose something even better.

Is there an answer to this problem? One elegant solution – it isn’t perfect – is Max-Diff. In a typical exercise you may have a dozen variables which you wish to see ranked meaningfully.

Max-Diff breaks the cognitive process down into may be seven or eight chunks. Each time the respondent is presented with four variables and asked to choose the most important, as well is the least important. They then repeat the exercise, but with different combinations of variables – they may do this seven or eight times. This gets us over the cognitive problems associated with ranking questions.

It also gets us over the analytical problems. Here, and I’m thinking of the Sawtooth Max-Diff module, each variable in question is ultimately scored with an index that clearly shows the relative preference or rejection. I can see the degree to which each was chosen as “best” or as “worst” – or I can see the net score (best minus worst.)

Thus we can see that Variable1 is easily preferred – and by a big margin – over the second ranked variable. From the data we might understand that six of the 12 variables were testing cluster together – neither particularly liked, nor disliked. In other words we get to see something of the thinking that goes behind the ranking scores. We understand better the nature of the consumer decisions.

If you applied this software to my various wine list selections over the past 10 years you would understand that Duncan Stuart – wine-wise – is driven by a rejection of high prices, and a rejection of certain varieties. By contrast, the sad facts would reveal, this writer is neither driven by supreme wine knowledge, or by a fail-safe palate. There: Max-Diff has just revealed my dodgy wine selection secrets.

Max-Diff is an easy module to use, it works well in online surveys, and the analysis is blindingly simple. As a research tool it fulfils the basic need we have to understand human choice-making. It understands that people are not all that mathematical in their approach, but are intuitive and sometimes less logical than the wine waiter and assorted guests might ever suspect.

Could an old-fashioned Italian grocery store be the future of grocery buying?

Montepulciano Italy - this micro store gave us foodie moments that don't exist in the mass marketing model of the supermarkets.
Montepulciano Italy – this micro store gave us foodie moments that don’t exist in the mass marketing model of the supermarkets.

Sometimes, during our recent holiday in northern Italy, we needed the Detective skills of a Sherlock Holmes to find the local grocery store. Having been bought up to look for a vast car park and a well lit supermarket as the signatures for fresh food buying, it was disconcerting to live in a series of villages – in Tuscany and Lake Garda – that appeared to be closed up for the season. Where do locals go for their daily bread?  The answer, as Holmes would have told us, is ‘alimentary’ which is the Italian word for corner foodstore. Go past one of these during siesta hour and all you’d see are shuttered doors.

These little shops, some of them little bigger than a hole in the wall liquor joint, turned out to be more like Aladdin’s cave when it came to tasty produce. We were never fluent in Italian, but the store owners – once it was our turn to be served – gave us full undivided attention. When we asked the cheese for example, they asked us in turn: local?

They meant the cheese. You could generally choose from a range of half a dozen non-local varieties, or from half a dozen cheeses that represented the locality. The same was true of prosciutto, and the bread of course was locally baked that day. Each day apart from our travellers budget breakfast of yoga and grapes, we dined on simple, artisanal, and fully delicious foods. I ate a lot of pasta, a ton of cheese, a truckload of bread – and over the five weeks I actually lost 5 kg. The diet had no added sugars and glutens and other bogus flavorings.

In fact it struck me one day that far from experiencing a traditional European grocery-buying process, I was in fact getting a taste of the future of FMCG.

What the supermarket model does is actually slow FMCG down. Instead of being delivered daily, so-called fresh bread is delivered every two or three days – and to compensate, and to retain that fresh fluffy feel, the bakers add gluten, sugar and various other nasties to simulate the fresh bread experience. Fail. The first thing we’ve chosen to do upon returning to New Zealand is to purchase one of those breadmakers, and to never again by the crap we purchased in the past from the supermarkets. The fresh Italian bread was really that life-changingly good!

But the same comment could be applied also to the local-ness of the products. We travelled extensively in northern Italy, and in each village the specialty foods – for example those cheeses – differed from region to region. Very often the manager of the food store knew the cheesemaker. Likewise he or she knew the bakers who delivered product fresh each morning.

This was the polar opposite of the New Zealand supermarket experience. Consider well-known New Zealand brands which are in fact grown, processed and packaged offshore. Consider health products – fish oil for example – which say fresh New Zealand on the pack, but originate, months previous, in factories based in South America. How can consumers enjoy the rich variety that comes from different regions, and from different seasons, when the supermarkets seek – for whatever reason – to homogenise their choices? In supermarket land, Braeburn apples grow 365 days a year. Strawberries are never out of season, but by the same token never quite in season either. The bread, as I said, is a simulation of the real thing.

Supermarkets represent the pinnacle of the production and distribution and marketing model circa 1959. Supermarkets were an amazing thing 60 years ago, but now in an age of RFID, social media, customised communications channels et cetera – not to mention a pickier health-conscious society, perhaps the old-fashioned ailmentary is closer to the future than the mass marketing model created during the age of the Jetsons. Support local I say. And start enjoying your food!