Benchmarking Studies – are they worthwhile?

How are we doing?  That’s the basic question everyone asks in the workplace. Back in 1999 I read a fabulous meta-study of hundreds of employee surveys and their methodologies, and the core question – the very heart of the employee’s anxiety was summed up as: “How am I doing?”

Managers adopt the royal “we” and ask the same thing: how are we doing?

Compared to whom? In the 1980 and 1990s this question usually lead us down the track toward benchmark studies. Benchmarking was quite the buzzword in the 90s, driven as it was by TQM and various Best Practice initiatives. Back then everyone wanted to be like IBM.

These days everyone wants to be like Apple (presumably they want to operate like anal control freaks – unhappy but proud of their work,) but that’s not the issue. The issue is around the whole idea of benchmarking.

It used to be that research companies loved these studies because if you could generate a big enough pool of normative data, then everyone had to come to you – the experts who have 890 studies of soap powder studies from around the globe. It was a license to print money. And better still, the client could tell their stakeholders: “Look – we’re in the top quartile!”  (The score may be crap but we’re better than most.)

But sooner or later benchmarks come unstuck. For a start there is the IBM effect. IBM used to be the company that everyone wanted to emulate – and just about every company that did went over the same cliff that IBM steered toward in the early 90s. They lost their way when the computing industry moved from a box focus to a software focus. Suddenly IBM had lost its mojo.  So the clones all disappeared while those with a unique story – the Apples and the Microsofts – who didn’t benchmark themselves to IBM, merrily went their own successful or at least adventurous way.

Then there is the problem of relevancy. If you benchmarked your bank’s performance against all the other banks in the world, would that even be useful? (As a reference point, how about the bank I deal with in Cambodia that didn’t put an auto-payment through because – and I quote – “we were having a party that afternoon.”) Is it actually useful for local banks here in NZ to compare themselves to these other operators? Does it make a local customer quietly glad that their branch rates higher than does the leading bank in Warsaw? I think not.

Would our client prefer to benchmark his salary against global norms? (Hey, we’ve got third world data included.)

But here, for the researcher is the ultimate rub. If a client insists on benchmarking by adjusting their survey to ask the same questions as “the survey they used in Canada,” then what we’re buying into is not just the idea of comparing results, but also into the idea that we may have to dumb down our questions; that we may need to run average research so we can really compare apples with apples.

I’ve tasted those apples. They’re dry and they’re floury.

Please. Give me crisp, fresh locally grown produce, and leave normative data for those who want to compare themselves to …er, normal.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s